Data Replication User s Manual (Disaster Recovery System Installation and Operation Guide)

Transcription

1 NEC Storage Software Data Replication User s Manual (Disaster Recovery System Installation and Operation Guide) IS027-34E

2 NEC Corporation No part of the contents of this book may be reproduced or transmitted in any form without permission of NEC Corporation. The contents of this book may be modified without notice in the future.

3 Preface This manual is bundled with NEC Storage RemoteDataReplication/DisasterRecovery to explain, through descriptions and examples, how Disaster Recovery functions are used for data replication and how disaster recovery systems are introduced and operated. This manual is intended for readers who already possess the basic knowledge about data replication that is essential for using Disaster Recovery functions to introduce and operate a disaster recovery system. For description of data replication, refer to the NEC Storage Software Data Replication User s Manual (Function Guide) (IS015), NEC Storage Software Data Replication User s Manual (Installation and Operation Guide) (IS016 or IS020) in accordance with your OS in use, and NEC Storage Software ControlCommand Command Reference (IS041). Refer to the NEC Storage Software Manual Guide (IS901) for a general description of NEC Storage and related manuals. Remarks 1. This manual explains functions implemented by the following program products: NEC Storage Manager and NEC Storage BaseProduct NEC Storage ControlCommand (Note) NEC Storage DynamicDataReplication NEC Storage RemoteDataReplication NEC Storage RemoteDataReplication/DisasterRecovery Note: NEC Storage ControlCommand is a program product that has integrated the following five functions. ReplicationControl SnapControl ReplicationControl/DisasterRecovery ProtectControl PowerControl 2. This manual is applicable to the program products of the following versions: NEC Storage Manager Ver9.7 or later NEC Storage BaseProduct Ver9.7 or later NEC Storage ControlCommand Ver9.7 or later

4 3. The NEC Storage Manager is referred to as ism or Storage Manager in this manual unless clearly specified. Also, the following terms refer to the corresponding NEC Storage software products. Term NEC Storage Software Product AccessControl BaseProduct ControlCommand DynamicDataReplication or DDR PathManager PerformanceMonitor PerformanceNavigator RemoteDataReplication or RDR RemoteDataReplication/DisasterRecovery NEC Storage AccessControl NEC Storage BaseProduct NEC Storage ControlCommand NEC Storage DynamicDataReplication NEC Storage PathManager NEC Storage PerformanceMonitor NEC Storage PerformanceNavigator NEC Storage RemoteDataReplication NEC Storage RemoteDataReplication/DisasterRecovery 4. The NEC Storage series disk array subsystem is referred to as a disk array in this manual unless clearly specified. Also, the following terms refer to the corresponding NEC Storage hardware products. Term NEC Storage Hardware Product D3 series D4 series D8 series M series A5000 * xxx and xxxx represent the model number. NEC Storage D3 series NEC Storage D4 series NEC Storage D8 series NEC Storage M series NEC Storage A The following terms in this manual refer to the NEC Storage software manuals. Term NEC Storage Software Manual User s Manual Configuration Setting Tool User s Manual (GUI) Configuration Setting Tool User s Manual (GUI) for the M Series Messages Handbook Data Replication User s Manual (Function Guide) Data Replication User s Manual (Installation and Operation Guide) Snapshot User s Manual (Function Guide) ControlCommand Command Reference Storage Manager Command Reference Partitioning User s Manual NEC Storage Software NEC Storage Manager User s Manual (UNIX) (IS001) NEC Storage Software NEC Storage Manager User s Manual (IS004) NEC Storage Software NEC Storage Manager User s Manual (IS050) *For the M series NEC Storage Software Configuration Setting Tool User s Manual (GUI) (IS007) NEC Storage Software Configuration Setting Tool User s Manual (GUI) for the M Series (IS051) NEC Storage Software Messages Handbook (IS010) NEC Storage Software Data Replication User s Manual (Function Guide) (IS015) NEC Storage Software Data Replication User s Manual (Installation and Operation Guide for Windows) (IS016) NEC Storage Software Data Replication User s Manual (Installation and Operation Guide for Linux) (IS020) NEC Storage Software Snapshot User s Manual (Function Guide) (IS030) NEC Storage Software ControlCommand Command Reference (IS041) NEC Storage Software NEC Storage Manager Command Reference (IS052) NEC Storage Software Partitioning User s Manual (IS043)

5 Term PathManager User s Manual (Linux) NEC Storage Software Manual NEC Storage Software PathManager User s Manual (Linux) (IS202) 6. Trademarks and registered trademarks Microsoft, Windows, Windows Server, and Windows Vista are trademarks or registered trademarks of Microsoft Corporation in the United States and other countries. UNIX is a registered trademark of The Open Group in the United States and other countries. VERITAS, VxVM, VxFS, NetBackup, VERITAS Volume Manager, VERITAS File System, and VERITAS NetBackup are trademarks or registered trademarks of Symantec Corporation or its affiliates in the United States and other countries. Linux is a trademark or registered trademark of Mr. Linus Torvalds in the United States and other countries. Other product names and company names, etc. are trademarks or registered trademarks of the associated companies. 7. This product includes the OSSs below. For details on the licenses, refer to Appendix A in this manual. Apache log4j ( flex ( OpenSSL ( 8. In this document, the capacity is calculated based on units of 1024 (for example 1 KB = 1024 bytes) unless otherwise specified. 9. In this document, matters to which careful attention needs to be paid will be described as follows: Be sure to observe the instructions. If the indications are ignored and the system is improperly operated, settings which have been already made might be affected. Type of Indication Type Description Describes contents which require users to pay special attention for operation. 1st Edition in November th Edition in October 2017

6 Contents Chapter 1 Overview of Disaster Recovery What is Disaster Recovery? Disaster Recovery Use Format Copy Mode Options for Disaster Recovery Systems Freshness (Up to Date) of Backup Data and Use Format Layout and Configuration of Disaster Recovery System Placement of Sites Selection of Configuration and Lines Chapter 2 Disaster Recovery Functions Atomic Group (ATgroup) ATgroup Configuration Display and Setting Functions ATgroup State Display Function ATgroup Replication Control Functions Copy of Order Guarantee Function Atomic-break Function Semi-Synchronous Copy Mode of Order Guarantee and Atomic Break Automatic Resynchronization Simultaneous Replication of Serially Configured Pair Remote Pair Operations RDR Quick Sync Swap Function for RDR Pair Operation of Partitioning Function Chapter 3 Install Software and Build Environment System Configuration Hardware Software Placement of Management Servers Placement of Control Volume Software Installation Installation of RemoteDataReplication/DisasterRecovery ReplicationControl/DisasterRecovery Installation and Environment Setting Building an ATgroup Pairs Registered to an ATgroup Steps for Building an ATgroup Automatic Resynchronization Operation Operating Procedure of Automatic Resynchronization (Snapshot type) Operating Procedure of Automatic Resynchronization (DDR type) Chapter 4 Software Operation Methods Storage Manager s ATgroup Function Operation Method (GUI) ATgroup Information Screen Displaying the ATgroup Configuration and State Creating an ATgroup Deleting an ATgroup Creating/Deleting an ATgroup Volume Renaming an ATgroup Changing the Allowed Response Time of MV Replicating an ATgroup Separating an ATgroup Restoring an ATgroup Forced Separation of an ATgroup Recovery from Fault Separation Forced Deletion of an ATgroup Outputting the ATgroup Information List in CSV format Saving an ATgroup and Registered Pair Information Setting the ATgroup Resynchronize Attribute i

7 Resetting the ATgroup Resynchronize Attribute ReplicationControl/DisasterRecovery Command Operation Methods (CLI) Replicating an ATgroup (ismrc_replicate) Separating an ATgroup (ismrc_separate) Restoring an ATgroup (ismrc_restore) Recovery from Fault Separation (ismrc_change) Waiting for the ATgroup Sync State to Change (ismrc_wait) Displaying the ATgroup Copy State (ismrc_query) Displaying the ATgroup List (ismrc_sense) Building an ATgroup (ismrc_atg) Remote Operation Commands Chapter 5 Operation Examples Backup Line Fault Recovery Use of RV when Disaster Occurs Copy Back Mode from RV to MV Restoration for Copy Back Mode Copy Back Mode by Pair Configuration Substitutions Copy Back Mode through Swap for Single RDR Pair Notes on Copy Back Mode Procedure through Swap for Single RDR Pair Recovery Procedure at Pair Swap Failure Copy Back Mode through Pair Swap in RDR/DR Configuration Copy Back Mode Procedure through Swap for RDR Pair Recovery Procedure at Pair Swap Failure Chapter 6 Notes Specifications of ATgroup Notes on Building an Environment Notes for Operations Appendix A License A.1 Apache log4j License A.2 Flex License A.3 OpenSSL License Index ii

8 This page is intentionally left blank. iii

9 Chapter 1 Overview of Disaster Recovery Chapter 1 Overview of Disaster Recovery Given the increasingly important role of information systems in the context of business activities, disaster recovery support for such information systems are needed more than ever. RemoteDataReplication and DisasterRecovery provide functions that lay the foundation for disaster recovery. This chapter gives a general description of disaster recovery systems that use RemoteDataReplication and DisasterRecovery software. 1.1 What is Disaster Recovery? Disaster recovery is a solution that enables businesses to recover operations after a system fault has occurred due to a disaster, accident, etc. The types of disasters that disaster recovery is designed to anticipate include natural disasters such as earthquakes and floods as well as fires and other man-made disasters caused by human error. Figure 1-1 Disaster Recovery When constructing a disaster recovery system, the anticipated disasters and the system repair level differ according to the system s environment and requirements. When the scope of effects from anticipated disasters goes beyond a particular building, data must be stored at remote locations to enable configuration of a system that will take over the operations of the affected system. Typically, disaster recovery systems feature copying 1

10 Chapter 1 Overview of Disaster Recovery of data to remote sites as an essential component in anticipation of disasters whose effects transcend one building. RemoteDataReplication and DisasterRecovery provide various methods for copying data to remote sites. Table 1-1 Anticipated Disasters Scope of Disaster Wide area (in or between states, provinces, etc.) Specified districts (within a city) Specified building Specified system Cause of Disaster Earthquake, etc. Flood, power outage, etc. Fire, water damage, terrorist attacks, etc. Hardware fault Software fault Human error Virus, etc. Two indicators are used to determine the repair level of a system: the recovery point objective (RPO) and the recovery time objective (RTO). The RPO indicates from which point in time data will be recovered. In smaller systems, the RPO can recover data from a point just before the disaster occurred. A smaller RPO value is used when creating a disaster recovery system in which disaster-related data loss must be minimized. The RTO indicates the amount of time required to recover and resume operations. When this RTO value is low, recovery of operations following a disaster occurs more quickly, thus reducing the down time. A smaller RTO value is used when creating a disaster recovery system in which down time must be minimized. Generally, setting smaller RPO and RTO values entails higher costs and makes both the creation and the administration of the system more difficult. Consequently, both the RPO and RTO are determined based on the required repair level, then the type of disaster recovery system to create is selected in order to suit these RPO and RTO values. RemoteDataReplication and DisasterRecovery are software that supports the configuration of these various types of disaster recovery systems. 2

11 Chapter 1 Overview of Disaster Recovery RPO Now Several hours ago System in which data loss must be minimized Mission critical business system Yesterday System in which down time is allowed High repair level, high degree of difficulty, high costs Last month Client Next week Tomorrow Several hours later Mail server No down time RTO Recovery Point Objective: This indicates point in time from which system can be recovered Recovery Time Objective: This indicates amount of time until system operations can be resumed Figure 1-2 Relations among Recovery Point Objective, Recovery Time Objective, and Repair Level 3

12 Chapter 1 Overview of Disaster Recovery 1.2 Disaster Recovery Use Format In disaster recovery systems, the use format is set by selecting options such as the repair point (the point in time from which data will be recovered), the maximum recovery time before data can be used again, the method used to copy data to remote sites (copy mode), and the type of synchronization (replication or separation) used while the system is operating. Copy mode options that can be selected for a disaster recovery system are described below, along with the freshness (up to date) of backup data and use formats Copy Mode Options for Disaster Recovery Systems The disaster recovery system s copy mode options are synchronous mode, semi synchronous copy mode of order guarantee, and background copy mode. Different use formats can be implemented depending on the copy mode. The features of each copy mode are described below. 4

13 Chapter 1 Overview of Disaster Recovery Synchronous mode In this mode, business I/O operations are fully synchronized and copied to a remote site. Because copying is always synchronized with business I/O operations, the data being copied is always consistent and up to date. Completion of business I/O operations is held pending until the copy operation is completed, so the response to business I/O operations depends on the line speed. Main site Backup site Application server (iv) (i) (ii) (iii) Synchronization Master Replication Disk array Disk array Processing sequence Description of processing Processing request source or response source Processing request destination or response destination (i) Write request to master volume Application server Main site s disk array (ii) Write request to replication volume Main site s disk array Backup site s disk array (iii) Write request completion response to replication volume Backup site s disk array Main site s disk array (iv) Write request completion response to master volume Main site s disk array Application server Figure 1-3 Overview of Operations in Synchronous Mode 5

14 Chapter 1 Overview of Disaster Recovery Semi-synchronous copy mode of order guarantee Data is copied to a remote site without being synchronized with business I/O operations. Since data is copied to business I/O operations via the guaranteed sequence method, the data being copied is consistent. When a high-speed line is used, this has almost no impact on business I/O responses. This mode uses a semi-synchronous buffer. Therefore, a semi-synchronous buffer must be set for both the main site and backup site in advance. Main site Backup site Application server (ii) (i) (iii) (1) (2) (iv) (1) (2) (3) Master Semi-synchronous guaranteed sequence (3) Replication Disk array Disk array Processing sequence Description of processing Processing request source or response source Processing request destination or response destination (i) Write request to master volume Application server Main site s disk array (ii) Write request completion response to master volume Main site s disk array Application server (iii) Write request to replication volume according to sequence of writing to master volume (Sequence is (1) (1), (2) (2), (3) (3) ) Main site s disk array Backup site s disk array (iv) Write request completion response according to sequence of writing to replication volume (Sequence is (1) (1), (2) (2), (3) (3)) Backup site s disk array Main site s disk array Figure 1-4 Overview of Operations in Semi-synchronous Copy Mode of Order Guarantee 6

15 Chapter 1 Overview of Disaster Recovery Background copy mode Copying is asynchronous in relation to business I/O. Since copying to a remote site is done in batches, there is no consistency in the data being copied. This has no impact on business I/O responses. Main site Backup site Application server (ii) (i) (iii) (iv) Background Master Replication Disk array Disk array Processing sequence Description of processing Processing request source or response source Processing request destination or response destination (i) Write request to master volume Application server Main site s disk array (ii) Write request completion response to master volume Main site s disk array Application server (iii) Write request to replication volume Main site s disk array Backup site s disk array (iv) Write request completion response to replication volume Backup site s disk array Main site s disk array Figure 1-5 Overview of Operations in Background Copy Mode Summary of copy mode features The following is a summary of each copy mode s features. Copy Mode Table 1-2 Summary of Copy Mode Features Consistency of Data Being Copied Freshness of Data Business I/O Response Synchronous mode Consistent Same as master Depends on line speed Semi-synchronous copy mode of order guarantee Consistent Almost same as master Does not depend on line speed if high-speed line is used Background copy mode Not consistent Not the latest data Does not depend on line speed 7

16 Chapter 1 Overview of Disaster Recovery Freshness (Up to Date) of Backup Data and Use Format The copy mode and sync state are selected based on how fresh (up to date) the backup data needs to be when used for recovery following a disaster, and this also helps determine the use format for operations. For example, if it is essential to have the latest data for recovery after a disaster, a high-speed line should be used to continually replicate in either synchronous mode or [semi synchronous copy mode of order guarantee], and the use format that copies consistent data with a guaranteed write order to a remote site should be selected. For cases where it is acceptable to recover using backup data after a disaster, the background copy mode that does not affect business I/O operations should be used and the use format that periodically copies data to a remote site via pause point backup should be selected. For cases where it is acceptable to simply store backup data to a remote site, the use format that periodically executes pause point backup to the remote site without placing a server at the backup site should be selected. Freshness of Backup Data (Up to Date) Table 1-3 Freshness (Up to Date) of Backup Data and Use Format Copy Mode Use Format Synchronous Mode Repair Point Time Until Data Recovery Recovery from replication during replicating Synchronous mode Semi synchronous copy mode of order guarantee Continual replication using high-speed line Continual replication using high-speed line Same as master Short (*1) Almost same as master Short (*1) Recovery from backup data Background copy mode Data is backed up at periodic pause points Backup point Long (*2) *1: Perform rollfoward, if necessary, when database has been configured. *2: Restoration from backup media is required. Rollback from log is required when database has been configured. 8

17 Chapter 1 Overview of Disaster Recovery 1.3 Layout and Configuration of Disaster Recovery System Points to consider when placing and configuring a disaster recovery system are described below Placement of Sites In a disaster recovery system, the sites are placed at physically dispersed locations so that faults will not occur simultaneously at both the main site and backup site in the wake of any anticipated disaster. Anticipated Scope of Disaster Table 1-4 Estimation of Distances between Sites (Examples) Wide area (in or between states, provinces, etc.) Specified districts (within a city) Specified building Cause of Disaster Earthquake, etc. Flood, power outage, etc. Fire, water damage, terrorist attacks, etc. Scope of Disaster s Effects Halts economic activities in entire area Halts economic activities in specified area Halts economic activities in specified companies Distance between Sites Several hundred kilometers Several dozen kilometers Several hundred meters to several kilometers 9

18 Chapter 1 Overview of Disaster Recovery Selection of Configuration and Lines To make sure that recovery is always possible in a disaster recovery system, hardware must be configured so that there are no single fault points whereby a single fault can stop the entire system s operations. This is why an independent, redundant configuration of servers, access paths, and power supplies are recommended. In a network that has a disaster recovery system, maintaining the quality of communication lines and redundancy is required. In particular, an independent and redundant configuration is recommended for the lines between the main site and backup site so that data can always be transferred to the backup site, even when a line fault has occurred. In addition, the line speed that is required varies according to the disaster recovery configuration and the amount of data updating that accompanies business tasks. Consequently, the required line speed must be estimated in advance. Terminal Terminal Terminal WAN Network equipment Network equipment Server Server Network equipment Network equipment Disk array Main site Figure 1-6 Network Configuration Example Disk array Backup site 10

19 Chapter 2 Disaster Recovery Functions Chapter 2 Disaster Recovery Functions This chapter describes the disaster recovery functions provided by RemoteDataReplication and RemoteDataReplication/DisasterRecovery. 2.1 Atomic Group (ATgroup) Disaster recovery functions are able to create a group of volume pairs which require consistency. A group that has such consistency is called an Atomic Group (hereafter abbreviated as ATgroup). When several RDR pairs are registered to an ATgroup, data can be copied from the master volume (Master Volume: MV) to a replication volume (Replication Volume: RV) in the order in which each volume is updated. Preserving the order in which each volume is updated enables the consistency of data to be maintained in the RV to be used for recovery. For example, when using a database for business tasks, consistency must be maintained among the volumes that comprise the database. Accordingly, all pairs in the volumes that comprise a database are registered to ATgroups. When an ATgroup is created for each business task and each pair is registered, data consistency can be maintained in the RV for each business task. In addition since ATgroups can be used to manage the state of individual groups, it enables state displays and operations on a per-group basis, which means that the usability of RV data after a disaster can be determined on a per-group basis. Main site Backup site MV1 RV1 MV2 RV2 MV3 RV3 Disk array Disk array ATgroup * Consistency can be maintained among RV1, RV2, and RV3 that are included in an ATgroup. Figure 2-1 ATgroup 11

20 Chapter 2 Disaster Recovery Functions The following describes, in relation to ATgroups, (1) the ATgroup configuration display and setting functions, (2) the state display function, (3) replication control functions, (4) the copy of order guarantee function, and (5) the atomic-break function. The number of ATgroups that can be created, the number of pairs that can be registered to ATgroups, and the number of ATgroups to which replication can be executed in semi synchronous copy mode of order guarantee are limited. For further description of ATgroups, refer to 6.1 Specifications of ATgroup. 12

21 Chapter 2 Disaster Recovery Functions ATgroup Configuration Display and Setting Functions The elements that configure the ATgroup can be displayed and set. Any ATgroup name can be set for an ATgroup, and these names can be used to distinguish among the ATgroups. The MV side of ATgroups in a database group is called a concentrator, and the RV side is called the distributor. The disk array on the concentrator side (MV side) is used to create an ATgroup, which is built by registering RDR pairs. Main site Backup site DB_Group1 MV1 RV1 MV2 RV2 Concentrator Distributor DB_Group2 MV3 RV3 MV4 RV4 Concentrator Distributor Disk array Disk array MV1-RV1, MV2-RV2: ATgroup name of pair included in DB_Group1 s ATgroup MV3-RV3, MV4-RV4: ATgroup name of pair included in DB_Group2 s ATgroup Figure 2-2 Building of ATgroup (Example) 13

22 Chapter 2 Disaster Recovery Functions Any pair that is registered to an ATgroup must meet the following conditions. DDR pairs cannot be registered to an ATgroup. Only RDR pairs can be registered to an ATgroup. A pair that is not the top-level pair cannot be registered to an ATgroup. Only the top-level pair can be registered. If the MV of the pair to be registered includes several RDR pairs, it cannot be registered to an ATgroup. Each RDR pair must be set as a single pair registered to an ATgroup. A registered pair in an ATgroup and the pair to be registered to an ATgroup must be separated. An RDR pair that exists among different disk arrays cannot be registered to a single ATgroup. The RDR pair must be in the same disk array. If the RV is also used as a snapshot, it cannot be registered. It is required that the RV is not used as a snapshot (except the disk array which supports the RDR/DR automatic resynchronization function). Pairs that can be registered to ATgroup are illustrated below. Main site Backup site (i) MV RV (ii) MV RV/MV RV RV Disk array Disk array [Description of pairs that can be registered to ATgroup] (i) Single RDR pairs are registered. (ii) A combination of the top-level single RDR pair and DDR pair can be registered as an RDR pair. Figure 2-3 Pairs that can be Registered to ATgroup 14

23 Chapter 2 Disaster Recovery Functions Pairs that cannot be registered to an ATgroup are illustrated below. Main site Backup site (1) Backup site (2) (i) MV RV MV (ii) RV/MV RV (iii) MV RV RV (iv) MV Replicate RV MV RV (v) MV RV (vi) MV RV/BV SV Disk array Disk array Disk array [Description of pairs that cannot be registered to an ATgroup] (i) Pair configured as a DDR pair. (ii) Pairs that is not the top-level pair. (iii) Several RDR pairs that are configured from a single MV. (iv) Pairs in an ATgroup that are not separate from pairs registered to an ATgroup. (v) Pairs in different disk arrays that are in a single ATgroup. (Each of these pairs can be registered to a separate ATgroup) (vi) RV is also used as a snapshot (except the disk array which supports the RDR/DR automatic resynchronization function). Figure 2-4 Pairs that cannot be Registered to ATgroup 15

24 Chapter 2 Disaster Recovery Functions ATgroup State Display Function The state of each pair in an ATgroup can be displayed per ATgroup. ATgroup states are displayed as ATgroup consistency states and ATgroup sync states. ATgroup consistency states The ATgroup consistency state indicates whether or not the data is consistent in all RVs included in the ATgroup. When the ATgroup consistency state is Atomic the data in the ATgroup s RVs is consistent and can be used to recover business operations. When the ATgroup consistency state is Non-atomic the RV data included in that ATgroup is not consistent data except when separation has been completed at a pause point in business operations. Thus, it is not normally used for recovery of business operations. ATgroup Consistency State Atomic Non-atomic Invalid Table 2-1 ATgroup Consistency States Description RV data is consistent as an ATgroup. RV can be accessed when ATgroup s sync state is separated. RV data is not consistent as an ATgroup. This state indicates that the ATgroup does not contain any pairs or has just been created. 16

25 Chapter 2 Disaster Recovery Functions ATgroup sync states These indicate the sync state of pairs in an ATgroup. The sync state of an ATgroup changes according to the sync state of each pair in the ATgroup. ATgroup Sync State Separated Separating (Sep/exec) Separate start (Sep/start) Fault separation (Fault) Fault recovering (Fault/recovering) Replicate synchronous (Rpl/sync) Replicating (Rpl/exec) Replicate start (Rpl/start) Replicate suspend (Rpl/suspend) Restoring (Rst/exec) Restore suspend (Rst/suspend) Invalid Table 2-2 ATgroup Sync States Description This state indicates that all pairs in the ATgroup have been separated. The state of pairs in the ATgroup is either separating or separated. In this state, separation has started for the ATgroup. This state indicates that the ATgroup contains a pair with a fault or a separated state caused by a forced operation. When the ATgroup consistency state is Atomic, RVs cannot be accessed. An operation that recovers from fault separation changes the ATgroup sync state to separated and enables RV access. This state indicates that the ATgroup is changing from fault separation to separated. This state indicates that all pairs in the ATgroup are synchronous due to replication. Pairs in the ATgroup are either being replicated or are synchronous. This state indicates that replication has started for an ATgroup. This state indicates that an abnormal suspend state has occurred for a pair in the ATgroup due to a fault while replicating. This state indicates that restoring will be or is being performed for an ATgroup. This state indicates that an abnormal suspend state has occurred for a pair in the ATgroup due to a fault while restoring. This state indicates that the ATgroup does not contain any pairs or has just been created. * The abbreviations have initial capital letters and are used to distinguish among the sync states of pairs. 17

26 Chapter 2 Disaster Recovery Functions ATgroup sync state when link fault occurs The ATgroup sync state is determined based on the state of the copy source, i.e., the concentrator side (MV side). If a link fault occurs, possibly due to a disk array fault or line fault, communication becomes disabled between the disk array on the concentrator side (MV side) and the disk array on the distributor side (RV side), so the state of the distributor side (RV side) immediately before the disaster occurred is maintained. Consequently, the ATgroup sync state of the concentrator side (MV side) does not match the ATgroup sync state of the distributor side (RV side). Therefore, the link state must be checked in order to confirm the ATgroup sync state. Line fault occurs during replicate synchronous state (Rpl/sync) Main site Backup site MV1 RV1 MV2 RV2 MV3 RV3 Disk array Disk array Access ATgroup sync state on concentrator side (MV side) State changes to fault separation (Fault) due to line fault Access ATgroup sync state on distributor side (RV side) Since state of concentrator side (MV side) cannot be obtained due to a line fault, the replicate synchronous state (Rpl/sync) before the fault occurred is maintained. Figure 2-5 ATgroup Sync State when Link Fault has Occurred 18

27 Chapter 2 Disaster Recovery Functions ATgroup Replication Control Functions Replication control for pairs registered to ATgroups is performed for the entire ATgroup. Such control operations cannot be performed for specific pairs. ATgroup replication Replication is executed for all pairs within the ATgroup. Immediately after replication of the ATgroup, the ATgroup sync state becomes replicate start (Rpl/start). When replication is being executed for all pairs within the ATgroup, the ATgroup sync state is replicating (Rpl/exec). After all pairs in the ATgroup have been copied, the ATgroup sync state changes to replicate synchronous (Rpl/sync) as soon as the activity state has been replicated and the sync state is synchronous. The ATgroup s copy mode is specified when the ATgroup is being replicated. If ATgroup replication is performed during synchronous mode or semi synchronous copy mode of order guarantee, the ATgroup consistency state becomes Atomic as soon as the ATgroup sync state becomes replicate synchronous (Rpl/sync). If ATgroup replication is performed during a background copy mode, the ATgroup consistency state stays Non-atomic if the ATgroup sync state becomes replicate synchronous (Rpl/sync). During the replicate start (Rpl/start) or replicating (Rpl/exec) state, the RV data is not consistent, regardless of the copy mode. Accordingly, before replicating the ATgroup, the data should be backed up to another volume or other operations or steps should be implemented to prevent loss of data consistency. Separation of ATgroup Pairs in the ATgroup are separated. If the ATgroup consistency state is Non-atomic, separation of the ATgroup is performed specifically for each pair in the ATgroup. As soon as separation of the ATgroup is completed, the ATgroup sync state becomes separate start (Sep/start). Once separation is being executed for all pairs in the ATgroup, the ATgroup sync state becomes separating (Sep/exec). When all pairs in the ATgroup are separate, the ATgroup sync state becomes separated. At that time, the ATgroup consistency state becomes Non-atomic. If ATgroup separation is executed when the ATgroup consistency state is Atomic, the data consistency of RVs corresponding to all pairs in the ATgroup is maintained and separation is performed for all pairs at the same time. At that point, the ATgroup s consistency state is maintained as Atomic. Immediately after ATgroup separation, separate (immediate) cannot be used, which means the immediate separation of RVs. 19

28 Chapter 2 Disaster Recovery Functions Restoration of ATgroup Restoration is performed for each pair in the ATgroup. In order to maintain the RV data, this ATgroup restoration processing is only an RV store (protect) operation that does not include any updating of RVs. When an ATgroup is being restored, each pair in the ATgroup is restored and the ATgroup sync state changes to restoring (Rst/exec). When restoration of the ATgroup is completed, the ATgroup sync state becomes separated. Accordingly, if ATgroup restoration is performed when the ATgroup consistency state is Atomic, the RV data is maintained while also maintaining Atomic as the ATgroup consistency state. ATgroup replication control and state transitions Various types of ATgroup replication control and state changes are outlined below. Non-atomic RV protect restore execution state Atomic RV protect restore execution state Restoring Restoring Separated state Restoring Restored Separated while consistent Restoring Restored Separated Separated Replicating Separated due to fault. RV access disabled Recovery from fault separation Replicating Separating Fault separation When differences between MV and RV are reflected Replicating Synchronized via sync mode or semi synchronous copy mode of order guarantee Replication state Atomic break or forced separation Separating Synchronized via background copy mode Replication state Replication state during background copy mode Replication during sync mode or semi synchronous copy mode of order guarantee. Copy mode of order guarantee to RVs. Figure 2-6 Outline of Replication Control and State Transitions 20

29 Chapter 2 Disaster Recovery Functions Copy of Order Guarantee Function When the ATgroup consistency state is Atomic, the order of updating from MVs to RVs per ATgroup is guaranteed in order to maintain RV data consistency. Data is written from concentrator (MV side) to the distributor (RV side) according to the order in which data was written to the MVs that comprise the ATgroup. This guaranteeing of the update order occurs only when the copy mode is either synchronous mode or semi synchronous copy mode of order guarantee and the ATgroup consistency state is Atomic. No such guaranteeing of the update order can occur when the copy mode is background copy mode or when the ATgroup consistency state is Non-atomic. Main site Backup site Application server Standby server (i) (ii) (iii) DB_Group1 MV1 (iii) RV1 MV2 (ii) RV2 MV3 (i) RV3 ATgroup s consistency state is Atomic Disk array Disk array * (i), (ii), and (iii) indicate the write order Figure 2-7 Copy of Order Guarantee Function 21

30 Chapter 2 Disaster Recovery Functions Atomic-break Function The atomic-break function forcibly separates all pairs in the ATgroup and forcibly separates the RVs in order to maintain the consistency of RV data when the order of copying to RV cannot be guaranteed. This function is executed when the ATgroup consistency state is Atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) but forced separation is required since the RV copy order cannot be guaranteed due to a disaster, line fault, or other fault problem. If data is written far beyond the line capacity, data copying performed in synchronization with RV is delayed. If this delay also causes substantial delay in MV writing, an atomic break is generated to stop copying to the replication volume in order to give priority to writing to MV. Execution of an atomic-break stops all copying to RVs, but the consistency of the RV data is maintained. After RVs are forcibly separated by an atomic-break, the option of enabling access to MVs can be specified as enabled or disabled during ATgroup replication. This MV Access after Atomic-break setting is described below. MV Access Continue (default) MV access can be continued even after an atomic-break has been executed. When an atomic-break is executed after a fault that does not affect business tasks (such as a line fault or RV fault), it does not affect business task access either. Since updating of MVs is enabled even after an atomic-break is executed, differences occur between the MV data and RV data after an atomic-break. For example, if a disaster causes a line fault to occur, a fault may later occur in the application server, which would result in loss of data updated after the line fault occurred. When an atomic-break is executed without a link fault such as an RV fault, the ATgroup sync state becomes fault separation (Fault). When changing from this state to RV use, an operation to recover from fault separation is executed on the distributor side (RV side) to enable RV use. Also, when an atomic-break is executed following a link fault such as a line fault, the distributor side (RV side) s ATgroup sync state remains replicate synchronous (Rpl/sync), as it was before the link fault occurred. When changing from this state to RV use, forced separation is executed on the distributor side (RV side), followed by an operation to recover from fault separation in order to enable RV use. MV Access Stop When an atomic-break is executed, MV access is stopped. In fact, execution of an atomic-break stops all business access, even when the fault that has occurred is a line fault or RV-related fault that does not affect business tasks. Since updating of MVs is not possible after an atomic-break, no differences will occur between MV data and RV data after the atomic-break is executed. For example, even if a disaster causes a line fault to occur first and an application server fault to occur later, no data loss will occur as long as the copy 22

31 Chapter 2 Disaster Recovery Functions mode is synchronous mode. To enable RV use after executing an atomic-break, a forced separation is first performed on the distributor side (RV side) to change to fault separation (Fault). Next, an operation to recover from fault separation is performed to enable RV use. When an MV access stop is specified, business tasks may be stopped without prior warning, so MV access stop should not be used ordinarily. 23

32 Chapter 2 Disaster Recovery Functions Semi-Synchronous Copy Mode of Order Guarantee and Atomic Break Semi-synchronous copy mode of order guarantee temporarily retains data in a buffer to copy write data to the master volume to a replication volume. However, if a large amount of data are written beyond the line capacity in a short time, free space in the buffer may become insufficient because disaster recovery cannot catch up with buffer data copying to the replication volume. In this case, even in semi-synchronous copy mode of order guarantee, data write to the subsequent master volume is delayed because disaster recovery waits for buffer vacancy. If this write delay to MV continues for a given length of time, an atomic break occurs to stop data copying to the replication volume so that priority is given to data write to the master volume. Some disk array models can adjust maintenance of replication volume copying and write priority to the master volume by setting the delay time (referred to as an allowed response time of MV) for judging this atomic break occurrence per ATgroup in semi-synchronous copy mode of order guarantee. To make this adjustment, use the performance monitoring function that is an option. Environment design 1. If MV write capacity is greater than line transfer capacity at the business peak of copying, a buffer write overflow occurs to delay the write response because the transfer capacity is exceeded. To avoid this delay, it is prerequisite to estimate the necessary line transfer capacity and prepare lines. 2. Determine the time (18 seconds or less) allowed as the delay of the MV write I/O response time from the business requirements. Note, however, that if a value less than 10 seconds is specified as the allowance time, an atomic break may be determined to have occurred even if a failure occurred on one side of a link is recovered by switching the link.. 3. Even if the MV write IO response is set to 18 seconds and a failover succeeded in case of a path failure, a fault separation might occur if the IO load is high (buffer utilization is high). If a fault separation occurred, the operation can be continued by using the automatic resynchronization described in the next section. Operation evaluation At the evaluation stage, perform measurement and analysis to check if estimation is correct in the same environment as the production environment. 4. Set the following statistical information and threshold using the performance monitoring function: Accumulation of maximum buffer usage on the log (accumulation by default) Accumulation of buffer wait time on the log (addition as LD information, maximum value/mean value) (accumulation by default) Setting of time (1 second) during which the threshold of the buffer wait time (maximum value) is treated as MV write delay occurrence 5. Set the allowed response time of MV for atomic break occurrence. 24

33 Chapter 2 Disaster Recovery Functions 6. Check atomic break occurrence and the threshold reporting condition by performing the job simultaneously while copying data. Analysis and countermeasures 7. Check if copy processing is enough for business write from the atomic break conditions, threshold excess report, and gathered performance statistical information. If no atomic break has occurred and the buffer wait time is not exceeded, the line capacity is enough. Check and store the vacancy condition of the maximum buffer usage as the original vacancy condition. The timing of atomic break operation when a copy error occurs serves as the allowed response time of MV. If no atomic break has occurred but the buffer wait time threshold is exceeded, judge whether to continue processing as is or whether to take appropriate countermeasures for the delay. If an atomic break has occurred, appropriate countermeasures are required because the line capacity is insufficient. 8. Select a method that meets the requirements from the following methods: Method 1: Enhance the line capacity to increase the transfer capacity. Method 2: If the line capacity is insufficient only in a specific time zone, consider to avoid the MV delay by performing separate operation in that time zone. Method 3: Consider memory purchase to increase the semi-synchronous buffer size. Method 4: Review the allowance time as the MV write I/O response. Production operation 9. Even in the production operation, the monitoring objects are the same as in the settings in step 3. However, the line capacity may become insufficient due to increase in the amount of business although it is sufficient at operation start. For this reason, monitor the portent and take the countermeasures on the preceding page as required. Use the threshold of the maximum buffer wait time as the threshold of the time when a write I/O delay begins to occur due to a buffer overflow. If the buffer has not still overflowed but the buffer usage is gradually increasing as compared to statistical information for the maximum buffer usage, you can reference this statistical information to determine if it is necessary to take appropriate countermeasures for the future buffer overflow. 25

34 Chapter 2 Disaster Recovery Functions Automatic Resynchronization In the semi synchronous copy mode of order guarantee, if an atomic break occurs due to a temporary high IO load or an intermittent fault on the line, the automatic resynchronization function checks the line state and automatically restarts the replication. There are two types of the automatic resynchronization function: snapshot type and DDR type. In the case of the snapshot type, a snapshot is created in the RV data when restarting replication in order to save consistent RV data of the remote site. The created snapshot will automatically be deleted when the ATgroup enters in sync state. In the case of the DDR type, the data of RV is backed up in a DDR pair for automatic resynchronization when restarting replication in order to save consistent RV data of the remote site. The snapshot type automatic resynchronization function is available in the following disk array. M5000 The DDR type automatic resynchronization function is available in the following disk array. M510, M710, M5000, A5000 The following products are required to use the snapshot type automatic resynchronization function. RemoteDataReplication Ver3 or later RemoteDataReplication/DisasterRecovery DynamicSnapVolume The following products are required to use the DDR type automatic resynchronization function. DynamicDataReplication RemoteDataReplication Ver3 or later RemoteDataReplication/DisasterRecovery 26

35 Chapter 2 Disaster Recovery Functions Automatic resynchronization states Table 2-3 ATgroup Automatic Resynchronization States ATgroup Automatic Description Resynchronization State monitoring recovering evacuating synchronizing postprocessing failure resuming restored invalid No atomic break occurred in the ATgroup for which the automatic resynchronization function is enabled. An atomic break occurred in the ATgroup for which the automatic resynchronization function is enabled, and replication has automatically been restarted. Sub-state of the automatic recovery. Before restarting replication of the ATgroup, the RV data is being saved. In the case of the snapshot type, a snapshot is being created. In the case of the DDR type, the RV data is being replicated to drv for automatic resynchronization. Sub-state of the automatic recovery. Replication of the ATgroup has been restarted, and differential data is being replicated. Sub-state of the automatic recovery. In the case of the snapshot type, the created snapshot is being deleted after synchronization of the ATgroup is complete. In the case of the DDR type full-time replication, replication of the DDR pair for automatic resynchronization is restarted after synchronization of the ATgroup is complete. The automatic resynchronization function started because an atomic break occurred, but failed. When restarting the automatic resynchronization function of the non-recoverable ATgroup, the ATgroup temporarily enters this state. The state of the ATgroup will automatically transition from this state to the automatic recovery state (recovering). In the case of the snapshot type, when the main site is damaged in a disaster, data has been restored collectively from the SV to the RV on the backup site. The automatic resynchronization function is disabled for the ATgroup. 1. The restored state is displayed only on the distributor. 2. The restored state is displayed only for ACOS-4. 27

36 Chapter 2 Disaster Recovery Functions Automatic resynchronization state transition Figure 2-8 Outline of Automatic Resynchronization State Transition 28

37 Chapter 2 Disaster Recovery Functions Enabling the automatic resynchronization function Set an ATgroup as a target of the automatic resynchronization function. With this setting, the automatic resynchronization function starts if an atomic break occurs during replication in semi synchronous copy mode of order guarantee. In the case of the snapshot type, before enabling the automatic resynchronization function of the ATgroup, create an SV of each RV in the ATgroup. In the case of the DDR type, before enabling the automatic resynchronization function of the ATgroup, create a DDR pair for automatic resynchronization for each RV of the ATgroup. To switch the automatic resynchronization type (snapshot type and DDR type) of the ATgroup, disable the automatic resynchronization function, and then enable the automatic resynchronization function again. 1. If an atomic break occurred because ATgroup forced separate was performed on the concentrator, the automatic resynchronization function does not start. 2. If an atomic break occurred because ATgroup forced separate was performed only on the distributor, the automatic resynchronization function starts. 3. It may take about one minute that the automatic resynchronization function starts after the occurrence of an atomic break. If the following conditions are satisfied after the automatic resynchronization function is enabled, automatic resynchronization is performed for the target ATgroup. 1. An atomic break occurs while replication is being performed in semi-synchronous copy mode of order guarantee. 2. The cause of the atomic break is not a forcible separation by a user. 3. Operations related to the atomic break are complete. 4. The consistency state of the ATgroup on the concentrator is Atomic. 5. The sync state of the ATgroup on the concentrator is Fault. 29

38 Chapter 2 Disaster Recovery Functions Disabling the automatic resynchronization function Exclude an ATgroup as a target of the automatic resynchronization function. With this setting, the automatic resynchronization function does not start even if an atomic break occurs. When the ATgroup automatic resynchronization state is recovering, the automatic resynchronization of the ATgroup can be forcibly disabled. In this case, however, there is a risk that a snapshot of the RV remains if using Snapshot type automatic resynchronization. After forcibly disabling the automatic resynchronization, therefore, manually delete the snapshot. Restarting the automatic resynchronization function During the automatic resynchronization processing, if the ATgroup automatic resynchronization enters failure state due to a line fault, you can restart the automatic resynchronization function after checking the line state and removing the cause of the failure. When you restart the automatic resynchronization function, it resumes processing from the point that the last processing was interrupted. 1. If the ATgroup automatic resynchronization state is failure due to a line fault, remove the cause of the failure, and then restart the automatic resynchronization function. 2. If the ATgroup enters Rpl/Suspend state, restarting the automatic resynchronization function may possibly fails. In this case, separate the ATgroup forcibly, and then restart the automatic resynchronization function. 3. If the ATgroup automatic resynchronization state is failure because the ATgroup enters Separated state by performing the operation such as separating (ismrc_separate) or state changing (ismrc_change), restarting the automatic resynchronization function fails. In this case, restart replication of the ATgroup, and then restart the automatic resynchronization function. 30

39 Chapter 2 Disaster Recovery Functions 2.2 Simultaneous Replication of Serially Configured Pair Remote data replication functions can be used to simultaneously replicate serially configured pairs. When replication is performed simultaneously, replication at the second and lower levels is enabled only when the copy mode is background copy mode. Also, replication control cannot be performed when copying is performed in different directions at the same time (such as when executing replication for the former and latter serially configured pairs), when the former and latter pair s activity state is separate, or when its sync state is separating. Main site Backup site ATgroup s sync state is replicate state or pair s sync state is synchronous (synchronous, semisynchronous, or background copy) MV1 RV1/MV2 Pair s sync state is synchronous (background copy) RV2 Disk array Disk array Combination of RDR + DDR pairs Main site Backup site (1) Backup site (2) ATgroup s sync state is replicate state or pair s sync state is synchronous (synchronous, semisynchronous, or background copy) Pair s sync state is synchronous (background copy) MV1 RV1/MV2 RV2 Disk array Disk array Disk array Combination of RDR + RDR pairs Figure 2-9 Example of Replication Implemented for Serially Configured Pairs 31

40 Chapter 2 Disaster Recovery Functions 2.3 Remote Pair Operations Remote data replication functions are able to control replication from the main site to a DDR pair at the backup site. They are also able to control replication from an RDR pair at the backup site to a DDR pair at the main site. Using these functions, backup operations using the backup site s DDR can be controlled from the main site. Main site Backup site Application server Standby server Operation instruction MV RV/MV Replicate, separate, restore Replication of serially configured DDR pairs can be controlled from MV side RV MV Replication of serially configured RDR and DDR pairs can be controlled from RV side Operation instruction Replicate, separate, restore RV/MV Disk array Replicate separate restore RV Disk array Figure 2-10 Example of Remote Control 32

41 Chapter 2 Disaster Recovery Functions 2.4 RDR Quick Sync The RDR quick sync is a function to eliminate differences from the RDR pair to establish the sync state without copying data from MV to RV when the MV and RV data in the RDR pair is already the same. For RDR operation, it is necessary to reflect all data in MV to RV through the RDR link line at startup, making the data state the same (replicate synchronous). If a low-speed line is used as the RDR link line in the configuration, it takes significant time to transfer a lot of data in batch. The RDR quick sync quickly realizes the sync state without putting load on the low-speed RDR link line by other means than the RDR link line (such as transfer using tape) when MV and RV are the same data. The RDR quick sync function is offered only for ACOS-4.only for ACOS-4. This version only supports the function to display the RDR quick sync state in the ism replication management. For details about the RDR quick sync function, refer to RDR Quick Sync in the Data Replication User s Manual (Function Guide). 33

42 Chapter 2 Disaster Recovery Functions 2.5 Swap Function for RDR Pair This function swaps the roles of MV and RV in the RDR pair. The swap function is more advantageous than unpairing and resetting the pair because it enables you to immediately switch the roles of MV and RV, avoids full copy during the replication operation after the processing, and maintains the MV data redundancy by RV. The swap function is available in the following disk arrays: D3 series, D8 series, M series, and A5000 The following product is required to use the swap function. RemoteDataReplication Ver3 or later For details on the swap function, refer to 2.10 Swap Function for RDR Pair in Data Replication User s Manual (Function Guide). To perform the swap operation for the ATgroup, perform the following operations. Unpair the RDR pair from the ATgroup and delete the empty ATgroup. Perform the RDR pair swap operation. After switching the sites roles, re-create the ATgroup and register the swapped RDR pair. 34

43 Chapter 2 Disaster Recovery Functions 2.6 Operation of Partitioning Function The partitioning function partitions resources (disk, cache memory, and port) of a disk array on a logic basis such as business to bind and manage virtual storage (VSPAR: VirtualStorage PARtition) that has independent resources. In the partitioning function, the virtual storage described above is referred to as a partition. A user who performs operation management of a specific partition is referred to as a partition user. A storage user (user who performs operation management of all disk arrays) gives the authority to handle a specific partition to a partition user. A partition user is also classified into the following three user levels as in the case of a storage user, and can operate the functions provided by performance optimization for the partitions for which authority the partition user has and can reference the provided information. L1: Allows only reference L2: Allows operation related with ATgroup replication on the operational level (copy operation) L3: Allows all operations For details on the partitioning function, refer to Partitioning User s Manual. However, note that the application range in RemoteDataReplication/DisasterRecovery is limited when you use the partitioning function for operation. In disaster recovery operation, processing is switched to the backup site when the main site fails, and operations such as failure determination and switching range over multiple systems. Therefore, disaster recovery operation must be performed by the storage user. For RDR backup, the grouped pairs can be backed up at once by using the grouping function of the ATgroup. In this case, the operation can be performed by a partition user because it is within the business system. The partition user must have the authority for all the pairs registered in the ATgroup (the authority for the partition to which MV or RV belongs) to operate the ATgroup when the partitioning function is introduced. For details on the specific conditions on each operation, refer to the following table. 35

44 Chapter 2 Disaster Recovery Functions Table 2-4 Operation Conditions for Partition User Imposed by the Partitioning Function Displaying the ATgroup configuration Operation Description Authorization Of the pairs registered in the ATgroup, those for which the partition user has the authority are displayed. The pairs for which the partition user does not have the authorization are not displayed. Creating an ATgroup The partition user can create an ATgroup. Deleting an ATgroup If the partition user has the authority for all the pairs registered in the ATgroup, the partition user can delete the ATgroup after deleting the registered pairs. * Registering a pair to an ATgroup Deleting a pair registered to an ATgroup Renaming an ATgroup Change the allowed response time of MV Replicating an ATgroup Separating an ATgroup Restoring an ATgroup Forced separation of an ATgroup Recovery from fault separation Forced deletion of an ATgroup Outputting the ATgroup information list in CSV format Saving an ATgroup and registered pair information The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition user must have the operation authority for all the pairs registered in the ATgroup. The partition use can output the ATgroup information list. Only the information on the pairs in the ATgroup for which the partition user has the authority is saved. If there are pairs for which the partition user does not have the authority, their setting information is not saved. * * * * * * * * * * * : A partition user can perform this operation without any restrictions. *: A partition user can perform this operation if certain conditions are met. 36

45 Chapter 3 Install Software and Build Environment Chapter 3 Install Software and Build Environment This chapter describes methods for software installation and environment building that are required when using disaster recovery functions. 3.1 System Configuration The hardware and software used to configure a disaster recovery system through disaster recovery functions are shown below. Main site Backup site Backup server Application server Standby server Backup server FC FC Tape library Tape library Management server Management server LAN LAN Management terminal Disk array Disk array Management terminal Figure 3-1 System Configuration 37

46 Chapter 3 Install Software and Build Environment Hardware Disk arrays These disk arrays are used for business tasks and backup. They are placed at the main site and backup site. Remote data replication functions (RemoteDataReplication) and disaster recovery functions (RemoteDataReplication/DisasterRecovery) must be installed. Application servers Application servers are the servers normally used for business tasks. They are placed at the main site. Standby servers Standby servers are available for business tasks. They are placed at a backup site. Backup servers Backup servers are used for ordinary backup operations. They are placed at both the main site and the backup sites. Management servers Management servers are used to perform ATgroup configuration settings and status monitoring. They are placed at the main site and backup sites, where they manage the local site s respective system. Management terminals These terminals are for management clients. Tape libraries These are tape drives that are used for backup operations. 38

47 Chapter 3 Install Software and Build Environment Software Storage control software This software controls disk arrays. Storage control software is included in BaseProduct. Storage Manager This software performs status monitoring of resources in disk arrays as well as configuration settings and replication management. The replication management functions include pair and ATgroup state display functions, setting functions, and replication control functions. Storage Manager is included in the BaseProduct. PerformanceMonitor This software provides functions to monitor performance when copying resources among disk arrays. PerformanceNavigator This software provides functions for the secure and quick performance analysis by displaying the performance information of the disk array in easy-to-understand graph and table of figures. AccessControl This software provides functions for setting access-enabled logical disks for individual application servers. DynamicDataReplication This software provides functions for data replication within disk arrays. RemoteDataReplication This software provides functions for data replication among disk arrays. RemoteDataReplication/DisasterRecovery This software provides disaster recovery functions such as ATgroup. RemoteDataReplication is required in order to install RemoteDataReplication/DisasterRecovery. However, RemoteDataReplication/DisasterRecovery does not support the usage in combination with RemoteDataReplication Asynchronous. ReplicationControl This software provides commands to control replication from application servers to pairs and to check pair status. 39

48 Chapter 3 Install Software and Build Environment ReplicationControl/DisasterRecovery This software provides commands to control replication from application servers to ATgroups and to check ATgroup states. The target hardware for installation of the above types of software is listed below. Disk array Hardware Application server and standby server Backup server Management server Management terminal Table 3-1 Target Hardware for Software Installation Software to be Installed Storage control software AccessControl DynamicDataReplication RemoteDataReplication RemoteDataReplication/DisasterRecovery ReplicationControl ReplicationControl/DisasterRecovery ReplicationControl Backup software Storage Manager PerformanceMonitor PerformanceNavigator ism client 40

49 Chapter 3 Install Software and Build Environment Placement of Management Servers In a disaster recovery system, a management server is placed at each site so that management tasks can be continued after a disaster has occurred. Storage Manager is installed in each site s management server, where it is set up to monitor that site s disk array. Clients connect to Storage Manager at one of these sites to determine the site s disk array status. Main site Backup site Management server Storage Manager Management server is placed at each site Management server Storage Manager Application server Standby server LAN LAN MV RV Disk array Disk array Figure 3-2 Example of Storage Manager Layout 41

50 Chapter 3 Install Software and Build Environment Placement of Control Volume In a disaster recovery system, a control volume (CV) must be placed so as to perform operations for RV in Not Ready state or replication operation at the remote site. A dedicated volume, which can always be accessed, is provided as a control volume for each disk array of all the servers operating ATgroups and pairs. A control volume has no limitation such as volume capacity, pool type, and RAID type. Therefore, bind a volume of minimum capacity to be created and assign it to a disk array. Application server Standby server Server 1 Server 2 Server 3 Server 4 ReplicationControl/ DisasterRecovery ReplicationControl/ DisasterRecovery ReplicationControl/ DisasterRecovery ReplicationControl/ DisasterRecovery Operation instruction Operation instruction Operation instruction Operation instruction CV1 CV2 CV3 CV4 MV1 RV/MV1 RV1 MV2 RV/MV2 RV2 MV3 RV/MV3 RV3 Disk array Disk array CV1: Control volume for Server 1 CV2: Control volume for Server 2 CV3: Control volume for Server 3 CV4: Control volume for Server 4 Figure 3-3 Example of Control Volume Layout 42

51 Chapter 3 Install Software and Build Environment 3.2 Software Installation The following describes the installation methods used for RemoteDataReplication/DisasterRecovery and ReplicationControl/DisasterRecovery. For description of the installation methods used for other software such as ism, ReplicationControl, and RemoteDataReplication, see the installation guide for each software product Installation of RemoteDataReplication/DisasterRecove ry (1) Operating environment Applicable hardware RemoteDataReplication/DisasterRecovery runs on disk array systems. Specifically, it operates on a disk array that includes remote data replication functions (RemoteDataReplication) and disaster recovery functions (RemoteDataReplication/DisasterRecovery). (2) Installation Before using RemoteDataReplication/DisasterRecovery, the RemoteDataReplication software must already be installed, the product version must support at least the physical disk capacity of the disk array, and licensing for the disk array must be cleared. License clearance is required in order to use RemoteDataReplication/DisasterRecovery functions ReplicationControl/DisasterRecovery Installation and Environment Setting (1) Operating Environment Operating system and linkage software For details on the OSs supporting this software and the linkage software that can be used in conjunction with this software, refer to the installation guide attached to this software. Required disk free space For details on the disk free space required to install this software, refer to the installation guide attached to this software. 43

52 Chapter 3 Install Software and Build Environment Required memory For details on the memory capacity required to use this software, refer to the installation guide attached to this software. (2) Installation For details on the installation procedure, refer to the installation guide attached to this software. (3) Uninstallation For details on the uninstallation procedure, refer to the installation guide attached to this software. (4) Update To update the ReplicationControl/DisasterRecovery software, uninstall the existing software, and then install the new software. For details on the installation and uninstallation procedure, refer to the installation guide attached to this software. (5) Software Environment Setting After the installation of the ReplicationControl/DisasterRecovery software, set the software environment. For details on the software environment setting, refer to the installation guide attached to this software. (6) Replication Operation Option Setting File Automount disabling must be set for the disaster recovery system operation. To set the automount disabling, specify NOTREADY for the UMOUNT_RVACC parameter in the [FILESYSTEMOPERATION] section of the replication operation option setting file (ini file). The following is a description example: [FILESYSTEMOPERATION] UMOUNT_RVACC=NOTREADY Also, when you use Windows Server 2008, execute the MOUNTVOL command with /N option, which is the OS standard for automount disabling, in addition to the setting above. MOUNTVOL /N For details on replication operation option setting file, refer to the ControlCommand Command Reference. Various operations on command execution can be set by using the replication operation option setting file. 44

53 Chapter 3 Install Software and Build Environment 3.3 Building an ATgroup The following describes how to build an ATgroup, which is required in order to build a disaster recovery system using disaster recovery functions. Various operations are executed by using the ism replication management functions or by using commands provided by ReplicationControl and ReplicationControl/DisasterRecovery. Design items that are required by the data replication function (such as selection of logical disk, nicknames, access control, pair configurations, and platform-specific items) must be taken into consideration along with data replication items. For description of this approach and the related settings, refer to the Data Replication User s Manual (Installation and Operation Guide) in accordance with your OS in use. 45

54 Chapter 3 Install Software and Build Environment Pairs Registered to an ATgroup In a disaster recovery system, there are no limits on the volume size, pool types, or RAID types used for business tasks. The volumes used for recovery must be copied to a backup site while maintaining consistency in each business task unit. For example, when using a database, consistency must be maintained in the volumes of database files, log files, etc., that comprise the database. Accordingly, pairs of volumes that comprise the database include volumes that have undergone software mirroring (such as by using database functions), and all of these volumes are registered to an ATgroup. An ATgroup is created for each set of business tasks, and the volume pairs that corresponding to individual business tasks are registered to this ATgroup. Main site Backup site DB_MV1 DB_RV1 DB DB_MV2 DB_RV2 DB_MV3 DB_RV3 Log LOG_MV1 LOG_MV2 LOG_RV1 LOG_RV2 Disk array Disk array Figure 3-4 ATgroup Configuration Example The number of ATgroups that can be created is limited by the number of pairs that can be registered to the ATgroups. For details on such ATgroups, refer to 6.1 Specifications of ATgroup. 46

55 Chapter 3 Install Software and Build Environment Steps for Building an ATgroup The steps for building ATgroups and configuration examples are described below. (1) Overview of ATgroup Build Steps The following are some configuration examples related to the steps for building ATgroups. First, build an ATgroup comprising two disk arrays whose names are LOCAL_ARRAY and REMOTE_ARRAY. Next, register two pairs of logical disks, one named DB_MV and DB_RV and other named LOG_MV and LOG_RV, to the ATgroup named DB_Group. Access controls are set for the respective volumes, so that only MVs in the ATgroup can be access from an application server, only RVs in the ATgroup can be accessed by the standby server, and only the ATgroup s DDR RVs at each site can be accessed from the backup server. Main site Backup site Application server Standby server Backup server Backup server Tape Tape DB_Group DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array Figure 3-5 Configuration Example for Steps in Building ATgroup 47

56 Chapter 3 Install Software and Build Environment (2) Flow of ATgroup Build Steps [Operations at main site] [Operations at backup site] Step 1. Create volume list (application server and backup server) Step 2. Build DDR pair (Main site) Step 3. Set up volumes (application server and backup server) Step 4. Create volume list (application server and backup server) Step 5. Create RDR pair (application server) Step 6. Create ATgroup (application server) Step 7. Register pairs to ATgroup (application server) Step 8. Replicate ATgroup (application server) Step 9. Unmount MV (application server) Step 10. Separate ATgroups (application server) Step 11a. Mount MV in ATgroup (application server) Step 11b. Mount RV in ATgroup (Standby server) Step 12. Create volume list (Standby server and backup server) Step 13. Build DDR pairs (Backup site) Step 14. Set up volumes (Standby server and backup server) * This flow of steps and the operation steps described below are just examples. Be sure to perform operations and steps that are suited to the actual environment. Figure 3-6 Flow of ATgroup Build Steps 48

57 Chapter 3 Install Software and Build Environment (3) Operation Steps Step 1. Create a volume list (application server and main site s backup server) To set up volumes at the main site, create a volume list on the application server and the main site s backup server. If these servers that are used to create the volume list are running ism on Linux, ism must be paused first. (Command example for Windows) ismvollist -cr (Command example for Linux) ismvollist -r * It is not necessary to create the volume list on main site s backup server with this step for the operation using the GPT disk. Step 2. Build DDR pairs (main site) Next, build DDR pairs. For description of the steps in building DDR pairs, refer to the Data Replication User s Manual (Installation and Operation Guide) in accordance with your OS in use. The steps described below assume that DDR pairs have been separated, so be sure to separate the DDR pairs beforehand. Step 3. Set up volumes (application server and main site s backup server) Set up volumes using the application server and main site s backup server. For description of the steps in setting up volumes, refer to the Data Replication User s Manual (Installation and Operation Guide) in accordance with your OS in use. Step 4. Create volume list (application server and main site s backup server) First connect the main site s disk array to the backup site s disk array, and then create a volume list using the application server and the main site s backup server. If these servers that are used to create the volume list are running ism on Linux, ism must be paused first. This step is not necessary if the main site s disk array and backup site s disk array were already connected at Step 1. The main site s disk array and backup site s disk array must be connected for the following steps. (Command example for Windows) ismvollist -cr (Command example for Linux) ismvollist -r 49

58 Chapter 3 Install Software and Build Environment Step 5. Create RDR pairs (application server) Create RDR pairs to be registered to an ATgroup. (Command example) ismrc_pair -pair -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_pair -pair -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld or ismrc_pair -pair -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv In addition, they can be created using the main site s ism software. Figure 3-7 Example of Using ism to Create RDR Pairs 50

59 Chapter 3 Install Software and Build Environment Step 6. Create ATgroup (application server) Create an ATgroup. (Command example) ismrc_atg -create -atg DB_Group -arrayname LOCAL_ARRAY -linkarrayname REMOTE_ARRAY In addition, it can be created using the main site s ism software. Figure 3-8 Example of Using ism to Create ATgroup The number of ATgroups that can be created is limited by the number of volume pairs that can be registered to the ATgroups. For details on such ATgroups, refer to 6.1 Specifications of ATgroup. 51

60 Chapter 3 Install Software and Build Environment Step 7. Register pairs to ATgroup (application server) Register RDR pairs to a created ATgroup. (Command example) ismrc_atg -add -atg DB_Group -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_atg -add -atg DB_Group -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld or ismrc_atg -add -atg DB_Group -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv Pairs can also be registered via the main site s ism software. Figure 3-9 Example of Pair Registration to ATgroup Using ism The number of ATgroups that can be created is limited by the number of volume pairs that can be registered to the ATgroups. For details on such ATgroups, refer to 6.1 Specifications of ATgroup. 52

61 Chapter 3 Install Software and Build Environment Step 8. Replicate ATgroup (application server) Replicate an ATgroup to reflect differences. (Command example) ismrc_replicate -atg DB_Group -wait This replication can also be performed using ism. Figure 3-10 Example of ATgroup Replication Using ism 53

62 Chapter 3 Install Software and Build Environment Step 9. Unmount MV in ATgroup (application server) (Windows) To copy pause point data to an RV in an ATgroup, first stop the current service that is accessing the target volume, then flush and unmount the files in the ATgroup s MV. (i) Stop the service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: ismrc_umount -drv F: (Linux) To copy pause point data to an RV in an ATgroup, first temporarily unmount the MV volume in the ATgroup if it is already mounted. To unmount, unmount the file system, then set the volume group as inactive or deport the disk group. (i) Unmount the file system. (Command example) umount /db umount /log (ii) Set the volume group as inactive or deport the disk group. If LVM is being used, the volume group is set as inactive. If VxVM is being used, deport the disk group. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 54

63 Chapter 3 Install Software and Build Environment Step 10. Separate ATgroup (application server) Once the ATgroup sync state is replicate/synchronous (Rpl/sync), separate the ATgroup, then separate the RVs in the ATgroup. (Command example) ismrc_separate -atg DB_Group -wait This can also be done using ism. Figure 3-11 Example of ATgroup Separation Using ism Step 11. Mount volume (MV or RV) in ATgroup (application server and standby server) (Windows) Mount a volume (MV or RV) in an ATgroup on an application server and a standby server. On an application server, use the ismrc_mount command to mount. Afterward, restart the service that was temporarily stopped for unmounting. On a standby server, start the Windows Computer Management utility to assign a drive letter to the disk. Example of mounting on an application server (i) Mount the file system. (Command example) ismrc_mount -drv E: ismrc_mount -drv F: 55

64 Chapter 3 Install Software and Build Environment (ii) Restart the service. (Command example) NET START ESMCommonService Example of mounting on a standby server Start the Computer Management utility and assign a drive letter to the disk. Even if a drive letter has already been assigned on the standby server, there is a chance that the drive letter may have been changed when building the environment, so be sure to check the assigned drive letter. (Linux) On an application server and standby server, activate the volume group for a volume (MV or RV) in an ATgroup, or import and mount the disk group. When using LVM, only activation of the volume group can be performed. When using VxVM on a standby server, use the -C option to import. Example of mounting on an application server (i) Activate the volume group or import the disk group. If using LVM, activate the volume group. If using VxVM, import the disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdg import dg01 vxdg import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Mount the file system. (Example of command when using LVM and mounting a VxFS file system) mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Example of command when using VxVM and mounting a VxFS file system) mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log 56

65 Chapter 3 Install Software and Build Environment Example of mounting on a standby server (i) Activate the volume group or import the disk group. When using LVM, only activation of the volume group can be performed. When using VxVM, use the -C option to clear and import the disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall On Linux servers, to use partition configuration for the first time on the RV side after creating, updating, and replicating a partition, it is necessary to restart the server to make the OS recognize the RV partition information. (ii) Mount the file system. (Example of command when using LVM and mounting a VxFS file system) mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Example of command when using VxVM and mounting a VxFS file system) mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log When using RemoteCluster, MV and RV in an ATgroup are operated in the same cluster. Therefore, changing the volume group ID at a standby server is not executed. For environment creation by RemoteCluster, refer to the RemoteCluster with MC/ServiceGuard User s Manual. 57

66 Chapter 3 Install Software and Build Environment Step 12. Creation of Volume List (Standby Server and Backup Site s Backup Server) Create a volume list using a standby server and the backup site s backup server. If these servers that are used to create the volume list are running ism on Linux, ism must be paused first. (Command example for Windows) ismvollist -cr (Command example for Linux) ismvollist -r Step 13. Build DDR pairs (backup site) Next, build DDR pairs. For description of the steps in building DDR pairs, refer to the Data Replication User s Manual (Installation and Operation Guide) in accordance with your OS in use. The steps described below assume that DDR pairs have been separated, so be sure to separate the DDR pairs beforehand. Step 14. Set up volumes (standby server and backup site s backup server) Set up volumes using the standby server and backup site s backup server. For description of the steps in setting up volumes, refer to the Data Replication User s Manual (Installation and Operation Guide) in accordance with your OS in use. When using LVM on Linux servers, after the ATgroup has been built, the cluster settings, etc., have been made for the various sites, and setup for starting business tasks has been completed, be sure to fetch and store the volume group setting information on each site s server. This volume group setting information should be backed up once setup for starting business tasks has been completed for the various servers. In some cases, the volume group setting information is used for fault repair. (Example of commands to fetch volume group setting information file) (Command used for application server) vgcfgbackup /dev/vg01 vgcfgbackup /dev/vg02 (Command used for standby server) vgcfgbackup /dev/vg01 vgcfgbackup /dev/vg02 58

67 Chapter 3 Install Software and Build Environment 3.4 Automatic Resynchronization Operation This section describes automatic resynchronization operations Operating Procedure of Automatic Resynchronization (Snapshot type) (1) Operation Flow of Automatic Resynchronization (Snapshot type) [Operations at Main Site] [Operations at Backup Site] Step 1. Build an ATgroup (main site and backup site) Step 2. Bind a snapshot reserve area (SRA) (backup site) Step 3. Add snapshot generations to RV in ATgroup (backup site) Step 4. Enable automatic resynchronization (snapshot type) of ATgroups (main site) * This flow of steps and the operation steps described below are just examples. Be sure to perform operations and steps that are suited to the actual environment. Figure 3-12 Operation Flow of Automatic Resynchronization (Snapshot type) Notes on using the snapshot type: 1. If the capacity of an SRA storing differences of snapshots is insufficient, there is a risk that snapshots are automatically deleted. Therefore, set the SRA capacity equal to or more than the total capacity of RVs. 2. It is recommended that an SRA be prepared for each ATgroup. 3. The number of generations to be added to an RV must be one. 4. Because the BV (RV) used by the automatic resynchronization function does not require a LV, do not bind any LV. 59

68 Chapter 3 Install Software and Build Environment (2) Procedure Step 1. Build an ATgroup (main site and backup site) Build an ATgroup. For more information about how to build an ATgroup, refer to Steps for Building an ATgroup. Step 2. Bind a snapshot reserve area (SRA) (backup site) Bind a snapshot reserve area (SRA). For the command to create a snapshot reserve area (SRA), refer to ismcfg srabind in Part IV Reference in the Storage Manager Command Reference. A snapshot reserve area (SRA) can also be bound by using ism on the main site. For more information, refer to Binding the Snapshot Reserve Area in the Snapshot User s Manual (Function Guide). Step 3. Add snapshot generations to RV in ATgroup (backup site) Add snapshot generations to an RV in ATgroup. For the command to add snapshot generations, refer to ismcfg generationadd in Part IV Reference in the Storage Manager Command Reference. Snapshot generations can also be added by using ism on the main site. For more information, refer to Adding a Snapshot Generation in the Snapshot User s Manual (Function Guide). Step 4. Enable automatic resynchronization (snapshot type) of ATgroups (main site) Enable automatic resynchronization (snapshot type) of ATgroups. For the command to enable automatic resynchronization of ATgroups, refer to ismrc_atg -update in the ControlCommand Command Reference. Automatic resynchronization of ATgroups can also be enabled by using ism on the main site. For more information, refer to Setting the ATgroup Resynchronize Attribute. 60

69 Chapter 3 Install Software and Build Environment Operating Procedure of Automatic Resynchronization (DDR type) (1) Operation Flow of Automatic Resynchronization (DDR type) [Operations at Main Site] [Operations at Backup Site] Step 1. Build an ATgroup (main site and backup site) Step 2. Create a DDR pair for automatic resynchronization for RV in ATgroup (backup site) Step 3. Replicate a DDR pair for automatic resynchronization (backup site) Step 4. Separate a DDR pair for automatic resynchronization (backup site) (*3) Step 5. Enable automatic resynchronization (DDR type) of ATgroups (main site) *1: This flow of steps and the operation steps described below are just examples. Be sure to perform operations and steps that are suited to the actual environment. *2: There are two operation types in the DDR type: DDR full-time replication and DDR separate. In the case of DDR full-time replication, replication of DDR will be restarted after resynchronization. In the case of DDR separate, replication of DDR will not be restarted. The default operation is DDR separate. For details about how to change the operation type, refer to the description about resync_fulltime_ddr_replicate of the [replication] section in Appendix F in the Configuration Setting Tool User s Manual (GUI) for the M Series, and specify the setting in the ism Express environment definition file on the RDR/DR main site. *3: When using the DDR full-time replication, skip step 4. Figure 3-13 Operation Flow of Automatic Resynchronization (DDR type) Notes on using the DDR type: When the DDR type is used, data of RV is backed up in a DDR pair for automatic resynchronization. Therefore, if the amount of differences of the DDR pair for automatic resynchronization is large, it may take a long time to save the differences. 61

70 Chapter 3 Install Software and Build Environment (2) Procedure Step 1. Bind an ATgroup (main site and backup site) Build an ATgroup. For more information about how to build an ATgroup, refer to Steps for Building an ATgroup. Step 2. Create a DDR pair for automatic resynchronization for RV in ATgroup (backup site) Set a DDR pair for automatic resynchronization to an RV in ATgroup. For more information about the command to create a DDR pair for automatic resynchronization, refer to ismrc_pair in the ControlCommand Command Reference. A DDR pair for automatic resynchronization can also be made by using ism on the main site. For more information, refer to Pair Setting/Unpair in the Data Replication User s Manual (Function Guide). Step 3. Replicate a DDR pair for automatic resynchronization (backup site) Replicate a DDR pair for automatic resynchronization set for an RV in ATgroup. For more information about how to replicate a DDR pair, refer to ismrc_separate in the ControlCommand Command Reference. A DDR pair for automatic resynchronization can also be replicated by using ism on the main site. For more information, refer to Replicate in the Data Replication User s Manual (Function Guide). Step 4. Separate a DDR pair for automatic resynchronization (backup site) Separate a DDR pair for automatic resynchronization set for an RV in ATgroup. For more information about how to separate a DDR pair, refer to ismrc_separate in the ControlCommand Command Reference. A DDR pair for automatic resynchronization can also be separated by using ism on the main site. For more information, refer to Separate in the Data Replication User s Manual (Function Guide). Step 5. Enable automatic resynchronization (DDR type) of ATgroups (main site) Enable automatic resynchronization (DDR type) of ATgroups. For the command to enable automatic resynchronization of ATgroups, refer to ismrc_atg -update in the ControlCommand Command Reference. Automatic resynchronization of ATgroups can also be enabled by using ism on the main site. For more information, refer to Setting the ATgroup Resynchronize Attribute. 62

71 Chapter 4 Software Operation Methods Chapter 4 Software Operation Methods This chapter describes operation methods for the ism ATgroup functions and ReplicationControl/DisasterRecovery commands. 4.1 Storage Manager s ATgroup Function Operation Method (GUI) When RemoteDataReplication/DisasterRecovery is installed, the ism replication functions enable ATgroup functions to be used under the following operation privilege. For further description of operation methods for ism and the ism replication functions, start method, operation privileges, event link function, and replication function, refer to either the User s Manual in accordance with your operating environment in use, or the Data Replication User s Manual (Function Guide). In addition, the ism configuration setting function enables use of the ATgroup s batch setting function. For details on batch settings, see the Configuration Setting Tool User s Manual (GUI). Function ATgroup configuration display, status display Table 4-1 List of ism ATgroup Functions Operation Privilege Description L1 L2 L3 Display the ATgroup configuration Display the ATgroup state Described in Section Create an ATgroup Delete an ATgroup ATgroup settings Register a pair to an ATgroup - - Delete a pair registered to an ATgroup Rename an ATgroup Change the allowed response time of MV ATgroup replication operations ATgroup forced operation, fault repair Save configuration information Replicate an ATgroup Separate an ATgroup Restore an ATgroup Forced separation of an ATgroup Recovery from fault separation Forced deletion of an ATgroup Output the ATgroup information list in CSV format Save ATgroup and registered pair information The screens displayed by the ism ATgroup functions and the operation of these functions are described below. 63

72 Chapter 4 Software Operation Methods ATgroup Information Screen The ATgroup Information screen is opened when using ATgroup functions. Select the target disk array in the Replication screen, then select the ATgroup Information tab. To display the Replication screen, either select [File] [Replication] in the Main (state monitoring) screen or select [Replication] on the toolbar. Select ATgroup Information tab. Figure 4-1 ATgroup Information Screen The items on the ATgroup Information screen s menu bar for ATgroup functions are described below. Among the menus, the [View] and [Help] menus do not contain any items pertaining specifically to ATgroup functions. For details on the [View] and [Help] menus, refer to the Data Replication User s Manual (Function Guide). Outputting the ATgroup Information List in CSV Format (4.1.14) Saving an ATgroup and Registered Pair Information (4.1.15) Close the replication management screen Figure 4-2 File Menu 64

73 Chapter 4 Software Operation Methods Creating an AT group (4.1.3) Deleting an ATgroup (4.1.4) Creating/Deleting an ATgroup Volume (4.1.5) Renaming an ATgroup (4.1.6) Figure 4-3 ATgroup Settings in Operations Menu Changing the Allowed Response Time of MV (4.1.7) Setting the ATgroup Resynchronize Attribute (4.1.16) Resetting the ATgroup Resynchronize Attribute (4.1.17) Replicating an ATgroup (4.1.8) Figure 4-4 ATgroup Operations in Operations Menu Separating an ATgroup (4.1.9) Restoring an ATgroup (4.1.10) Forced separation of an ATgroup (4.1.11) Forced Deletion of an ATgroup (4.1.13) Recovery from Fault Separation (4.1.12) Figure 4-5 ATgroup Forced Operations in Operations Menu 65

74 Chapter 4 Software Operation Methods Displaying the ATgroup Configuration and State Properties in the ATgroup information screen and Replication screen can be used to check the ATgroup configuration display and state display. ATgroup information screen The state of the following items can be checked via the ATgroup information screen. ATgroup list Configuration of pairs registered to ATgroup Configuration information on ATgroup s active mode, etc. ATgroup consistency state ATgroup sync state Link state The ATgroup list, which appears near in the upper window of this screen, provides various types of information about ATgroups. When an ATgroup is selected from this list, information about pairs registered to the selected ATgroup is displayed in the lower window. Figure 4-6 ATgroup Configuration and State Display Screen [Description of items in upper window of ATgroup information screen] (i) Type This indicates the ATgroup s location (concentrator or distributor). Display items Concentrator Selected disk array s ATgroup is on the concentrator side (MV side) Distributor Selected disk array s ATgroup is on the distributor side (RV side) 66

75 Chapter 4 Software Operation Methods Icons (green) Concentrator s state is normal. (light blue) Distributor s state is normal. (green) Fault occurred at concentrator. (light blue) Fault occurred at distributor. Link fault has occurred or disk array is frozen. (ii) ATgroup Name This indicates ATgroup name. (iii) ATgroup Consistency State This indicates the ATgroup consistency state. Display items Atomic ATgroup consistency state is Atomic. Consistency of RV data is maintained in ATgroup. Non-atomic ATgroup consistency state is Non-atomic. Consistency of RV data is not maintained in ATgroup. Blank (nothing displayed) ATgroup consistency state is invalid. Either ATgroup contains no pairs or ATgroup has just been created. (iv) ATgroup Sync State This indicates the ATgroup sync state. Display items Separated ATgroup sync state is separated. All pairs included in ATgroup are separated. Sep/exec ATgroup sync state is separating (sep/exec). Pairs included in ATgroup are either separated or being separated. Sep/start ATgroup sync state is separate start (Sep/start). Separation of pairs in ATgroup has been started. Fault ATgroup sync state is fault separation (Fault). Pairs included in ATgroup have experienced a separate state by a fault or forced operation. Fault/recovering ATgroup sync state is fault recovering (Fault/recovering). Pairs included in ATgroup have experienced a separate state by separated, a fault, or forced operation. 67

76 Chapter 4 Software Operation Methods Rpl/sync ATgroup sync state is replicate synchronous (Rpl/sync). All pairs included ATgroup have been synchronized by replication. Rpl/exec ATgroup sync state is replicating (Rpl/exec). Pairs included in ATgroup have been synchronized or are being synchronized by replication. Rpl/start ATgroup sync state is replicate start (Rpl/start). Replication of pairs in ATgroup has started. Rpl/suspend ATgroup sync state is replicate suspend (Rpl/suspend). Pairs included in ATgroup have experienced a fault, causing replication to be suspended. Rst/exec ATgroup sync state is restoring (Rst/exec). Pairs included in ATgroup will be restored or are being restored. Rst/suspend ATgroup sync state is restore suspend (Rst/suspend). Pairs included in ATgroup have experienced a fault, causing restoration to be suspended. Blank (nothing displayed) ATgroup consistency state is invalid. Either ATgroup contains no pairs or ATgroup has just been created. (v) Link Disk Array Name This indicates an identification name given to the link disk array. If blank is given, it indicates that the link disk array name has not been decided. (vi) LINK State This indicates state of linkage between disk arrays. Display items Normal Link state is normal. Fault Communication fault has occurred between disk arrays or link is invalid. Freeze Data replication functions are frozen in the disk array to be connected. Blank (nothing displayed) Either ATgroup contains no pairs or ATgroup has just been created. (vii) Copy Mode This indicates ATgroup s copy mode. Display items Synchronous All replication of pairs in ATgroup is being executed in synchronous mode. Semi-synchronous All replication of pairs in ATgroup is being executed in semi synchronous copy mode 68

77 Chapter 4 Software Operation Methods of order guarantee. If the ATgroup consistency state is Atomic, copying is executed in a guaranteed order. Background Copy All replication of pairs in ATgroup is being executed in background copy mode. Blank (nothing displayed) ATgroup has not been replicated. (viii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. (ix) Difference Quantity This indicates the total amount of differential volume between the ATgroup s concentrator (MV side) and distributor (RV side). (x) Atomic-break mode This indicates MV access mode after an atomic-break. Display items MV Access Continue Access to MVs can be continued after an atomic-break is executed. MV Access Stop Access to MVs is stopped after an atomic-break is executed. Blank (nothing displayed) Pairs in ATgroup are not being replicated or copy mode is background copy mode. (xi) Concentrator Number This indicates the concentrator number in hexadecimal. (xii) Resynchronize state This indicates the ATgroup resynchronization state. Display items Monitoring The ATgroup is being monitored for any atomic break. Evacuating This is a sub-state of the automatic recovery. Before restarting replication of the ATgroup, the RV data is being saved. In the case of the snapshot type, a snapshot is being created. In the case of the DDR type, the RV data is being replicated to drv for automatic resynchronization. Synchronizing This is a sub-state of the automatic recovery. In this state, the ATgroup replication is restarted and differential data is replicated. Postprocessing This is a sub-state of the automatic recovery. In this state, In the case of the snapshot type, the created snapshot is being deleted after synchronization of the ATgroup is complete. In the case of the DDR type full-time replication, replication of the DDR 69

78 Chapter 4 Software Operation Methods pair for automatic resynchronization is restarted after synchronization of the ATgroup is complete. Failure When an atomic break occurred, automatic recovery of the ATgroup attempted, but failed. Resuming The automatic resynchronization function has been restarted for the ATgroup that failed to automatically recover, and the ATgroup automatic resynchronization function is now preparing for restart. Blank (nothing displayed) The resynchronization function is not supported. (xiii) Semi Copy Time This indicates the semi-synchronous time. Display items - The semi-synchronous time is disabled for the concentrator. Blank (nothing displayed) The semi-synchronous time is not displayed for the concentrator. [Description of items in lower part of ATgroup information screen] The lower part of this screen displays information about pairs registered to ATgroups. The pair information items that appear in this screen are the same as those in the Replication Information screen (except for ATgroup name). For details on the pair information items in the Replication Information screen, refer to the Data Replication User s Manual (Function Guide). In the case of a partition user, the volumes for which the partition user does not have the management authorization are not displayed at the bottom of the ATgroup Information screen. 70

79 Chapter 4 Software Operation Methods Replication screen s disk array properties Perform the following steps to display the Replication screen s disk array properties. Select the target disk array in the Replication screen, and then select [View] [Properties] in the menu. Select the target disk array, then right-click the mouse and select [Properties]. Use this properties window to check the properties of the ATgroup. (i) (ii) (iii) (iv) (v) Figure 4-7 Replication Screen s Disk Array Properties [Description of ATgroup information in Replication screen s disk array properties] (i) Maximum Number of Concentrator This indicates the maximum number of concentrators in an ATgroup that can be built for the target disk array. (ii) Maximum Number of Concentrator Simultaneously Executing on Semi Synchronous Copy Mode This indicates the maximum number of concentrators that can be executed with semi synchronous copy mode specified for the target disk array. 71

80 Chapter 4 Software Operation Methods (iii) Maximum Number of Volume of All ATgroup This indicates the maximum number of pairs that can be registered to ATgroup for the target disk array system. (iv) Maximum Number of Volume of One ATgroup This indicates the maximum number of ATgroups to which one pair can be registered. (v) Function of specifying Link Disk Array Name on creating an ATgroup This indicates whether to enable the function of specifying the link disk array name when registering an ATgroup. 72

81 Chapter 4 Software Operation Methods Creating an ATgroup The ATgroup Creation screen is used to create ATgroups. Displaying ATgroup Creation screen Perform either of the following steps to open the ATgroup Creation screen. Select a disk array for creating an ATgroup on the concentrator side (MV side), then select [Operation] [ATgroup] [ATgroup Setting] [ATgroup Creation] in the menu. Select a disk array for creating an ATgroup on the concentrator side (MV side), then right-click the mouse on the upper part of the ATgroup information screen and select [ATgroup Setting] [ATgroup Creation]. < Function of specifying Link Disk Array Name on creating an ATgroup is ON in the disk array properties> (display the disk array properties of the disk array to be created from the replication screen of ism to check the setting) (i) (ii) (iii) Figure 4-8 ATgroup Creation Screen [Description of ATgroup Creation screen] (i) Disk Array Name This indicates the name of the disk array for the ATgroup to be created on the concentrator side (MV side). (ii) Link Disk Array Name Specify the link disk array name of the pair to be added to the ATgroup. (iii) ATgroup Name Enter the name of the ATgroup to be created in this text box. The ATgroup name can include up to 32 characters consisting of alphanumeric characters, underbars (_) and/or forward slashes (/). After selecting the link disk array name of an ATgroup to be created and entering the name of the ATgroup, click the [Execute] button to display a confirmation dialog before creating an ATgroup. 73

82 Chapter 4 Software Operation Methods <Function of specifying Link Disk Array Name on creating an ATgroup is OFF in the disk array properties> (i) (ii) Figure 4-9 ATgroup Creation Screen [Description of ATgroup Creation screen] (i) Disk Array Name This indicates the name of the disk array for the ATgroup to be created on the concentrator side (MV side). (ii) ATgroup Name Enter the name of the ATgroup to be created in this text box. The ATgroup name can include up to 32 characters consisting of alphanumeric characters, underbars (_) and/or forward slashes (/). After entering the name of the ATgroup to be created, click the [Execute] button to display a confirmation dialog before creating the ATgroup. 74

83 Chapter 4 Software Operation Methods Handling of errors during ATgroup creation When an error occurs during ATgroup creation, the following message dialog appears. Be sure to execute handling for the respective error. Error Message Table 4-2 Handling of Errors During ATgroup Creation Description of Error and Handling [02043] Not all the link paths between the specified Disk Arrays are in the normal state. [02073] There is no available Link Disk Array. [02074] Specified Link Disk array does not support RemoteDataReplication/Disaster Recovery or the license for RemoteDataReplication/Disaster Recovery is not available. [02237] Please input an ATgroup Name. [02238] Cannot use characters other than alphanumeric characters, "/" and "_". [02239] Cannot set the default ATgroup Name. [02241] Cannot set due to monitor stopped or freezing. [02242] The same ATgroup Name exists. [02243] Cannot create it because maximum registered number is arrived. [02257] Failed to create for exceeding the maximum ATgroup of the Link Disk Array which has different specification. The link state between the disk array creating an ATgroup and disk array selected with the link disk array name is invalid. Check the link state of the disk array and restore the link state. An ATgroup cannot be registered because there is no link disk array. An ATgroup cannot be registered to the link disk array. Select the link disk array with which an ATgroup can be registered and re-execute. The ATgroup name has not been entered. Enter the ATgroup name now. A prohibited character was entered. Enter only alphanumeric characters, underbars (_), or forward slashes (/) when specifying an ATgroup name (up to 32 characters). The default ATgroup name is reserved by the system. Enter a different ATgroup name. ism has suspended monitoring of or has frozen the target disk array. If ism has suspended monitoring, restart the disk array monitoring. If it has frozen the target disk array, defreeze it. The system already includes an ATgroup with the same name. Enter a unique ATgroup name. The maximum number of ATgroups is set in the system properties. Delete unneeded ATgroups. For details on ATgroups, refer to 6.1 Specifications of ATgroup. ATgroup registration failed because ATgroups have been registered to all the concentrator numbers 0000h to 003fh. The maximum number of ATgroups that can be created between the disk arrays with different specifications (disk arrays whose maximum numbers of concentrators in the disk array properties are different) is based on the disk array with the lower specification. Namely, ATgroups are registered to the concentrator numbers 0000h to 003fh. However, when 192 ATgroups or more have already been registered between the disk arrays whose maximum numbers of concentrators in the disk array properties are the same, the maximum number of ATgroups to be additionally created decreases. Delete unnecessary ATgroups created in the concentrator numbers 0000h to 003fh of ATgroup information and re-create a new ATgroup for registration. For ATgroup specifications, refer to 6.1 Specifications of ATgroup. 75

84 Chapter 4 Software Operation Methods Conditions for creation of ATgroup (i) When creating an ATgroup, the ATgroup name you enter cannot exceed 32 characters, can include only alphanumeric characters, underbars (_), and/or forward slashes, and must be unique within the system. (ii) The disk array for the ATgroup to be created on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iii) On the disk array for the ATgroup to be created on the concentrator side (MV side), replication functions must not be in frozen state. (iv) The specified ATgroup name is not the existing one. (v) The number of ATgroups which have already been registered to the specified disk array must not exceed the maximum number. (vi) When the Function of specifying Link Disk Array Name on creating an ATgroup in the disk array properties is ON (display the disk array properties of the disk array to be created from the replication screen of ism to check the setting), the link state between the disk arrays specified with the link disk array name must be the normal state. (vii) When the Function of specifying Link Disk Array Name on creating an ATgroup in the disk array properties is ON (display the disk array properties of the disk array to be created from the replication screen of ism to check the setting), the concentrator number must be smaller than or equal to the number of specifications available for registering ATgroup of the disk array specified with the link disk array name. 76

85 Chapter 4 Software Operation Methods Deleting an ATgroup The ATgroup Deletion screen is used to delete ATgroups. Items displayed on ATgroup Deletion screen Perform either of the following steps to open the ATgroup Deletion screen. Select the concentrator for the ATgroup to be deleted in the ATgroup information screen, then select [Operation] [ATgroup] [ATgroup Setting] [ATgroup Deletion] in the menu. Select the concentrator for the ATgroup to be deleted in the ATgroup information screen, then right-click the mouse and select [ATgroup Setting] [ATgroup Deletion]. (i) (ii) Figure 4-10 ATgroup Deletion Screen [Description of ATgroup Deletion screen] (i) Disk Array Name This indicates the name of the disk array for the ATgroup to be deleted on the concentrator side (MV side). (ii)atgroup Name This indicate the name of the ATgroup to be deleted in this text box. Click the [Execute] button to open a confirmation dialog before deleting the ATgroup. 77

86 Chapter 4 Software Operation Methods Handling of errors during ATgroup deletion When an error occurs during ATgroup deletion, the following message dialog appears. Be sure to execute handling for the respective error. Error Message Table 4-3 Handling of Errors During ATgroup Deletion Description of Error and Handling [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Created volume exists) [02250] Failed in processing execution (Resync state is recovering) Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup to normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. A pair has been registered to the specified ATgroup. Delete the registered pair from the ATgroup. If the resynchronization function is disabled for the ATgroup, enable the function. Conditions for deletion of ATgroup (i) All pairs registered to the target ATgroup must be deleted beforehand and the ATgroup sync state must be invalid. (ii) The disk array for the ATgroup to be deleted on the concentrator side (MV side) must be set for monitoring by ism and must be in monitoring state. (iii) The disk array for the ATgroup to be deleted on the concentrator side (MV side) must not have frozen replication functions. 78

87 Chapter 4 Software Operation Methods Creating/Deleting an ATgroup Volume The ATgroup Volume Creation/Deletion screen is used to register pairs to or delete pairs from an ATgroup. Items displayed on ATgroup Volume Creation/Deletion screen Perform either of the following steps to open the ATgroup Volume Creation/Deletion screen. In the ATgroup information screen, select the concentrator of the ATgroup whose pair will be registered or deleted, then select [Operation] [ATgroup] [ATgroup Setting] [ATgroup Volume Creation/Deletion] in the menu. In the ATgroup information screen, select the concentrator of the ATgroup whose pair will be registered or deleted, then right-click the mouse and select [ATgroup Setting] [ATgroup Volume Creation/Deletion]. (i) (ii) (iii) (iv) (v) (vi) (vii) (viii) (ix) Figure 4-11 ATgroup Volume Creation/Deletion Screen 79

88 Chapter 4 Software Operation Methods [Description of ATgroup Volume Creation/Deletion screen] (i) Disk Array Name This indicates the name of the disk array on the concentrator side (MV side) where a pair will be added or deleted. (ii)atgroup Name This indicates the name of the ATgroup where a pair will be added or deleted. (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. (iv) Registered Volume List This is a list of currently registered pairs. When deleting a registered pair from an ATgroup, select the target pair from this list. (v) [Add] Select this to add (register) a pair. (vi) [Delete] Select this to delete a registered pair. (vii) OS Type Select a format to narrow the list of candidate volumes. (viii) Link Disk Array Name Select this to narrow the list of candidate pairs to those registered to the link disk array. If a pair has already been registered or the link disk array has been specified at ATgroup creation, the link disk array is specified automatically. (ix) Candidate Volume List This displays a list of registered candidate volumes. When registering a pair to an ATgroup, select the pair from this list. If you select a pair from the candidate volume list and then select the [Add] button, a confirmation dialog will appear before the selected pair is registered to the ATgroup. If you select a pair from the registered volume list and then select the [Delete] button, a confirmation dialog will appear before the selected registered pair is deleted from the ATgroup. In the case of a partition user, the volumes in the pairs for which the partition user does not have the management authorization are not displayed in (iv) Registered Volume List and (ix) Candidate Volume List. 80

89 Chapter 4 Software Operation Methods Handling of errors when adding or deleting pair in ATgroup If an error occurs when adding or deleting a pair in an ATgroup, the following message dialog will appear. Be sure to execute handling for the respective error. Error Message Table 4-4 Handling of Errors when Adding or Deleting Pair in ATgroup Description of Error and Handling [02058] ATgroup Volume Creation failed(number of faulted volume). [02059] ATgroup Volume Deletion failed (number of faulted volume). [02243] Cannot create it because maximum registered number is arrived. [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Link failure) [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) [02258] Specified ATgroup cannot involve the volume paired with the disk array which has different specification. Please check the Concentrator Number, and retry creating an ATgroup. [02250] Failed in processing execution (Resync state is recovering) Either the volume state is unexecutable or a disk array is abnormal. If the volume state is unexecutable, check the execution conditions of pair creation and volume state. If a disk array is abnormal, recover the disk array and restore the ATgroup to normal state. A disk array is abnormal. Check the disk array. Recover the disk array and restore the ATgroup to normal state. The maximum number of volumes that can be registered is set in the system properties. Delete unneeded volumes. For details on such ATgroups, refer to 6.1 Specifications of ATgroup. Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup to normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. After ATgroup Volume Creation/Deletion screen was displayed, the ATgroup was deleted. Check whether the specified ATgroup has already been deleted. The link between the disk arrays is invalid. Check and restore the link status. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. It indicates that the volume paired with the disk array with different specification cannot be added to the ATgroup registered to the concentrator number 0040h or more. The maximum number of ATgroups that can be created is based on the disk array with lower specifications between the disk arrays with different specifications (disk arrays whose maximum number of concentrators in the disk array properties are different). Namely, ATgroups are registered to the concentrator numbers 0000h to 003fh. Re-create an ATgroup to add a new volume to the ATgroup. For ATgroup specifications, refer to 6.1 Specifications of ATgroup. If the resynchronization function is disabled for the ATgroup, enable the function. 81

90 Chapter 4 Software Operation Methods Conditions for adding/deleting pair in ATgroup (i) Any pairs to be registered to an ATgroup must meet the following conditions. DDR pairs cannot be registered to an ATgroup. Only RDR pairs can be registered to an ATgroup. A pair that is not the top-level pair cannot be registered to an ATgroup. Only the top-level pair can be registered. If the MV of the pair to be registered includes several RDR pairs, it cannot be registered to an ATgroup. Each RDR pair must be set as a single pair registered to an ATgroup. A registered pair in an ATgroup and the pair to be registered to an ATgroup must be separated. An RDR pair that exists among different disk arrays cannot be registered to a single ATgroup. The RDR pair must be in the same disk array. Pairs cannot be registered when an RV is also used as a snapshot. It is required that an RV is not used as a snapshot (except the disk array which supports the RDR/DR automatic resynchronization function). Concentrator number must be 003fh or less when registering a pair between the disk arrays whose ATgroup specifications are different with the ATgroup. If the MV or RV of the pair to be registered is a protected volume, it cannot be registered to an ATgroup. (ii) To add a pair to an ATgroup, the ATgroup sync state must be invalid, separated, or fault separation. (iii) To delete a registered pair from an ATgroup, the ATgroup sync state must be separated, or fault separation. (iv) The disk array for the ATgroup where a pair will be added or deleted on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (v) On the disk array for the ATgroup where a pair will be added or deleted on the concentrator side (MV side), replication functions must not been in frozen state. (vi) When adding and deleting a pair, the link state between disk arrays must be normal. (vii) In the case of the partition user, the management authorization for all the pairs registered in the ATgroup to which the pair is registered or from which the pair is deleted is required. 82

91 Chapter 4 Software Operation Methods Renaming an ATgroup The ATgroup Rename screen is used to rename ATgroups. Items displayed on ATgroup Rename screen Perform either of the following steps to open the ATgroup Rename screen. In the ATgroup information screen, select the concentrator of the ATgroup where the name will be changed, then select [Operation] [ATgroup] [ATgroup Setting] [ATgroup Rename] in the menu. In the ATgroup information screen, select the concentrator of the ATgroup where the name will be changed, then right-click the mouse and select [ATgroup Setting] [ATgroup Rename]. (i) (ii) (iii) Figure 4-12 ATgroup Rename Screen [Description of ATgroup Rename screen] (i) Disk Array Name This indicates the name of the disk array on the concentrator side (MV side) where the ATgroup name will be changed. (ii) Current ATgroup Name This indicates the current ATgroup name that will be changed. (iii) New ATgroup Name Enter the new ATgroup name in this text box. The ATgroup name can include up to 32 characters consisting of alphanumeric characters, underbars (_) and/or forward slashes (/). After entering the name of the ATgroup to be created, click the [Execute] button to display a confirmation dialog before creating the ATgroup. 83

92 Chapter 4 Software Operation Methods Handling of errors when during ATgroup name revision When an error occurs during ATgroup name revision, the following message dialog appears. Be sure to execute handling for the respective error. Error Message Table 4-5 Handling of Errors During ATgroup Revision Description of Error and Handling [02043] Not all the link paths between the specified Disk Arrays are in the normal state. [02237] Please input an ATgroup Name. [02238] Cannot use characters other than alphanumeric characters, "/" and "_". [02239] Cannot set the default ATgroup Name. [02240] Demanded process had already been executed. [02241] Cannot set due to monitor stopped or freezing. [02242] The same ATgroup Name exists. [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) A fault has occurred, causing an abnormal link state for the specified disk array. Repair the fault. The ATgroup name has not been entered. Enter the ATgroup name now. A prohibited character was entered. Enter only alphanumeric characters, underbars (_), or forward slashes (/) when specifying an ATgroup name (up to 32 characters). The default ATgroup name is reserved by the system. Enter a different ATgroup name. The old ATgroup name and new ATgroup name are the same. Enter a different ATgroup name. ism has suspended monitoring of or has frozen the target disk array. If ism has suspended monitoring, restart the disk array monitoring. If it has frozen the target disk array, defreeze it. The system already includes an ATgroup with the same name. Enter a unique ATgroup name. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. Conditions for revising ATgroup name (i) When revising an ATgroup, the ATgroup name you enter cannot exceed 32 characters, can include only alphanumeric characters, underbars (_), and/or forward slashes, and must be unique within the system. (ii) The disk array for the ATgroup to be changed on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iii) The disk array for the ATgroup to be changed on the concentrator side (MV side) must not have frozen replication functions. (iv) The link state for the disk array must be normal (when a pair has been registered to ATgroup). (v) In the case of the partition user, the management authorization for all the pairs registered in the ATgroup to be changed is required. 84

93 Chapter 4 Software Operation Methods Changing the Allowed Response Time of MV The Change of the Allowed Response Time of MV screen is used to change the allowed response time of MV. Items displayed on the Change of the Allowed Response Time of MV screen Perform either of the following steps to display the Change of Allowed Response Time of MV screen: From the ATgroup Information screen, select the concentrator of the ATgroup you want to change. Then, from the displayed menu, select [Operation] [ATgroup] [ATgroup Setting] [The Change of the Allowed Response Time of MV]. From the ATgroup Information screen, select the concentrator of the ATgroup you want to change, click the right-mouse button, and select [ATgroup Setting] [The Change of the Allowed Response Time of MV] from the displayed menu. (i) (ii) (iii) Figure 4-13 The Change of the Allowed Response Time of MV Screen [Description of the Change of the Allowed Response Time of MV screen] (i) ATgroup Name This indicates the name of the disk array on the concentrator side (MV side) on which the allowed response time of MV is to be changed. (ii) The Allowed Response Time of MV This indicates the allowed response time of MV range. (iii) Change Enter a new allowed response time of MV. You can 85

94 Chapter 4 Software Operation Methods specify a new allowed response time of MV within the allowed response time of MV range. Clicking the [Change] button displays the confirmation dialog box and changes the allowed response time of MV. Handling of errors when changing an allowed response time of MV If an allowed response time of MV change error occurs, the following message dialog will appear. Take an appropriate action for each error. Error Message Table 4-6 Handling of Errors When Changing the Allowed Response Time of MV Description of Error and Handling [02043] All the link paths between the specified Disk Arrays are in the abnormal state. [02081] Specify a value within the range of <minimum-value> -<maximum-value>. [02241] Setting cannot be done because monitoring is suspended or frozen in the link destination. [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) A fault has occurred, causing an abnormal link state for the specified disk array. Repair the fault. A value that is not within the range is being used. Set an allowed response time of MV within the range of <minimum-value> to <maximum-value> seconds. ism has suspended monitoring of or has frozen the target disk array. If ism has suspended monitoring, restart the disk array monitoring. If it has frozen the target disk array, defreeze it. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. Conditions for changing the allowed response time of MV (i) The allowed response time of MV to be changed must be specified within the range. (ii) The disk array for the ATgroup to be changed on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iii) The disk array for the ATgroup to be changed on the concentrator side (MV side) must not have frozen replication functions. (iv) The link state for the disk array must be normal (when a pair has been registered to ATgroup). (v) In the case of the partition user, the management authorization for all the pairs registered in the ATgroup to be changed is required. 86

95 Chapter 4 Software Operation Methods Replicating an ATgroup The ATgroup Replicate screen is used to execute ATgroup replication. Items displayed on ATgroup Replicate screen Perform either of the following steps to open the ATgroup Replicate screen. In the ATgroup information screen, select [ATgroup], then select [Operation] [ATgroup] [ATgroup Operation] [ATgroup Replicate] in the menu. In the ATgroup information screen, select [ATgroup], then right-click the mouse and select [ATgroup Operation] [ATgroup Replicate]. (i) (ii) (iv) (iii) (v) (vi) (vii) (viii) (ix) (x) Figure 4-14 ATgroup Replicate Screen 87

96 Chapter 4 Software Operation Methods [Description of ATgroup Replicate screen] (i) ATgroup Name This indicates the name of the ATgroup to be replicated. (ii) Type This indicates the ATgroup s location (concentrator or distributor). (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. (iv) ATgroup Consistency State This indicates the ATgroup consistency state. (v) ATgroup Sync State This indicates the ATgroup sync state. (vi) Registered Volume List This section provides a list of pairs included in the ATgroup to be replicated, the status of each pair, and the replication results. The pair information items that appear in this screen are the same as those in the Replication Operations screen (except for the ATgroup name). (vii) Range This indicates the copy range for each pair during replication. Settings Only Difference Only the differences between MV and RV for each pair will be copied. All The full range for each pair will be copied. (viii) Sync Mode of Copy Data Enter the copy mode to be used during replication. Settings Synchronous This sets synchronous copy mode. When the ATgroup sync state is replicate synchronous (Rpl/sync), the ATgroup consistency state becomes Atomic. Semi-synchronous This sets semi synchronous copy mode of order guarantee. When the ATgroup sync state is replicate synchronous (Rpl/sync), the ATgroup consistency state becomes Atomic, and copying is performed in a guaranteed order. Background Copy This sets background copy mode as the copy mode. When the ATgroup sync state is replicate synchronous (Rpl/sync), the ATgroup consistency state remains Non-atomic. 88

97 Chapter 4 Software Operation Methods (ix) RV Status This indicates the restriction for access from a host until replication is complete. Settings Not Ready This disables to perform operation for a volume from a host. Read Only This enables only reading a volume from a host. * You must be careful when specifying Read Only. Refer to the description about RV access restrictions in Data Replication User s Manual (Function Guide). (x) MV Access after Atomic-break This indicates MV access after an atomic-break. Settings MV Access Continue This enables access to MVs to be continued after an atomic-break is executed. MV Access Stop This sets MV access to be stopped after an atomic-break. The MV is normally stopped after an atomic-break, so this setting is normally not necessary. Click the [Execute] button to display a confirmation dialog before replicating the ATgroup. 89

98 Chapter 4 Software Operation Methods Handling of errors during ATgroup replication The following message dialog appears if an error occurs during ATgroup replication. Be sure to execute handling for the respective error. Error Message Table 4-7 Handling of Errors During ATgroup Replication Description of Error and Handling [02250] Failed in processing execution (No created volume) [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (MV Outside ism Management) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Number of created pair is increased/reduced) [02250] Failed in processing execution (Volume status is wrong) [02250] Failed in processing execution (All Link Path Abnormal) [02250] Failed in processing execution (Maximum Number of Concentrator Simultaneously Executing on Semi Synchronous Copy Mode arrived) [02250] Failed in processing execution (There are pairs without the management in the ATgroup) No (volume) pairs have been registered to the specified ATgroup. Register a pair. Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup s normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. The specified ATgroup s concentrator side (MV side) is not being managed by ism. Use ism that manages the ATgroup s concentrator side (MV side) to execute replication. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. After the ATgroup Replicate Execution screen was displayed, the configuration of pairs registered to the ATgroup was changed. Close the ATgroup Replicate Execution screen and re-execute replication. The ATgroup s sync state is unexecutable. Change to a sync state that does enable the processing. A fault has occurred, causing an abnormal link state for the specified disk array. Repair the fault. It has reached the maximum number of concentrators executable in semi synchronous copy mode. Adjust the number to be executed within the limit. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. 90

99 Chapter 4 Software Operation Methods Conditions for replication of ATgroup (i) The ATgroup sync state must be one of the following. Separated Fault separation (Fault) Replicate suspend (Rpl/suspend) Replicate start (Rpl/start) (ii) When replication will be executed for an ATgroup whose ATgroup sync state is replicate suspend (Rpl/suspend), replication must be set before the ATgroup sync state becomes replicate suspend (Rpl/suspend). (iii) The disk array for the ATgroup to be created on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iv) The disk array for the ATgroup to be created on the concentrator side (MV side) must not have frozen replication functions. (v) The link state for the disk array must be normal. (vi) Any RVs included in the ATgroup must be unmounted from the system. (vii) When you specify semi synchronous copy mode, the maximum number of concentrators simultaneously executing in semi synchronous copy mode must not be exceeded. (viii) The update prevention state for the RVs included in the ATgroup must not be prevent. (ix) When using the asynchronous remote data replication function (RemoteDataRelication Asynchronous), the background copy mode must be specified. (x) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. (xi) All the replication reserved volumes used as MV or RV registered in the specified ATgroup are not a fault condition when using the M series and A5000. Notes on replicating an ATgroup (i) When ATgroup replication is executed, the ATgroup consistency state becomes Non-atomic, regardless of the ATgroup s copy mode. When the ATgroup s copy mode is either synchronous mode or semi synchronous copy mode of order guarantee, if the ATgroup sync state is switched to replicate synchronous (Rpl/sync), the ATgroup consistency state will change to Atomic. When the copy mode is background copy mode, if the ATgroup sync state is switched to replicate synchronous (Rpl/sync), the ATgroup consistency state will remain Non-atomic. (ii) If the ATgroup s replication is executed while in background copy mode, an atomic-break will not be executed. (iii) If MV Access Stop is set for MV Access after Atomic-break, business operations may be stopped unexpectedly, so this setting should normally be avoided. (iv) When ATgroup replication is executed for an ATgroup whose sync state is replicate suspend (Rpl/suspend), any suspended copying is restarted. The ATgroup s replication settings specified before replicate suspend (Rpl/suspend) are set to MV Access after Atomic-break, RV Status, and Copy Mode. 91

100 Chapter 4 Software Operation Methods (v) There is an upper limit for the number of ATgroups that can be replicated concurrently in semi synchronous copy mode of order guarantee. Replication cannot be executed if the upper limit is exceeded. (vi) Before executing ATgroup replication, make sure that any RVs included in the ATgroup have been unmounted. If replication is executed while one or more RVs are mounted, the system s buffer image will no longer match the disk s image, which can result in abnormal RV data or abnormalities in the system connected to the RV. 92

101 Chapter 4 Software Operation Methods Separating an ATgroup The ATgroup Separate screen is used to separate ATgroups. Items displayed on ATgroup Separate screen Perform either of the following steps to open the ATgroup Separate screen. In the ATgroup information screen, select an ATgroup, then select [Operation] [ATgroup] [ATgroup Operation] [ATgroup Separate] in the menu. In the ATgroup information screen, select an ATgroup, then right-click the mouse and select [ATgroup Operation] [ATgroup Separate]. (iv) (i) (ii) (vi) (iii) (v) (vii) (viii) Figure 4-15 ATgroup Separate Screen [Description of ATgroup Separate screen] (i) ATgroup Name (ii) Type This indicates the name of the ATgroup to be separated. This indicates the ATgroup s location (concentrator or distributor). (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. 93

102 Chapter 4 Software Operation Methods (iv) ATgroup Consistency State (v) ATgroup Sync State (vi) Copy Mode (vii) Atomic-break mode (viii) Registered Volume List This indicates the ATgroup consistency state. This indicates the ATgroup sync state. This indicates the ATgroup s copy mode. This indicates MV access after an atomic-break. This section provides a list of pairs included in the ATgroup to be separated, the status of each pair, and the separation results. The pair information items that appear in this screen are the same as those in the Replication Operations screen (except for the ATgroup name). Click the [Execute] button to display a confirmation dialog before separating the ATgroup. Handling of errors during ATgroup separation The following message dialog appears if an error occurs during ATgroup separation. Be sure to execute handling for the respective error. Error Message Table 4-8 Handling of Errors During ATgroup Separation Description of Error and Handling [02250] Failed in processing execution (No created volume) [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (MV Outside ism Management) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Number of created pair is increased/reduced) [02250] Failed in processing execution (Volume status is wrong) [02250] Failed in processing execution (All Link Path Abnormal) No (volume) pairs have been registered to the specified ATgroup. Register a pair. Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup s normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. The specified ATgroup s concentrator side (MV side) is not being managed by ism. Use ism that manages the ATgroup s concentrator side (MV side) to execute separation. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. After the ATgroup Separate screen was displayed, the configuration of pairs registered to the ATgroup was changed. Close the ATgroup Separate screen and re-execute separation. The ATgroup s sync state is unexecutable. Change to a sync state that does enable the processing. A fault has occurred, causing an abnormal link state for the specified disk array. Repair the fault. 94

103 Chapter 4 Software Operation Methods Error Message [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) Description of Error and Handling A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. Conditions for separation of ATgroup (i) The ATgroup consistency state and ATgroup sync state must be one of the following. The ATgroup consistency state is Atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) The ATgroup consistency state is Non-atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) The ATgroup consistency state is Non-atomic and the ATgroup sync state is replicating (Rpl/exec) The ATgroup consistency state is Non-atomic and the ATgroup sync state is separate start (Sep/start) (ii) The disk array for the ATgroup to be created on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iii) The disk array for the ATgroup to be created on the concentrator side (MV side) must not have frozen replication functions. (iv) The link state for the disk array must be normal. (v) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. (vi) All the replication reserved volumes used as MV or RV registered in the specified ATgroup are not a fault condition when using the M series and A5000. Notes on separating an ATgroup (i) If the ATgroup consistency state is Non-atomic when ATgroup separation is executed, separation is executed for each pair in the ATgroup. During this processing, the ATgroup consistency state remains Non-atomic. (ii) When the ATgroup sync state is separated, the access restriction on RVs included in the ATgroup is Read/write accessible. (iii) When the ATgroup consistency state is Non-atomic, pause point data can be maintained in an RV by unmounting the MV s file system or by executing using the database function s backup mode before separating the ATgroup. (iv) When separating ATgroup, Separation (immediate) which can use RV immediately after the separation start cannot be used. 95

104 Chapter 4 Software Operation Methods Restoring an ATgroup The ATgroup Restore screen is used for ATgroup restoration. Items displayed on ATgroup Restore screen Perform either of the following steps to open the ATgroup Restore screen. In the ATgroup information screen, select an ATgroup, then select [Operation] [ATgroup] [ATgroup Operation] [ATgroup Restore] in the menu. In the ATgroup information screen, select an ATgroup, then right-click the mouse and select [ATgroup Operation] [ATgroup Restore]. (i) (iv) (ii) (iii) (v) (vi) (vii) (viii) Figure 4-16 ATgroup Restore Screen [Description of ATgroup Restore screen] (i) ATgroup Name This indicates the name of the ATgroup to be restored. (ii) Type This indicates the ATgroup s location (concentrator or distributor). (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the 96

105 Chapter 4 Software Operation Methods (iv) ATgroup Consistency State (v) ATgroup Sync State (vi) Registered Volume List parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. This indicates the ATgroup consistency state. This indicates the ATgroup sync state. This section provides a list of pairs included in the ATgroup, each pair s state, and the restoration results. The pair information items that appear in this screen are the same as those in the Replication Operations screen (except for the ATgroup name). (vii) Copy Range This indicates the copy range for each pair during restoration. Settings Only Difference Only the differences between MV and RV for each pair will be copied. All The full range for each pair will be copied. (viii) RV Status This indicates the restriction for access from a host until restoration is complete. Settings Not Ready This disables to perform operation for a volume from a host. Read Only This enables only reading a volume from a host. * You must be careful when specifying Read Only. Refer to the description about RV access restrictions in Data Replication User s Manual (Function Guide). Click the [Execute] button to display a confirmation dialog before restoring the ATgroup. 97

106 Chapter 4 Software Operation Methods Handling of errors during ATgroup restoration The following message dialog appears if an error occurs during ATgroup restoration. Be sure to execute handling for the respective error. Error Message Table 4-9 Handling of Errors During ATgroup Restoration Description of Error and Handling [02250] Failed in processing execution (No created volume) [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (MV Outside ism Management) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Number of created pair is increased/reduced) [02250] Failed in processing execution (Volume status is wrong) [02250] Failed in processing execution (All Link Path Abnormal) [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) No (volume) pairs have been registered to the specified ATgroup. Register a pair. Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup s normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. The specified ATgroup s concentrator side (MV side) is not being managed by ism. Use ism that manages the ATgroup s concentrator side (MV side) to execute replication. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. After the ATgroup Restore Execution screen was displayed, the configuration of pairs registered to the ATgroup was changed. Close the ATgroup Restore Execution screen and re-execute restoration. The ATgroup s sync state is unexecutable. Change to a sync state that does enable the processing. A fault has occurred, causing an abnormal link state for the specified disk array. Repair the fault. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. Conditions for restoration of ATgroup (i) The ATgroup sync state must be one of the following. Separated Restore suspend (Rst/suspend) (ii) The disk array for the ATgroup to be created on the concentrator side (MV side) will be monitored by ism and must be in monitoring state. (iii) The disk array for the ATgroup to be created on the concentrator side (MV side) must not have frozen replication functions. 98

107 Chapter 4 Software Operation Methods (iv) The link state for the disk array must be normal. (v) The MV and RVs must be unmounted from the system. (vi) The update prevention state for the MV and RVs included in the ATgroup must not be prevent. (vii) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. (viii) All the replication reserved volumes used as MV or RV registered in the specified ATgroup are not a fault condition when using the M series and A5000. Notes on restoring an ATgroup (i) When an ATgroup is restored, it is only for an RV store (protect) operation that does not include any updating of RVs. When restoration of the ATgroup is completed, the ATgroup sync state becomes separated. (ii) When ATgroup restoration is executed, the access restriction setting for any RV included in the ATgroup becomes the specified state. When ATgroup restoration is completed and the ATgroup sync state becomes separated, the access restriction on RVs included in the ATgroup is Read/write accessible. (iii) Before executing ATgroup replication, make sure that any MVs or RVs included in the ATgroup have been unmounted. If replication is executed while one or more MVs or RVs are mounted, the system s buffer image will no longer match the disk s image, which can result in abnormal MV data or abnormalities in the system connected to the MV and RV. 99

108 Chapter 4 Software Operation Methods Forced Separation of an ATgroup The ATgroup Forced Separate screen is used to forcibly separate ATgroups. When using an RV in the ATgroup after a link fault has occurred or when the ATgroup itself is abnormal, forced separation of the ATgroup is executed on the concentrator side (MV side) and distributor side (RV side) respectively. Items displayed on ATgroup Forced Separate screen Perform either of the following steps to open the ATgroup Forced Separate screen. In the ATgroup information screen, select an ATgroup, then select [Operation] [ATgroup] [ATgroup Forced Operation] [ATgroup Forced Separate] in the menu. In the ATgroup information screen, select an ATgroup, then right-click the mouse and select [ATgroup Forced Operation] [ATgroup Forced Separate]. (i) (ii) (iv) (vi) (viii) (iii) (v) (vii) Figure 4-17 ATgroup Forced Separate Screen [Description of ATgroup Forced Separate screen] (i) ATgroup Name This indicates the name of the ATgroup to be forcibly separated. (ii) Type This indicates the ATgroup s location (concentrator or distributor). (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the 100

109 Chapter 4 Software Operation Methods (iv) ATgroup Consistency State (v) ATgroup Sync State (vi) Copy Mode (vii) Atomic-break mode (viii) Registered Volume List parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. This indicates the ATgroup consistency state. This indicates the ATgroup sync state. This indicates the ATgroup s copy mode. This indicates MV access after an atomic-break. This section provides a list of pairs included in the ATgroup to be forcibly separated, the status of each pair, and the forced separation results. The pair information items that appear in this screen are the same as those in the Replication Operations screen (except for the ATgroup name). Click the [Execute] button to display a confirmation dialog before forcibly separating the ATgroup. Handling of errors during Forced separation of ATgroup The following message dialog appears if an error occurs during Forced separation of ATgroup. Be sure to execute handling for the respective error. Error Message Table 4-10 [02250] Failed in processing execution (No created volume) [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Number of created pair is increased/reduced) [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) Handling of Errors During Forced Separation of ATgroup Description of Error and Handling No (volume) pairs have been registered to the specified ATgroup. Register a pair. Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup s normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. After the ATgroup Forced Separate screen was displayed, the configuration of pairs registered to the ATgroup was changed. Close the Forced Separation of ATgroup screen and re-execute forced separation. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. 101

110 Chapter 4 Software Operation Methods Conditions for Forced separation of ATgroup (i) The ATgroup sync state must not be invalid (Invalid), separated (Separated), fault separation (Fault), or fault recovering (Fault/recovering). (ii) The target disk array will be monitored by ism and must be in monitoring state. (iii) The target disk array must not have frozen replication functions. (iv) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. (v) All the target replication reserved volumes which are registered in the specified ATgroup are not a fault condition when using the M series and A5000. Caution point for Forced separation of ATgroup (i) Forced separation of ATgroup is performed on the concentrator side (MV side) and distributor side (RV side) respectively. 102

111 Chapter 4 Software Operation Methods Recovery from Fault Separation Recovery from fault separation is performed on the Recover from Fault separation state screen. It is executed to make the ATgroup be in the separated state, which has been in the fault separation state due to a fault or forced operation. Items displayed on Recover from Fault separation state screen Perform either of the following steps to open the Recover from Fault separation state screen. In the ATgroup information screen, select ATgroup, then select [Operation] [ATgroup] [ATgroup Forced Operation] [Recover from Fault separation state] in the menu. In the ATgroup information screen, select ATgroup, then right-click the mouse and select [ATgroup Forced Operation] [Recover from Fault separation state]. (i) (ii) (iv) (iii) (v) (vi) Figure 4-18 Recover from Fault separation state Screen [Description of Recover from Fault separation state screen] (i) ATgroup Name This indicates the name of the ATgroup to be recovered from fault separation. (ii) Type This indicates the ATgroup s location (concentrator or distributor). 103

112 Chapter 4 Software Operation Methods (iii) Number of Registered Volume (iv) ATgroup Consistency State (v) ATgroup Sync State (vi) Registered Volume List This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. This indicates the ATgroup consistency state. This indicates the ATgroup sync state. This section provides a list of pairs included in the target ATgroup, the status of each pair, and the execution results. The pair information items that appear in this screen are the same as those in the Replication Operations screen (except for the ATgroup name). Click the [Execute] button to display a confirmation dialog before recovering from fault separation. Handling of errors when recovering from fault separation The following message dialog appears if an error occurs when recovering from fault separation. Be sure to execute handling for the respective errors. Error Message Table 4-11 [02250] Failed in processing execution (ATgroup status is wrong:execution condition disagreement) [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (Number of created pair is increased/reduced) [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) Handling of Errors when Recovering from Fault Separation Description of Error and Handling Either the specified ATgroup is abnormal or the ATgroup s sync state is unexecutable. If the specified ATgroup is abnormal, restore the ATgroup s normal state. If the ATgroup s sync state is unexecutable, change the sync state to executable. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. After the Recover from Fault separation state screen was displayed, the configuration of pairs registered to the ATgroup was changed. Close the Recover from Fault separation state screen and then re-execute. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. 104

113 Chapter 4 Software Operation Methods Conditions for recovery from fault separation (i) The ATgroup sync state must be fault separation state. (ii) The operation target disk array will be monitored by ism and must be in monitoring state. (iii) The operation target disk array must not have frozen replication functions. (iv) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. 105

114 Chapter 4 Software Operation Methods Forced Deletion of an ATgroup The ATgroup Forced Deletion screen is used to forcibly delete ATgroups. Forced deletion of ATgroup is executed when an ATgroup becomes abnormal due to a fault, etc., or when invalid ATgroup information remains on the distributor side (RV side). Forced deletion of ATgroup is executed from the distributor side (RV side). Also, if invalid ATgroup information remains on the concentrator side (MV side), the ATgroup is first separated, or is first forcibly separated from the concentrator side (MV side), so that the ATgroup can then be deleted. Next, Forced deletion of ATgroup is executed in order to delete the invalid ATgroup information. Items displayed on ATgroup Forced Deletion screen Perform either of the following steps to open the ATgroup Forced Deletion screen. In the ATgroup information screen, select the ATgroup, then select [Operation] [ATgroup] [ATgroup Forced Operation] [ATgroup Forced Deletion] in the menu. In the ATgroup information screen, select the ATgroup, then right-click the mouse and select [ATgroup Forced Operation] [ATgroup Forced Deletion]. (i) (ii) (iv) (iii) (v) Figure 4-19 ATgroup Forced Deletion Screen [Description of ATgroup Forced Deletion screen] (i) ATgroup Name This indicates the name of the ATgroup to be forcibly deleted. (ii) Type This indicates the ATgroup s location. (iii) Number of Registered Volume This indicates number of pairs registered to the ATgroup. In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. If the user has the authorization for all the pairs registered in the ATgroup, the user can operate the ATgroup. (iv) ATgroup Consistency State This indicates the ATgroup consistency state. (v) ATgroup Sync State This indicates the ATgroup sync state. Click the [Execute] button to display a confirmation dialog before forcibly deleting the ATgroup. 106

115 Chapter 4 Software Operation Methods Handling of errors during Forced deletion of ATgroup The following message dialog appears if an error occurs during forced deletion of an ATgroup. Be sure to execute handling for the respective error. Error Message Table 4-12 Handling of Errors During Forced Deletion of ATgroup Description of Error and Handling [02064] Volume Deletion was failed. [02065] ATgroup Deletion was failed. [02250] Failed in processing execution (Monitoring stopped) [02250] Failed in processing execution (Freeze) [02250] Failed in processing execution (No specified ATgroup) [02250] Failed in processing execution (ATgroup status is wrong: execution condition disagreement) [02250] Failed in processing execution (There are pairs without the management authorization in the ATgroup) The disk array is abnormal. Check the state of the disk array. Recover the disk array and restore ATgroup to normal state. Failed in deleting ATgroups. Check the state of the unit and re-execute. ism has suspended monitoring of the target disk array. Restart the disk array monitoring. The target disk array has been frozen. Defreeze it. The specified ATgroup does not exist. Check whether the specified ATgroup has already been deleted. The specified ATgroup is not normal or ATgroup sync state is not in the executable state. If the specified ATgroup is not normal, restore it to the normal state. If ATgroup sync state is not the executable state, change it to the executable state. A partition user must operate the ATgroup for which the partition user has the management authorization, or a storage user must perform the operation. Conditions for Forced deletion of ATgroup (i) When deleting the distributor (a) The disk array on the ATgroup s distributor side (RV side) will be monitored by ism and must be in monitoring state. (b) The disk array on the ATgroup s distributor side (RV side) must not have frozen replication functions. (c) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. (ii) When deleting the concentrator (a) The disk array on the ATgroup s concentrator side (MV side) will be monitored by ism and must be in monitoring state. (b) Replication functions of the disk array on the ATgroup s concentrator side (MV side) is not in the freeze state. (c) ATgroup sync state of the specified ATgroup is separated, fault separation, or invalid. (d) In the case of the partition user, the management authorization for all the pairs registered in the specified ATgroup is required. 107

116 Chapter 4 Software Operation Methods Caution point for Forced deletion of ATgroup Forced deletion of ATgroup is executed when an ATgroup becomes abnormal due to a fault, etc., or when invalid ATgroup information remains on the distributor side (RV side). Forced deletion of ATgroup is executed from the distributor side (RV side). Also, if invalid ATgroup information remains on the concentrator side (MV side), the ATgroup is first separated, or is first forcibly separated from the concentrator side (MV side), so that the ATgroup can then be deleted. Next, Forced deletion of ATgroup is executed in order to delete the invalid ATgroup information Outputting the ATgroup Information List in CSV format The ATgroup information list that is displayed in the ATgroup information screen can be output in CSV format by selecting [File] [CSV Output of Information List] in the ATgroup information screen s menu. For details on outputting this list information display in CSV format, refer to the Data Replication User s Manual (Function Guide). In the case of the partition user, Number of Registered Volume (Authorized) is displayed. The number of the pairs that the partition user can manage is displayed in the parentheses. 108

117 Chapter 4 Software Operation Methods Saving an ATgroup and Registered Pair Information Both pair information and the information about pairs registered to an ATgroup can be saved by selecting [File] [Save the Pair Setting Information]. This saved ATgroup and registered pair information can be used to make batch settings via Storage Manager Configuration Setting screen. For details on saving pair information, refer to the Data Replication User s Manual (Function Guide). For details on batch settings, refer to the Configuration Setting Tool User s Manual (GUI). In the case of a partition user, if the user does not have the management authorization for some pairs in the ATgroup, the setting information on those pairs cannot be saved. Only the information on the pairs that the partition user has the management authorization is saved. When you attempt such an operation, the following message screen appears. To save all the pair information, the storage user must perform the operation. Figure 4-20 Registration Pair Information Saving Confirmation Screen When a partition user uses the [Replication Setting] function of the [Batch Setting] in configuration setting with the file saved by the storage user, the partition user can set only the pairs for which the user has the authority. 109

118 Chapter 4 Software Operation Methods Setting the ATgroup Resynchronize Attribute The ATgroup Resynchronize Attribute Setting screen is used to specify the ATgroup resynchronize attribute setting. Displaying the ATgroup Resynchronize Setting screen Perform either of the following steps to open the Resynchronize Attribute Setting screen. Select the concentrator for the ATgroup to be changed from the ATgroup information screen, and then select [Operation] [ATgroup] [ATgroup Setting] [Resynchronize Attribute Setting] in the menu. Right-click the concentrator for the ATgroup to be changed from the ATgroup information screen, and then select [ATgroup Setting] [Resynchronize Attribute Setting] in the shortcut menu. (i) (ii) (iii) Figure 4-21 ATgroup Resynchronize Attribute Setting Screen [Description of the ATgroup Resynchronize Attribute Setting screen] (i) ATgroup Name This indicates the name of the ATgroup that is the target of the ATgroup resynchronize attribute. (ii) Resynchronize Attribute Setting Select the option button to valid, invalid, or force invalid the ATgroup resynchronize attribute. (iii) Resynchronize Attribute Type Select the option button to set the ATgroup resynchronize attribute type. You can select the resynchronize attribute type that is supported by the disk array in use. When the [Resynchronize Attribute Setting] is valid, the 110

119 Chapter 4 Software Operation Methods resynchronize attribute type cannot be changed. If you want to change the resynchronize attribute type on the disk array supporting multiple resynchronize attribute types, set the [Resynchronize Attribute Setting] to invalid and then change the resynchronize attribute types. Clicking the [Set] button displays the confirmation dialog box to ask whether the specified ATgroup resynchronize attribute setting is applied. Handling of errors of ATgroup Resynchronize Attribute Setting When an error occurs while the ATgroup resynchronize attribute setting is specified, the following messages are displayed. Take appropriate action for the displayed error message. Table 4-13 Handling of Errors during ATgroup Resynchronize Attribute Setting Error Message Description of Error and Handling [02087] Resynchronize attribute setting was failed [02250] Failed in processing execution (Resync state is not invalid) [02250] Failed in processing execution (Resync state is invalid) [02250] Failed in processing execution (Resync state is recovering) [02250] Failed in processing execution (The SV count of distributor's RV is not 1) [02250] Failed in processing execution (The number of automatic resynchronization DDR pair for the distributor's RV is not 1) Failed to specify the ATgroup resynchronize attribute setting. Check the status of the disk array, and then specify the setting again. If the resynchronization function is disabled for the ATgroup, enable the function. If the resynchronization function is disabled for the ATgroup, enable the function. If the resynchronization function is disabled for the ATgroup, enable the function. The volume is not usable for setting resynchronize attribute. Check whether the SV count of distributor's RV is 1. The volume is not usable for setting resynchronize attribute. Check whether automatic resynchronization DDR pair for the distributor's RV exists. Execution Conditions of Resynchronize Attribute Setting (i) Execution Conditions of set Resynchronize Attribute is valid. (a) The disk array where the specified ATgroup is stored is monitored. (b) The disk array where the specified ATgroup is stored is not in the Freeze state. (c) The disk array where the specified ATgroup is stored is within the management (d) The link path between disk arrays is normal. (e) The type of the specified ATgroup is concentrator. (f) The SV count of distributor's RV is only 1(snapshot type). (g) The number of automatic resynchronization DDR pair for the distributor's RV is only 1(DDR type). (h) When the ATgroup Resynchronize state of the specified ATgroup is Invalid. 111

120 Chapter 4 Software Operation Methods (i) User is the storage user. (ii) Execution Conditions of set Resynchronize Attribute is invalid. (a) The disk array where the specified ATgroup is stored is monitored. (b) The disk array where the specified ATgroup is stored is not in the Freeze state. (c) The disk array where the specified ATgroup is stored is within the management. (d) The type of the specified ATgroup is concentrator. (e) When the ATgroup Resynchronize state of the specified ATgroup is Monitoring, Failure, Resuming. (f) User is the storage user. (iii) Execution Conditions of set Resynchronize Attribute is force invalid. (a) The disk array where the specified ATgroup is stored is monitored. (b) The disk array where the specified ATgroup is stored is not in the Freeze state. (c) The disk array where the specified ATgroup is stored is within the management. (d) The type of the specified ATgroup is concentrator. (e) When the ATgroup Resynchronize state of the specified ATgroup is not Invalid. (f) User is the storage user. 112

121 Chapter 4 Software Operation Methods Resetting the ATgroup Resynchronize Attribute The ATgroup Resynchronize Attribute Resetting screen is used to specify the ATgroup resynchronize attribute resetting. Displaying the Resynchronize Resetting screen Perform either of the following steps to open the Resynchronize Attribute Setting screen. Select the concentrator for the ATgroup to be changed from the ATgroup information screen, and then select [Operation] [ATgroup] [ATgroup Setting] [Resynchronize Attribute Resetting] in the menu. Right-click the concentrator for the ATgroup to be changed from the ATgroup information screen, and then select [ATgroup Setting] [Resynchronize Attribute Resetting] in the shortcut menu. (i) (ii) Figure 4-22 ATgroup Resynchronize Attribute Resetting Screen [Description of the ATgroup Resynchronize Attribute Resetting screen] (i) Disk Array Name This indicates the name of the disk array including the ATgroup for which the resynchronize attribute resetting. (ii) ATgroup Name This indicates the name of the ATgroup for which the resynchronize attribute resetting. Clicking the [Reset] button displays the confirmation dialog box to ask whether the specified ATgroup resynchronize attribute resetting is applied. 113

122 Chapter 4 Software Operation Methods Handling of errors of ATgroup Resynchronize Attribute Resetting When an error occurs while the ATgroup resynchronize attribute resetting is specified, the following messages are displayed. Take appropriate action for the displayed error message. Table 4-14 Handling of Errors during ATgroup Resynchronize Attribute Resetting Error Message Description of Error and Handling [02088] Resynchronize attribute resetting was failed [02250] Failed in processing execution (Resync state is not failure) [02250] Failed in processing execution (The SV count of distributor's RV is not 1) [02250] Failed in processing execution (The number of automatic resynchronization DDR pair for the distributor's RV is not 1) Failed to specify the ATgroup resynchronize attribute resetting. Check the status of the disk array, and then specify the resetting again. If the resynchronization function is disabled for the ATgroup, enable the function. The volume is not usable for resetting resynchronize attribute. Check whether the SV count of distributor's RV is 1. The volume is not usable for resetting resynchronize attribute. Check whether automatic resynchronization DDR pair for the distributor's RV exists. Execution Conditions of Resynchronize Attribute Resetting (i) The disk array where the specified ATgroup is stored is monitored. (ii) The disk array where the specified ATgroup is stored is not in the Freeze state. (iii) The disk array where the specified ATgroup is stored is within the management. (iv) The link path between disk arrays is normal. (v) The type of the specified ATgroup is concentrator. (vi) The SV count of distributor's RV is only 1(snapshot type). (vii) The number of automatic resynchronization DDR pair for the distributor's RV is only 1(DDR type). (viii) When the ATgroup Resynchronize state of the specified ATgroup is Failure. (ix) User is the storage user. 114

123 Chapter 4 Software Operation Methods 4.2 ReplicationControl/DisasterRecove ry Command Operation Methods (CLI) Commands provided by ReplicationControl/DisasterRecovery can be used to perform the following operations. ATgroup operations ATgroup operations include building an ATgroup, ATgroup replication control, and ATgroup status checking. Remote operation Replication control of pairs established within a remote disk array that is not directly connected to the business application server can be executed, along with replication control of ATgroups. Table 4-15 List of ATgroup Operation Commands Command Name Operation Description Section ismrc_replicate Replicate Executes replication of ATgroup ismrc_separate Separate Executes separation of ATgroup ismrc_restore Restore Executes restoration of ATgroup ismrc_change Recovery from fault separation Executes recovery from fault separation ismrc_wait Wait for change in ATgroup sync state Waits for change in ATgroup sync state ismrc_query Display ATgroup s copy state Displays the copy state of ATgroup and pairs registered to ATgroup ismrc_sense Display ATgroup list Display ATgroup list information ismrc_atg Build ATgroup Creates/deletes ATgroup, or registers/deletes pairs in ATgroup Table 4-16 List of Remote Operation Commands Command Name Operation Description Section ismrc_replicate Replicate Executes replication of remote pair. ismrc_separate Separate Executes separation of remote pair. ismrc_restore Restore Executes restoration of remote pair. ismrc_change Change copy mode Changes remote pair s copy mode. ismrc_wait Wait for state change Waits for change in remote pair s state ismrc_query Display copy state Displays remote pair s copy state. ismrc_sense Display setting information Displays remote pair s setting information. 115

124 Chapter 4 Software Operation Methods Replicating an ATgroup (ismrc_replicate) To replicate an ATgroup, specify the target ATgroup in the ismrc_replicate command and execute the command. Executable construct of ismrc_replicate command (Executable construct of replicate ATgroup command) ismrc_replicate -atg ATgroup name [-cprange diff full ] [ -atmode continue stop ] [-cpmode sync semi bg ] [-rvacc rv_access ] [-nowait -wait [sense interval] ] [-arrayname disk array name ] [-iopath direct manager ] (Executable construct of ordinary pair replication) ismrc_replicate -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] [-cprange copy_range ] [-cpmode copy_mode ] [-rvacc rv_access ] [-nowait -wait [second] ] [ -iopath iopath ] ismrc_replicate -file file_name [-cprange copy_range ] [-cpmode copy_mode ] [-rvacc rv_access ] [-nowait -wait [second] ] [ -iopath iopath ] (Executable construct of command version check) ismrc_replicate -ver (Executable construct of command usage display) ismrc_replicate -?... (Windows) ismrc_replicate -help... (UNIX) The following describes replication of ATgroups. For description of normal pair replication, refer to the ControlCommand Command Reference. Specifiable options and parameters for ATgroup replication (i) Specification of ATgroup (required) -atg ATgroup name This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required for execution of ATgroup replication. (ii) Specification of copy range -cprange diff full This specifies the copy range. When diff is specified, only the differential between each pair s pair MV and RV data will be copied. When full is specified, the entire range of data in each pair s MV and RV will be copied. 116

125 Chapter 4 Software Operation Methods The default value when this option is omitted is diff. (iii) Specification of MV access after atomic-break -atmode continue stop This specifies enabling MV access after an atomic-break. When continue is specified, MV access continue is set and MV access is enabled even after an atomic-break has been executed. When stop is specified, an MV access stop is set so that MV access is disabled after an atomic-break has been executed. Ordinarily, stop should not be specified since setting an MV access stop can result in unexpected stops in business operations. The default value when this option is omitted is continue. (iv) Specification of copy mode -cpmode sync semi bg This specifies the copy mode during replication. When sync is specified, synchronous mode is set. When semi is specified, semi synchronous copy mode of order guarantee is set. When bg is specified, background copy mode is set. The default value when this option is omitted is sync. (v) Specification of RV access restriction -rvacc ro nr This specifies the RV access restriction. When ro is specified, only referencing (reading) RV is permitted. When nr is specified, accessing RV is not permitted. The default value when this option is omitted is nr. (vi) Specification of wait for copy completion -nowait -wait [sense interval] This specifies whether or not to wait for completion of copying before using a command to change the ATgroup sync state to replicate synchronous (Rpl/sync). When -nowait is specified, the ATgroup Replicate Execution command can be executed immediately, without waiting for completion of copying. When -wait is specified, execution of this command is held pending until copying is completed. When a sense interval value is specified after -wait, completion of copying is monitored for the interval time set by the value. The specifiable range for this value is 1 to 30 (seconds). If no value is specified here, the value set via the replication control option settings file (Windows) or by the RPL_WATCHDEV environment variable (UNIX) becomes valid. If neither sense interval nor RPL_WATCHDEV is specified, the default sense interval is 5 seconds. The default value when this option is omitted is -nowait. 117

126 Chapter 4 Software Operation Methods (vii) Specification of disk array name -arrayname disk array name (viii) Specification of I/O issue path -iopath direct manager This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Messages output during ATgroup replication (Windows) [Start message] Replicate Start ATG:atgroup YYYY/MM/DD hh:mm:ss [Message during execution] Replicating... [End message] Replicate Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for the RPL_ATGPAIRMSG parameter in the [ATGROUP] section of the replication operation option setting file. For details on the replication operation option setting file, refer to the ControlCommand Command Reference. [Start message] Replicate Start YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name 118

127 Chapter 4 Software Operation Methods path [Message during execution] Replicating... [End message] Replicate Normal End YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path Description of message YYYY/MM/DD hh:mm:ss atgroup disk_number ld_name type volume name path Replicating... Start/end date/time (YYYY/MM/DD hh:mm:ss) ATgroup name Disk number Logical disk name Use format Mounting point volume name Drive path (mounted to drive letter or NTFS folder) This message is displayed to indicate when ATgroup replication is being executed. The trailing dots are refreshed until the ATgroup sync state becomes replicate synchronous. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of copying. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the [WAIT] section in the replication control option settings file. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Start message] Replicate Start ATG: atgroup YYYY/MM/DD hh:mm:ss [Message during execution] Replicating

128 Chapter 4 Software Operation Methods [End message] Replicate Normal End ATG: atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for RPL_ATGPAIRMSG of the environment variable. For details on the environment variable, refer to the ControlCommand Command Reference. [Start message] Replicate Start YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type [Message during execution] Replicating... [End message] Replicate Normal End YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type Description of message YYYY/MM/DD hh:mm:ss atgroup special_file_name ld_name type Replicating... Start/end date/time (YYYY/MM/DD hh:mm:ss) ATgroup name Special file name Logical disk name Use format This message is displayed to indicate when ATgroup replication is being executed. The trailing dots are refreshed until the ATgroup sync state becomes replicate synchronous. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of copying. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the environment variable. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. 120

129 Chapter 4 Software Operation Methods Conditions for replication of ATgroup (i) The target ATgroup s sync state must be one of the following. Separated Fault separation (Fault) Replicate suspend (Rpl/suspend) Replicate start (Rpl/start) (ii) When replication is executed for an ATgroup whose ATgroup sync state is replicate start (Rpl/start) or replicate suspend (Rpl/suspend), MV access after atomic-break, RV access restriction, and copy mode should not be set, or the setting at replication starting must be specified. (iii) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. (iv) Creation and updating of the volume list must be completed. (v) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (vi) There must not be any frozen replication functions in any disk array that belongs to this command s target ATgroup. (vii) The link state for the disk array must be normal. (viii) Any RVs included in the ATgroup must be unmounted from the system. (ix) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. (x) The MV or RV in the ATgroup must not be in the not in use state. (xi) The pool to which the MV or RV in the ATgroup belongs must not be in the rotation stop state. (xii) The RV in the ATgroup must not be in the prevent state. (xiii) All the replication reserved volumes used as MV or RV registered in the specified ATgroup are not a fault condition when using the M series and A5000. (xiv)when the semi-synchronous copy mode is specified to replicate an AT group, a semi-synchronous buffer must be set in advance. However, the above conditions (iii) to (v) are not applied to the operation via ism. Notes on replicating an ATgroup (i) When ATgroup replication is executed, the ATgroup consistency state becomes Non-atomic, regardless of the ATgroup s copy mode. When the ATgroup s copy mode is either synchronous mode or semi synchronous copy mode of order guarantee, if the ATgroup sync 121

130 Chapter 4 Software Operation Methods state is switched to replicate synchronous (Rpl/sync), the ATgroup consistency state will change to Atomic. When the copy mode is background copy mode, if the ATgroup sync state is switched to replicate synchronous (Rpl/sync), the ATgroup consistency state will remain Non-atomic. (ii) If the ATgroup s replication is executed while in background copy mode, an atomic-break will not be executed. (iii) If MV Access Stop is set for MV Access after Atomic-break, business operations may be stopped unexpectedly, so this setting should normally be avoided. (iv) When ATgroup replication is executed for an ATgroup whose sync state is replicate suspend (Rpl/suspend), any suspended copying is restarted. (v) There is an upper limit for the number of the ATgroups that can be replicated concurrently in semi synchronous copy mode of order guarantee. Replication cannot be executed when this limit is exceeded. (vi) Before executing ATgroup replication, make sure that any RVs included in the ATgroup have been unmounted. If replication is executed while one or more RVs are mounted, the system s buffer image will no longer match the disk s image, which can result in abnormal RV data or abnormalities in the system connected to the RV. (vii) On a UNIX system, only a super user can run this command. 122

131 Chapter 4 Software Operation Methods Separating an ATgroup (ismrc_separate) To separate an ATgroup, specify the target ATgroup in the ismrc_separate command and execute the command. Executable construct of ismrc_separate command (Executable construct of ATgroup separation) ismrc_separate -atg ATgroup name [-force ] [-nowait -wait [sense interval] ] [-arrayname disk array name ] [ -iopath direct manager ] (Executable construct of ordinary pair separation) ismrc_separate -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] [-rvacc rv_access ] [-rvuse rv_use -force force_arg ] [-nowait -wait [second] ] [ -iopath iopath ] ismrc_separate -file file_name [-rvacc rv_access ] [-rvuse rv_use -force force_arg ] [-nowait -wait [second] ] [ -iopath iopath ] (Executable construct of command version check) ismrc_separate -ver (Executable construct of command usage display) ismrc_separate -?...(Windows) ismrc_separate -help...(unix) The following describes separation of ATgroups. For description of normal pair separation, refer to the ControlCommand Command Reference. Specifiable options and parameters for ATgroup separation (i) Specification of ATgroup (required) -atg ATgroup name This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required for execution of ATgroup separation. (ii) Specification of forced separation -force This option is specified to forcibly separate an ATgroup. It is used when normal separation cannot be executed, such as when using an RV in an ATgroup 123

132 Chapter 4 Software Operation Methods after a link fault has occurred, or when a system fault or a line fault between disk arrays has occurred. If this option is omitted, normal separation is executed. (iii) Specification of wait for completion of ATgroup separation -nowait -wait [sense interval] This specifies whether or not to wait for completion of ATgroup separation before using a command to change the ATgroup sync state to separated. When -nowait is specified, the ATgroup Separate command can be executed immediately, without waiting for completion of separation. When -wait is specified, execution of this command is held pending until separation is completed. When a sense interval value is specified after -wait, completion of ATgroup separation is monitored for the interval time set by the value. The specifiable range for this value is 1 to 30 (seconds). If no value is specified here, the value set via the replication control option settings file (Windows) or by the RPL_WATCHDEV environment variable (UNIX) becomes valid. If neither sense interval nor RPL_WATCHDEV is specified, the default sense interval is 5 seconds. The default value when this option is omitted is -nowait. (iv) Specification of disk array name -arrayname disk array name This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. (v) Specification of I/O issue path -iopath direct manager This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. 124

133 Chapter 4 Software Operation Methods Messages output during ATgroup separation (Windows) [Start message] Separate Start ATG:atgroup [Message during execution] Separating... YYYY/MM/DD hh:mm:ss [End message] Separate Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for the RPL_ATGPAIRMSG parameter in the [ATGROUP] section of the replication operation option setting file. For details on the replication operation option setting file, refer to the ControlCommand Command Reference. [Start message] Separate Start YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path [Message during execution] Separating... [End message] Separate Normal End YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path 125

134 Chapter 4 Software Operation Methods Description of message YYYY/MM/DD hh:mm:ss atgroup disk_number ld_name type volume name path Separating... Start/end date/time (YYYY/MM/DD hh:mm:ss) ATgroup name Disk number Logical disk name Use format Mounting point volume name Drive path (mounted to drive letter or NTFS folder) This message is displayed to indicate when ATgroup separation is being executed. The trailing dots are refreshed until the ATgroup s separation is completed. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of copying. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the [WAIT] section in the replication control option settings file. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Start message] Separate Start ATG:atgroup YYYY/MM/DD hh:mm:ss [Message during execution] Separating... [End message] Separate Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for RPL_ATGPAIRMSG of the environment variable. For details on the environment variable, refer to the ControlCommand Command Reference. [Start message] Separate Start YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type 126

135 Chapter 4 Software Operation Methods [Message during execution] Separating... [End message] Separate Normal End YYYY/MM/DD hh:mm:ss ATG: atgroup MV: special_file_name ld_name type RV: special_file_name ld_name type Description of message YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name special_file_name Special file name ld_name Logical disk name type Use format Separating... This message is displayed to indicate when ATgroup separation is being executed. The trailing dots are refreshed until the ATgroup has been separated. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of separation. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the environment variable. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for separation of ATgroup (i) This command s target ATgroup consistency state and the ATgroup sync state must be one of the following. However, this does not apply to forced separation (when the -force option has been specified). The ATgroup consistency state is Atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) The ATgroup consistency state is Non-atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) The ATgroup consistency state is Non-atomic and the ATgroup sync state is replicating (Rpl/exec) The ATgroup consistency state is Non-atomic and the ATgroup sync state is separate start (Sep/start) 127

136 Chapter 4 Software Operation Methods (ii) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. (iii) Creation and updating of the volume list must be completed. (iv) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (v) There must not be any frozen replication functions in any disk array that belongs to this command s target ATgroup. However, this is not applied to forced separation when the -force option has been specified. (vi) The link state for the disk array must be normal. However, this does not apply to forced separation (when the -force option has been specified). (vii) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. (viii) The MV or RV in the ATgroup must not be in the not in use state. (ix) The pool to which the MV or RV in the ATgroup belongs must not be in the rotation stop state. (x) All the replication reserved volumes used as MV or RV registered in the specified ATgroup are not a fault condition when using the M series and A5000. However, the above conditions (ii) to (iv) are not applied to the operation via ism. Notes on separating an ATgroup (i) If the ATgroup consistency state is Non-atomic when ATgroup separation is executed, separation is executed for each pair in the ATgroup. During this processing, the ATgroup consistency state remains Non-atomic. (ii) When the ATgroup sync state is separated, the access restriction on RVs included in the ATgroup is Read/write accessible. (iii) When the ATgroup sync state is Atomic and forced separation is performed, the access restriction for RVs in the ATgroup remains in Read Only or Not Ready. To enable read/write access to RVs in the ATgroup, execute the operation to recover from fault separation on the distributor side (RV side). Recovery from fault separation also changes the ATgroup sync state to separated. (iv) When the ATgroup consistency state is Non-atomic, pause point data can be maintained in an RV by unmounting the MV s file system or by executing using the database function s backup mode before separating the ATgroup. (v) When volumes on both the concentrator side (MV side) and the distributor side (RV side) are not being recognized from the save server, if Forced separation of ATgroup is executed by 128

137 Chapter 4 Software Operation Methods specifying the -force option, volumes on the concentrator side (MV side) and distributor side (RV side) will be recognized and executed by the respective servers. (vi) Immediately after ATgroup separation, the RV can be used, so RV Separate (immediate) cannot be used. (vii) On a UNIX system, only a super user can run this command. 129

138 Chapter 4 Software Operation Methods Restoring an ATgroup (ismrc_restore) To restore an ATgroup, specify the target ATgroup in the ismrc_restore command and execute the command. Executable construct of ismrc_restore command (Executable construct of ATgroup restoration) ismrc_restore -atg ATgroup name [-cprange diff full ] [-rvacc rv_access ] [-nowait -wait [sense interval] ] [-arrayname disk array name ] [ -iopath direct manager ] (Executable construct of normal pair restoration) ismrc_restore -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] [-cprange copy_range ] [-cpmode copy_mode ] [-rvacc rv_access ] [-nowait -wait [second] ] [-mode operation_mode ] [ -iopath iopath ] ismrc_restore -file file_name [-cprange copy_range ] [-cpmode copy_mode ] [-rvacc rv_access ] [-nowait -wait [second] ] [-mode operation_mode ] [ -iopath iopath ] (Executable construct of command version check) ismrc_restore -ver (Executable construct of command usage display) ismrc_restore -?... (Windows) ismrc_restore -help... (UNIX) The following describes restoration of ATgroups. For description of normal pair restoration, refer to the ControlCommand Command Reference. Specifiable options and parameters for ATgroup restoration (i) Specification of ATgroup (required) -atg ATgroup name This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required for execution of ATgroup restoration. (ii) Specification of copy range -cprange diff full This specifies the copy range. When diff is specified, 130

139 Chapter 4 Software Operation Methods only the differential between each pair s pair MV and RV data will be copied. When full is specified, the entire range of data in each pair s MV and RV will be copied. The default value when this option is omitted is diff. (iii) Specification of RV access restriction -rvacc ro nr This specifies the RV access restriction. When ro is specified, only referencing (reading) RV is permitted. When nr is specified, accessing RV is not permitted. The default value when this option is omitted is nr. (iv) Specification of wait for completion of ATgroup restoration -nowait -wait [sense interval] This specifies whether or not to wait for completion of ATgroup restoration before using a command to change the ATgroup sync state to restored. When -nowait is specified, the ATgroup Restore Execution command can be executed immediately, without waiting for completion of restoration. When -wait is specified, execution of this command is held pending until restoration is completed. When a sense interval value is specified after -wait, completion of ATgroup restoration is monitored for the interval time set by the value. The specifiable range for this value is 1 to 30 (seconds). If no value is specified here, the value set via the replication control option settings file (Windows) or by the RPL_WATCHDEV environment variable (UNIX) becomes valid. If neither sense interval nor RPL_WATCHDEV is specified, the default sense interval is 5 seconds. The default value when this option is omitted is -nowait. (v) Specification of disk array name -arrayname disk array name This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. (vi) Specification of I/O issue path -iopath direct manager This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this 131

140 Chapter 4 Software Operation Methods option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Messages output during ATgroup restoration (Windows) [Start message] Restore Start ATG:atgroup YYYY/MM/DD hh:mm:ss [Message during execution] Restoring... [End message] Restore Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for the RPL_ATGPAIRMSG parameter in the [ATGROUP] section of the replication operation option setting file. For details on the replication operation option setting file, refer to the ControlCommand Command Reference. [Start message] Restore Start YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path [Message during execution] Restoring... [End message] Restore Normal End YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name 132

141 Chapter 4 Software Operation Methods path RV: disk_number ld_name type volume name path Description of message YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name disk_number Disk number ld_name Logical disk name type Use format volume name Mounting point volume name path Drive path (mounted to drive letter or NTFS folder) Restoring... This message is displayed to indicate when ATgroup restoration is being executed. The trailing dots are refreshed until the ATgroup s restoration is completed. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of restoration. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the environment variable. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Start message] Restore Start ATG:atgroup YYYY/MM/DD hh:mm:ss [Message during execution] Restoring... [End message] Restore Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for RPL_ATGPAIRMSG of the environment variable. For details on the environment variable, refer to the ControlCommand Command Reference. 133

142 Chapter 4 Software Operation Methods [Start message] Restore Start YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type [Message during execution] Restoring... [End message] Restore Normal End YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type Description of message YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name special_file_name Special file name ld_name Logical disk name type Use format Restoring... This message is displayed to indicate when ATgroup restoration is being executed. The trailing dots are refreshed until the ATgroup s restoration is completed. The messages for when execution of this command is in progress and when it has been completed are output only when -wait has been specified to set a wait for completion of restoration. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the environment variable. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for restoration of ATgroup (i) This command s target ATgroup sync state must be one of the following. Separated Restore suspend (Rst/suspend) (ii) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to 134

143 Chapter 4 Software Operation Methods which the ATgroup belongs must be registered in the volume list and must be accessible. (iii) Creation and updating of the volume list must be completed. (iv) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (v) There must not be any frozen replication functions in any disk array that belongs to this command s target ATgroup. (vi) The link state for the disk array must be normal. (vii) The MV and RVs must be unmounted from the system. (viii) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. (ix) The MV or RV in the ATgroup must not be in the not in use state. (x) The pool to which the MV or RV in the ATgroup belongs must not be in the rotation stop state. (xi) The MV in the ATgroup must not be in the prevent state. (xii) All the replication reserved volumes used as MV or RV registered in the specified ATgroup are not a fault condition when using the M series and A5000. However, the above conditions (ii) to (iv) are not applied to the operation via ism. Notes on restoring an ATgroup (i) ATgroup restoration is performed only as RV restore (protect). When ATgroup restoration is completed, the ATgroup sync state changes to separated. (ii) When ATgroup restoration is completed, the ATgroup sync state changes to separated and the access restriction on RVs included in the ATgroup is read/write accessible. (iii) Before executing ATgroup restoration, make sure that any MVs or RVs included in the ATgroup have been unmounted. If ATgroup restoration is executed while one or more MVs or RVs are mounted, the system s buffer image will no longer match the disk s image, which can result in abnormal MV data or abnormalities in the system connected to the MV and RV. (iv) On a UNIX system, only a super user can run this command. 135

144 Chapter 4 Software Operation Methods Recovery from Fault Separation (ismrc_change) Recovery from fault separation, specify the target ATgroup in the ismrc_change command and execute the command. Recovery from fault separation is executed when using MVs or RVs that have been in the fault separation state due to a fault or forced operation. Executable construct of ismrc_change command (Executable construct of recovery from fault separation) ismrc_change -atg ATgroup name -force separated [-arrayname disk array name ] [ -iopath direct manager ] (Executable construct of copy control state change for ordinary pair) ismrc_change -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] -suspend -resume -sync -semi -bg [ -iopath iopath ] ismrc_change -file file_name -suspend -resume -sync -semi -bg [ -iopath iopath ] (Executable construct of command version check) ismrc_change -ver (Executable construct of command usage display) ismrc_change -?... (Windows) ismrc_change -help... (UNIX) Recovery from fault separation is described below. For description of copy control state changes for ordinary pairs, refer to the ControlCommand Command Reference. Specifiable options and parameters for recovery from fault separation (i) Specification of ATgroup (required) -atg ATgroup name This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required when recovering from fault separation. (ii) Specification of ATgroup sync state change (required) -force separated This option is specified to forcibly change an ATgroup s active mode. When recovering from fault separation, specifying separated forcibly changes the ATgroup sync state to the separated state, which enables to use MV or RV. Specification of this option 136

145 Chapter 4 Software Operation Methods (iii) Specification of disk array name -arrayname disk array name (iv) Specification of I/O issue path -iopath direct manager is required when recovering from fault separation. This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Messages output when recovering from fault separation (Windows) [Start message] Change Start ATG:atgroup YYYY/MM/DD hh:mm:ss [End message] Change Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for the RPL_ATGPAIRMSG parameter in the [ATGROUP] section of the replication operation option setting file. For details on the replication operation option setting file, refer to the ControlCommand Command Reference. [Start message] Change Start YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path 137

146 Chapter 4 Software Operation Methods [End message] Change Normal End YYYY/MM/DD hh:mm:ss ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path Description of message YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name disk_number Disk number ld_name Logical disk name type Use format volume name Mounting point volume name path Drive path (mounted to drive letter or NTFS folder) For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. For description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Start message] Change Start ATG:atgroup YYYY/MM/DD hh:mm:ss [End message] Change Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the start and end messages by specifying OUTPUT for RPL_ATGPAIRMSG of the environment variable. For details on the environment variable, refer to the ControlCommand Command Reference. [Start message] Change Start YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type 138

147 Chapter 4 Software Operation Methods [End message] Change Normal End YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type Description of message YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name special_file_name Special file name ld_name Logical disk name type Use format For detailed description of this command s display items and return values, and environment variables, refer to the ControlCommand Command Reference. For description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for recovery from fault separation (i) This command s target ATgroup sync state must be fault separation. (ii) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. (iii) Creation and updating of the volume list must be completed. (iv) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (v) There must not be any frozen replication functions in any disk array that belongs to this command s target ATgroup. (vi) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. (vii) All the replication reserved volumes used as MV or RV registered in the specified ATgroup are not a fault condition when using the M series and A5000. However, the above conditions (ii) to (iv) are not applied to the operation via ism. Notes for recovery from fault separation (i) On a UNIX system, only a super user can run this command. 139

148 Chapter 4 Software Operation Methods Waiting for the ATgroup Sync State to Change (ismrc_wait) To wait for a change in the ATgroup sync state, specify the target ATgroup in the ismrc_wait command and execute the command. This command to wait for change in ATgroup sync state is executed in order to wait for the ATgroup sync state to change from Replicate ATgroup, Separate ATgroup, or Restore ATgroup. Once the sync state changes to the target sate, execution of this command is ended normally. Executable construct of ismrc_wait command (Executable construct of wait for change in ATgroup sync state) ismrc_wait -atg ATgroup name -cond sync sep fault atsep atfault [ -interval sense interval ] [ -limit limit time nolim ] [-arrayname disk array name ] [ -iopath direct manager ] (Executable construct of wait for completion of normal pair replication, restoration, or separation) ismrc_wait -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] -cond condition [-interval interval_time ] [-limit limit_time ] [ -iopath iopath ] ismrc_wait -rv volume -rvflg rv_flg -cond condition [-interval interval_time ] [-limit limit_time ] [ -iopath iopath ] ismrc_wait -file file_name -cond condition [-interval interval_time ] [-limit limit_time ] [ -iopath iopath ] (Executable construct of command version check) ismrc_wait -ver (Executable construct of command usage display) ismrc_wait -?... (Windows) ismrc_wait -help... (UNIX) Waiting for in a change in the ATgroup sync state is described below. For description of waiting for completion of normal pair replication, restoration, or separation, refer to the ControlCommand Command Reference. 140

149 Chapter 4 Software Operation Methods Specifiable options and parameters for wait for change in ATgroup sync state (i) Specification of ATgroup (required) -atg ATgroup name This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required when executing wait for change in ATgroup sync state. (ii) Specification of condition for ending wait (required) -cond sync sep fault atsep atfault This specifies conditions for ending a wait. When sync is specified, this command terminates normally when ATgroup sync state becomes replicate synchronous (Rpl/sync). When sep is specified, this command terminates normally when the ATgroup sync state becomes separated. When fault is specified, this command terminates normally when the ATgroup sync state becomes either separated or fault separation. When atsep is specified, this command terminates normally when the ATgroup consistency state becomes Atomic and the ATgroup sync state becomes separated. When atfault is specified, this command terminates normally when the ATgroup consistency state becomes Atomic and the ATgroup sync state becomes separated or fault separation. This option is required when waiting for a change in the ATgroup sync state. (iii) Specification of sense interval -interval sense interval This specifies the sense (monitoring) interval when waiting for a change in the ATgroup sync state. Any integer from 1 to 30 (seconds) can be specified in sense interval as the sense interval. The default value when this option is omitted is the value set by replication control option settings file (Windows) or the RPL_WATCHDEV environment variable (UNIX). If neither sense interval nor RPL_WATCHDEV has been specified, the sense interval becomes 5 seconds. (iv) Specification of monitoring limit time -limit limit time nolim This specifies the limit time when waiting for a change in the ATgroup sync state. Any integer from 0 to (seconds) can be specified in limit time as the limit time, or nolim (unlimited) can be specified. The default value when this option is omitted is the value 141

150 Chapter 4 Software Operation Methods (v) Specification of disk array name -arrayname disk array name set by the replication control option settings file (Windows) or the environment variable RPL_LIMWATCHDEV (UNIX). If neither this option nor RPL_LIMWATCHDEV has been specified, the wait time is unlimited. This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. (vi) Specification of I/O issue path -iopath direct manager This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Messages output when waiting for change in ATgroup sync state (Windows) [Message during execution] Waiting... [End message] activity_state Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the end message by specifying OUTPUT for the RPL_ATGPAIRMSG parameter in the [ATGROUP] section of the replication operation option setting file. For details on the replication operation option setting file, refer to the ControlCommand Command Reference. [Message during execution] Waiting... [End message] activity_state Normal End YYYY/MM/DD hh:mm:ss 142

151 Chapter 4 Software Operation Methods ATG: atgroup MV: disk_number ld_name type volume name path RV: disk_number ld_name type volume name path Description of message activity_state Activity state of pair for which wait has ended YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name disk_number Disk number ld_name Logical disk name type Use format volume name Mounting point volume name path Drive path (mounted to drive letter or NTFS folder) Waiting... This message is displayed to indicate when waiting is in progress. The trailing dots are refreshed until the waiting has ended. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the [WAIT] section in the replication control option settings file. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Message during execution] Waiting... [End message] activity_state Normal End ATG:atgroup YYYY/MM/DD hh:mm:ss The MV and RV pair information can be displayed in the end message by specifying OUTPUT for RPL_ATGPAIRMSG of the environment variable. For details on the environment variable, refer to the ControlCommand Command Reference. [Message during execution] Waiting

152 Chapter 4 Software Operation Methods [End message] activity_state Normal End YYYY/MM/DD hh:mm:ss ATG:ATgroup MV:special_file_name ld_name type RV:special_file_name ld_name type Description of message activity_state Activity state of pair for which wait has ended YYYY/MM/DD hh:mm:ss Start/end date/time (YYYY/MM/DD hh:mm:ss) atgroup ATgroup name special_file_name Special file name ld_name Logical disk name type Use format Waiting... This message is displayed to indicate when waiting is in progress. The trailing dots are refreshed until the waiting has ended. Output of the execution in progress message can be suppressed by specifying UNUSE as the RPL_WAITMSG parameter of the [WAIT] section in the replication control option settings file. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for waiting for change in ATgroup sync state (i) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. (ii) Creation and updating of the volume list must be completed. (iii) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (iv) There must not be any frozen replication functions in any disk array that belongs to this command s target ATgroup. (v) The link state for the disk array must be normal. (vi) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. The above conditions (i) to (iii) are not applied to the operation via ism. 144

153 Chapter 4 Software Operation Methods Notes on waiting for the ATgroup sync state to change (i) While waiting, if the ATgroup sync state becomes replicate suspend (Rpl/suspend) or restore suspend (Rst/suspend), the wait will stop and end abnormally. (ii) While waiting, if any of the following occurs, the wait will stop and end abnormally. Abnormal link ATgroup link state is detected Copy of ATgroup s differential quantity of volume cannot be obtained Frozen target disk array is detected (iii) On a UNIX system, only a super user can run this command. 145

154 Chapter 4 Software Operation Methods Displaying the ATgroup Copy State (ismrc_query) To display an ATgroup s copy state, specify the target ATgroup in the ismrc_query command and execute the command. This command is used in order to display a particular ATgroup s copy state. Executable construct of ismrc_query command (Executable construct of ATgroup copy state display) ismrc_query -atg ATgroup name [-arrayname disk array name ] [ -iopath direct manager ] (Executable construct of replication state display for normal pair) ismrc_query -mv volume -mvflg mv_flg [-rv volume -rvflg rv_flg ] [ -iopath iopath ] ismrc_query -rv volume -rvflg rv_flg [ -iopath iopath ] ismrc_query -file file_name [ -iopath iopath ] (Executable construct of command version check) ismrc_query -ver (Executable construct of command usage display) ismrc_query -?... (Windows) ismrc_query -help... (UNIX) The following describes the ATgroup s copy state. For description of displaying the replication state of normal pairs, refer to the ControlCommand Command Reference. Options and parameters available for ATgroup s copy state display (i) Specification of ATgroup (required) -atg ATgroup name This specifies this command s target ATgroup. Enter the name of this command s target ATgroup as ATgroup name. This option is required when executing this command to display an ATgroup s copy state. (ii) Specification of disk array name -arrayname disk array name This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. When disk arrays are connected to both the concentrator side (MV side) and the distributor side (RV side), specify the 146

155 Chapter 4 Software Operation Methods (iii) Specification of I/O issue path -iopath direct manager disk array to be displayed. If this option is omitted, the target disk array is automatically searched. This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Information displayed by ATgroup copy state display (Windows) [Displayed information] ATG: atgroup Disk Array Name diskarray_name Site site ATG State atg_consistency_state(atg_sync_state) ATG Path State atg_path_state Atomic-break Mode atomic_break_mode Copy Mode copy_mode Resync State resync_state Recync Type resync_type Semi Copy Time semi_cptime ATG Copy Diff atg_copy_diffsize Pair Entry pair_entry_count Allowed Response allowed_response_time MV: Disk No. LD Name Type Volume Name Path RV: Disk No. LD Name Type Volume Name Path Activity State Sync State disk_number ld_name type volume_name path disk_number ld_name type volume_name path activity sync 147

156 Chapter 4 Software Operation Methods Copy Control State <state> Start Time <state> End Time Separate Diff Copy Diff RV Access Previous Active copy YYYY/MM/DD hh:mm:ss YYYY/MM/DD hh:mm:ss separate_diffsize copy_diffsize rvacc prev_state Description of display items atgroup ATgroup name diskarray_name Target disk array name site ATgroup s location (concentrator or distributor) atg_consistency_state ATgroup consistency state atg_sync_state ATgroup s sync state atg_path_state ATgroup s link state atomic_break_mode MV access after atomic-break copy_mode ATgroup s copy mode resync_state Resynchronization state resync_type Resynchronization type semi_cptime The time of the semi copy The time will be displayed on the distributor side only. atg_copy_diffsize Copy differential quantity of volume in ATgroup pair_entry_count Number of registered pairs in ATgroup allowed_response_time Allowance time for MV delay that is an ATgroup attribute disk_number Disk number ld_name Logical disk name type Use format volume_name Mounting point volume name path Drive path (mounted to drive letter or NTFS folder) activity Pair s activity state sync Pair s sync state copy Copy control state YYYY/MM/DD hh:mm:ss <state> Start Time indicates this operation s start date/time and <state> End Time indicates its end date/time (<state> differs according to the state) separate_diffsize Copy differential quantity of volume when separate state occurs copy_diffsize Copy differential quantity of volume when replicate or restore state occurs rvacc RV access restriction prev_state Pair s sync state before state transition 148

157 Chapter 4 Software Operation Methods For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Displayed information] ATG: atgroup Disk Array Name diskarray_name Site site ATG State atg_consistency_state(atg_sync_state) ATG Path State atg_path_state Atomic-break Mode atomic_break_mode Copy Mode copy_mode Resync State resync_state Resync Type resync_type Semi Copy Time semi_cptime ATG Copy Diff atg_copy_diffsize Pair Entry pair_entry_count Allowed Response allowed_response_time MV: Special File LD Name Type RV: Special File LD Name Type Activity State Sync State Copy Control State <state> Start Time <state> End Time Separate Diff Copy Diff RV Access Previous Active special_file_name ld_name type special_file_name ld_name type activity sync copy YYYY/MM/DD hh:mm:ss YYYY/MM/DD hh:mm:ss separate_diffsize copy_diffsize rvacc prev_state Description of display items atgroup diskarray_name ATgroup name Target disk array name 149

158 Chapter 4 Software Operation Methods site ATgroup s location (concentrator or distributor) atg_consistency_state ATgroup consistency state atg_sync_state ATgroup s sync state atg_path_state ATgroup s link state atomic_break_mode MV access after atomic-break copy_mode ATgroup s copy mode resync_state Resynchronization state resync_type Resynchronization type semi_cptime The time of the semi copy The time will be displayed on the distributor side only. atg_copy_diffsize Copy differential quantity of volume in ATgroup pair_entry_count Number of registered pairs in ATgroup allowed_response_time Allowance time for MV delay that is an ATgroup attribute special_file_name Special file name ld_name Logical disk name type Use format activity Pair s activity state sync Pair s sync state copy Copy control state YYYY/MM/DD hh:mm:ss <state> Start Time indicates this operation s start date/time and <state> End Time indicates its end date/time (<state> differs according to the state) separate_diffsize Copy differential quantity of volume when separate state occurs copy_diffsize Copy differential quantity of volume when replicate or restore state occurs rvacc RV access restriction prev_state Pair s sync state before state transition For detailed description of this command s display items and return values, and environment variables, refer to the ControlCommand Command Reference. For description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for displaying ATgroup s copy state (i) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be displayed belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. However, if the target ATgroup does not include any registered pairs and if a disk array set to the ATgroup can be accessed, only information about that ATgroup is displayed. (ii) Creation and updating of the volume list must be completed. 150

159 Chapter 4 Software Operation Methods (iii) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (iv) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. However, the above conditions (i) to (iii) are not applied to the operation via ism. Notes on displaying the ATgroup copy state (i) On a UNIX system, only a super user can run this command. 151

160 Chapter 4 Software Operation Methods Displaying the ATgroup List (ismrc_sense) The ismrc_sense command is used to display an ATgroup list. This command is used in order to display an ATgroup list or setting information about a particular ATgroup. Executable construct of ismrc_sense command (Executable construct for displaying ATgroup list) ismrc_sense -atg [ATgroup name] [ -site conc dist ] [ -linkarrayname] [ -arrayname disk array name ] [ -iopath direct manager ] (Executable construct for displaying specific volume information) ismrc_sense -vol volume -volflg vol_flg [ -attr ] [-protect ] [ -iopath iopath ] ismrc_sense -file file_name [ -attr ] [-protect ] [ -iopath iopath ] (Executable construct of command version check) ismrc_sense -ver (Executable construct of command usage display) ismrc_sense -?... (Windows) ismrc_sense -help... (UNIX) The following describes the displaying of ATgroup lists. For description of displaying specific volume information, refer to the ControlCommand Command Reference. Specifiable options and parameters for ATgroup list display (i) Specification of ATgroup information (options must be specified, parameters are not required) -atg [ATgroup name] This is specified to fetch ATgroup information. If no parameters are specified, the ATgroup list is displayed. To display particular ATgroup s information, specify the target ATgroup s name as ATgroup name. Specification of this option is required in order to display an ATgroup list. (Specification of parameters is not required.) (ii) Specification of site information -site conc dist This specifies the site (concentrator or distributor) of the ATgroup whose information will be displayed. Specify conc to display an ATgroup connected to the 152

161 Chapter 4 Software Operation Methods concentrator side (MV side), or dist to specify an ATgroup connected to the distributor side (RV side). If this option is omitted, all accessible ATgroup information on both the concentrator side (MV side) and distributor side (RV side) will be displayed. An error will occur if incorrect site information is entered in order to display particular ATgroup s information, or in order to specify a disk array name. (iii) Specification of disk array name -arrayname disk array name This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. This option is used to explicitly specify the target disk array. If this option is omitted, the target disk array is automatically searched. (iv) Specification of link disk array name -linkarrayname This specifies the disk array name of the link disk array configuring the ATgroup. (v) Specification of I/O issue path -iopath direct manager This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. Information displayed when displaying ATgroup list (Windows) [Displayed information] No. ATG Name Site Entry Disk Array Name xxxxh atgroup site entry diskarray_name Specifying -linkarrayname can display the disk array name of the link disk array configuring the ATgroup. 153

162 Chapter 4 Software Operation Methods [Displayed information] No. ATG Name Site Entry Disk Array Name Link Disk Array Name xxxxh atgroup site entry diskarray_name link_diskarray_name Description of display items xxxx Concentrator number (a unique number is assigned to each concentrator, for each related disk array) atgroup ATgroup name site ATgroup s location (concentrator or distributor) entry Number of registered pairs in ATgroup diskarray_name Disk array name link_diskarray_name Link disk array name For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [Displayed information] No. ATG Name Site Entry Disk Array Name xxxxh atgroup site entry diskarray_name Specifying -linkarrayname can display the disk array name of the link disk array configuring the ATgroup. [Displayed information] No. ATG Name Site Entry Disk Array Name Link Disk Array Name xxxxh atgroup site entry diskarray_name link_diskarray_name 154

163 Chapter 4 Software Operation Methods Description of display items xxxx atgroup site entry diskarray_name link_diskarray_name Concentrator number (a unique number is assigned to each concentrator, for each related disk array) ATgroup name ATgroup s location (concentrator or distributor) Number of registered pairs in ATgroup Disk array name Link disk array name For detailed description of this command s display items and return values as well as environment variables, refer to the ControlCommand Command Reference. For description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for displaying ATgroup list (i) If the operating mode from the system is set to secure mode in the disk array to which the ATgroup to be displayed belongs or in the link destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list and must be accessible. If the operating mode from the system is set to normal mode, one or more of the logical disks to which the ATgroup belongs must be registered in the volume list and must be accessible. However, if the target ATgroup does not include any registered pairs and if a disk array set to the ATgroup can be accessed, only information about that ATgroup is displayed. (ii) Creation and updating of the volume list must be completed. (iii) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (iv) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. However, the above conditions (i) to (iii) are not applied to the operation via ism. Notes on displaying the ATgroup list (i) On a UNIX system, only a super user can run this command. 155

164 Chapter 4 Software Operation Methods Building an ATgroup (ismrc_atg) The ismrc_atg command is used to build ATgroups. This command is used to create or delete an ATgroup, to change attributes, or to register or delete a volume pair to an ATgroup. Executable construct of ismrc_atg command (Executable construct for building an ATgroup) ismrc_atg -create -delete -add -remove -update -atg ATgroup name [ -force ] [-mv volume -mvflg ld drv mdir mvol sfn [-rv volume rvflg ld drv mdir mvol sfn ] -file file ] [-arrayname disk array name ] [-linkarrayname link disk array name ] [ -allowedresponse allowed response time of MV ] [ -resync resynchronizemode [-resynctype resynchronizetype] ] [ -iopath direct manager ] (Executable construct of command version check) ismrc_atg -ver (Executable construct of command usage display) ismrc_atg -?... (Windows) ismrc_atg -help... (UNIX) Specifiable options and parameters for build ATgroup (i) Specification of operation (required) -create -delete -add -remove -update This specifies whether the operation will be to create or delete an ATgroup, to change attributes, or to register or delete a pair in an ATgroup. When -create is specified, the target ATgroup will be created. When -delete is specified, the target ATgroup will be deleted. When -add is specified, a volume pair will be registered to a previously created ATgroup. When -update is specified, the attribute of the ATgroup is updated. When -remove is specified, a previously registered pair will be deleted from an ATgroup. Specification of one of these options is required. (ii) Specification of ATgroup (required) -atg ATgroup name This specifies the ATgroup to be built. Enter the name of the ATgroup to be built as ATgroup name. The ATgroup name can include up to 32 characters consisting of alphanumeric characters, underbars (_) and/or forward slashes (/). This option must be 156

165 Chapter 4 Software Operation Methods specified in order to build an ATgroup. (iii) Specification of forced deletion (required for Forced deletion of ATgroup) -force This specifies that the target ATgroup will be forcibly deleted. This option is valid only when deletion of an ATgroup has been specified. Forced deletion of ATgroup is specified when an ATgroup becomes abnormal due to a fault, etc., or when invalid ATgroup information remains on the concentrator side (MV side) or distributor side (RV side). (iv) Specification of pair to be registered or delete (required for registration or deletion of volume pair in ATgroup) -mv volume -mvflg ld drv mdir mvol sfn [-rv volume -rvflg ld drv mdir mvol sfn] -file file These options specify the volume pair to be registered to or deleted from an ATgroup. These options are valid only when specifying either -add to register a pair or -remove to delete a previously registered pair, in which case these options are required. The value specified in volume indicates the type of volume for the -mvflg or -rvflg option. When ld is specified in -mvflg or -rvflg, the volume specification is a logical disk name. When drv is specified, (for Windows only), the volume specification becomes a drive letter. When mdir is specified (for Windows only), the volume specification becomes the mount destination for the NTFS folder. When mvol is specified (for Windows only), the volume specification is the mounting point volume name. When sfn is specified (for UNIX only), the volume specification becomes a special file name. When only one pair has been registered in a configuration, specification of -rv and -rvflg can be omitted for the specified MV. Instead of -mv, -mvflg, -rv, or -rvflg option, the pair information can be entered in the -file option to enable specification of the replication control file. An example of a replication control file specification is shown below. For details, refer to the ControlCommand Command Reference. Format of replication control file MV_Type:MV RV_Type:RV 157

166 Chapter 4 Software Operation Methods Description of replication control file MV_Type This specifies the volume format specified for the MV. Formats that can be specified to build an ATgroup are: ld, drv (Windows only), mdir (Windows only), mvol (Windows only), and sfn (UNIX only). MV This specifies the MV_Type format for the MV volume. RV_Type This specifies the volume format for the RV. Formats that can be specified to build an ATgroup are: ld, drv (Windows only), mdir (Windows only), mvol (Windows only), and sfn (UNIX only). RV This specifies the RV_Type format for the RV volume. (v) Specification of disk array name (required for creation or deletion of ATgroup) -arrayname disk array name This specifies the target disk array. Enter the name of the target disk array (that can be accessed for operations) as disk array name. When creating an ATgroup or registering a pair to an ATgroup, specify the name of the disk array on the concentrator side (MV side) which will be accessible. This option is required when creating or deleting an ATgroup. This option is used to explicitly specify the target disk array when registering or deleting a pair in an ATgroup. (vi) Specification of link disk array name (required for creation of ATgroup) -linkarrayname link disk array name This specifies the disk array name of the link (distributor side) disk array when creating an ATgroup with the disk array on the concentrator side which supports the function specifying the link disk array. With the disk array which does not support this function, it is not required to specify this option. And it is ignored even if specified. The supporting situation of the function specifying the link disk array can be checked by viewing the disk array properties of the disk array to be created from the replication screen of ism. (vii) Specification of allowed response time of MV -allowedresponse allowed response time of MV This specifies, in seconds, the allowed response time of MV that is an ATgroup attribute. A value from 0 to 18 seconds (integer) can be specified. The default value is 18 seconds. 158

167 Chapter 4 Software Operation Methods (viii) Specification of the resynchronize mode -resync resynchronizemode The mode of the resynchronize, one of the following can be specified. on Enable the resynchronize mode of the ATgroup. off Disable the resynchronize mode of the ATgroup. forceoff Force to disable the resynchronize mode of the ATgroup. resume Start up the automatic resynchronization function of the ATgroup which is in the failure state. The automatic resynchronization function will be started up during one minute. The value can be specified with on or off only while the -create parameter is specified. If the value is omitted while the -create parameter is specified, it will be specified with the off by default. (ix) Specification of the resynchronize type -resynctype resynchronizetype The mode of the resynchronize, one of the following can be specified. ddr Specify the type with DDR. snapshot Specify the type with Snapshot. The value can be specified only while the -resync parameter is set by on. If the value is omitted while the -resync parameter is set by on, it will be set with the Snapshot type by default. (x) Specification of I/O issue path -iopath direct manager This specifies the I/O issue path of replication operation for disk arrays. When specifying direct, the I/O issue is directly performed. When specifying manager, the I/O issue is performed via ism. If this option is omitted, the value specified in IOPATH of the replication operation option setting file (Windows) or RPL_IOPATH of the environment variables (UNIX) is valid. The default value is direct. 159

168 Chapter 4 Software Operation Methods Information displayed when building an ATgroup (Windows) [End message] ism13247: Command has completed successfully. (code=pppp-xxxx-yyyy-zzzz) Description of display items pppp-xxxx-yyyy-zzzz This differs according to the processing. For detailed description of this command s display items and return values, and for details on the replication control option settings file, refer to the ControlCommand Command Reference. In addition, for description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. (UNIX) [End message] ism13247: Command has completed successfully. (code=pppp-xxxx-yyyy-zzzz) Description of display items pppp-xxxx-yyyy-zzzz This differs according to the processing. For detailed description of this command s display items and return values, and environment variables, refer to the ControlCommand Command Reference. For description of the end message that is displayed when an abnormal end occurs, refer to the Messages Handbook. Conditions for building an ATgroup (i) When creating an ATgroup, the disk array on the target ATgroup s concentrator side (MV side) must be accessible. (ii) When creating an ATgroup, the ATgroup name you enter cannot exceed 32 characters, can include only alphanumeric characters, underbars (_), and/or forward slashes (/), and must be unique within the system. (iii) When creating an ATgroup with the disk array on the concentrator side which supports the function specifying the link disk array, it is required to specify the link disk array using the -linkarrayname option. (iv) When deleting an ATgroup, the disk array on the target ATgroup s concentrator side (MV side) must be accessible. However, if using the -force option to set forced deletion, either the disk array on the concentrator side (MV side) or the disk array on the distributor side (RV side) of the ATgroup where the forced operation is executed can be accessible. 160

169 Chapter 4 Software Operation Methods (v) To delete an ATgroup, all registered pairs in that ATgroup must be deleted, and the ATgroup s sync state must become invalid. (vi) To register a volume pair to an ATgroup, the target pair must meet the following conditions. DDR pairs cannot be registered to an ATgroup. (Only RDR pairs can be registered to an ATgroup.) A pair that is not the top-level pair cannot be registered to an ATgroup. (Only the top-level pair can be registered.) If the MV of the pair to be registered includes several RDR pairs, it cannot be registered to an ATgroup. Each RDR pair must be set as a single pair registered to an ATgroup. A registered pair in an ATgroup must be separate from the pair to be registered to an ATgroup. An RDR pair that exists among different disk arrays cannot be registered to a single ATgroup. (The RDR pair must be in the same disk array.) If the RV of the RDR pair to be registered is used as a BV (base-volume) of the snapshot function, it cannot be registered to an ATgroup (except the disk array which supports the RDR/DR automatic resynchronization function). If the MV or RV of the RDR pair to be registered is protected by the data retention function, it cannot be registered to an ATgroup. (vii) When registering a pair to an ATgroup, the target pair s MV or RV must be registered to a volume list and must be accessible. (viii) When registering a pair to an ATgroup, the ATgroup sync state must be invalid, separated, or fault separation. (ix) When deleting a pair from an ATgroup, the ATgroup sync state must be either separated or fault separation. (x) When executing Forced deletion of ATgroup on the concentrator side (MV side), ATgroup sync state must be invalid, separated, or fault separation. (xi) When updating the allowed response time of MV that is an ATgroup attribute, the ATgroup sync state must be Invalid, Separated, or Fault. (xii) Creation and updating of the volume list must be completed. (xiii) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress). (xiv) There must not be any frozen replication functions in any disk array that belongs to the target ATgroup. (xv) When creating an ATgroup and registering a pair to an ATgroup, the link state for the disk array must be normal. (xvi) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. (xvii) If any of the following functions is executed when operating mode from the system is set to secure mode in the disk array to which the ATgroup to be operated belongs or in the link 161

170 Chapter 4 Software Operation Methods destination disk array, the MVs or RVs of all the pairs in the ATgroup must be registered in the volume list: Forced deletion of ATgroup Deletion of registered pair from ATgroup Pair registration in ATgroup However, the above conditions (vii), (xi), (xii), and (xvii) are not applied to the operation via ism. Notes on building an ATgroup (i) Forced deletion of ATgroup (using the -force option) is executed when the ATgroup becomes abnormal due to a fault, etc., and invalid ATgroup information remains on the concentrator side (MV side) or distributor side (RV side). When invalid ATgroup information remains on the concentrator side (MV side), separate the ATgroup or perform Forced separation of ATgroup on the concentrator side (MV side) to enable the ATgroup to be deleted. Next, execute Forced deletion of ATgroup to delete the invalid ATgroup information. At this time, it may take several minutes to complete Forced deletion of ATgroup. (ii) On a UNIX system, only a super user can run this command. 162

171 Chapter 4 Software Operation Methods Remote Operation Commands Replication control and display commands (ismrc_replicate, ismrc_separate, ismrc_restore, ismrc_change, ismrc_wait, ismrc_query, ismrc_sense, ismrc_updprevent, and ismrc_swap) can be executed for remote pairs that have already been set. Remote operations cannot be executed using commands for disk array information display, pair setting, and ATgroup, or commands that display server information (ismrc_arrayinfo, ismrc_ldlist, ismrc_pair, and ismvollist). Executable construct of remote operation commands for pairs The executable construct of remote operation commands for pairs is the same as for commands that involve ordinary pairs. The pair targeted for remote operations can be specified in command parameters. However, when a remote operation is executed for a pair, only the logical disk name can be specified in the pair specification. For description of executable constructs related to operations involving ordinary pairs, refer to the Data Replication User s Manual (Function Guide) and the ControlCommand Command Reference. Range of remote operation commands for pairs Remote operation commands for pairs can be used for operations involving linked pairs in a range that includes the accessible volumes in the command s target pair and extends to the next level of disk arrays. The following figures illustrate configurations of pairs that can be used in remote operations and configurations of pairs that cannot be used in remote operations. Operation Range for the ATgroup The operation commands for the ATgroup can only be executed from the server that has access to the MV or RV of the pair belonging to the ATgroup (main site s application server or backup site s standby server). Note that the command to build the ATgroup (ismrc_atg) can only be executed from the main site. The following shows configuration examples of pairs that can be operated, pairs that cannot be operated, and ATgroup operation range. 163

172 Chapter 4 Software Operation Methods This first figure shows pairs for which remote operation commands can be executed from the main site. Main site Application server Backup site Standby server Operation instruction MV (i) RV/MV Operation instruction (ii) RV MV (iii) RV/MV (iv) RV/MV (v) RV Operation instruction MV (vi) (vii) RV/MV (viii) RV RV Disk array Disk array [Description of pairs available for operations] (i) RDR pairs accessible via MV (ii) DDR pairs linked via disk array of this command s target pair MV and next level s disk array (iii) DDR pairs accessible via MV (iv) RDR pairs linked via disk array of this command s target pair MV (v) DDR pairs linked via disk array of this command s target pair MV and next level s disk array (vi) DDR pairs accessible via RV (vii) RDR pairs linked via disk array of this command s target pair RV (viii) DDR pairs linked via disk array of this command s target pair RV and next level s disk array Figure 4-23 Pairs Available for Remote Operation Commands Executed from Main Site 164

173 Chapter 4 Software Operation Methods The next first shows pairs for which remote operation commands can be executed from a backup site. Main site Application server Backup site Standby server MV (i) Operation instruction RV/MV (ii) RV Operation instruction MV (ii) RV (iii) RV MV (iv) RV/MV (v) RV/MV (vi) Operation instruction RV MV (vii) (v) RV/MV (vi) Operation instruction RV RV Disk array Disk array [Description of pairs available for operations] (i) DDR pairs linked via disk array of this command s target pair RV and next level s disk array (ii) RDR pairs accessible via RV (iii) DDR pairs linked via disk array of this command s target pair RV and next level s disk array (iv) DDR pairs linked via disk array of this command s target pair RV and next level s disk array (v) RDR pairs linked via disk array of this command s target pair RV (vi) DDR pairs accessible via RV (vii) DDR pairs linked via disk array of this command s target pair RV and next level s disk array Figure 4-24 Pairs Available for Remote Operation Commands Executed from Backup Site 165

174 Chapter 4 Software Operation Methods This figure shows pairs for which remote operation commands cannot be executed. Main site Backup site (1) Application server Standby server (1) Backup site (2) Standby server (2) Operation instruction MV RV/MV (i) RV Operation instruction MV (ii) RV/MV Operation instruction RV MV RV/MV (iii) RV Disk array Disk array Disk array [Reasons why remote operation commands cannot be executed] (i) Disk array is two levels from MV for this command s target pair. (ii) Disk array is two levels from RV for this command s target pair. (iii) Disk array is two levels from MV for this command s target pair. Figure 4-25 Pairs for which Remote Operation Commands Cannot be Executed 166

175 Chapter 4 Software Operation Methods This figure shows the operation range for the ATgroup. Main site Application server Backup site Standby site Operation instruction Operation instruction (i) MV ATgroup (ii) RV MV RV MV RV Disk array Disk array [Description of ATgroup that can be operated] (i) ATgroup containing the MV accessible from the main site. However, if the operating mode of the main site from the application server is set to secure mode, the MVs of all the pairs in the ATgroup must be registered in the volume list. (ii) ATgroup containing the RV accessible from the backup site. However, if the operating mode of the main site from the standby server is set to secure mode, the RVs of all the pairs in the ATgroup must be registered in the volume list. Figure 4-26 Operation Range for the ATgroup 167

176 Chapter 4 Software Operation Methods Conditions for execution of remote operation commands (i) Remote operation commands can be used for operations involving linked pairs in a range that includes the accessible volumes in the command s target pair (which have been registered in the volume list) and extends to the next level of disk arrays. (ii) The target pairs for remote operations must be specified in the command s parameters as the target pairs. (iii) Logical disk names must be specified for remote operation pairs. (iv) Replication functions must not be frozen in the disk arrays for the pairs for which remote operations are executed. (v) When remote operations are executed for pairs at the next level, the link state for the disk array must be normal. (vi) Creation and updating of the volume list must be completed. (vii) If a UNIX server is used, creation of an ism special file must not be in progress (execution of the ismmknod command must not be in progress) (viii) If a Windows server is used, the execution user must belongs to Administrators group. However, in a Windows Server 2008 environment in which User Account Control (UAC) is enabled, to log on as a user other than the user belonging to the OS local system administrators group and perform operations, elevate to the system administrator in advance. 168

177 Chapter 5 Operation Examples Chapter 5 Operation Examples If a disaster occurs at the main site in a disaster recovery system, operations executed for the scenarios described below must be taken into consideration to enable recovery of the assumed essentials for business. For example, in a scenario where an equipment fault occurs during recovery from a disaster, one must consider what operation would be needed to perform a time-based backup at a backup site. In a scenario where an equipment fault or line fault occurs and then a disaster occurs, one must devise a recovery method that prevents the loss of the latest backup data. Key points to consider for operations under various scenarios are listed in the following table. Scenario Table 5-1 Considerations for Operations Under Various Scenarios Considerations for Operations During normal operations When an equipment fault or line fault occurs When a disaster occurs When restoring from backup site to main site Copy mode and sync state during operations Time-based backup Switchover of business tasks at scheduled times or at night, etc. Steps for recovery from line fault, Steps for recovery from disk fault (MV or RV) Steps for recovery from server fault Countermeasures to a disaster following an equipment fault or line fault Fault detection and reporting methods Etc. Recovery steps at backup site Operation of backup site after recovery Countermeasures to a disaster at a backup site Disaster detection and reporting methods Etc. Data restoration steps Etc. The considerations for these operations differ somewhat depending on the specific disaster recovery configuration and the content of business tasks. The steps described below are for a scenario in which the current copy mode is semi synchronous copy mode of order guarantee and the current sync state is continual replication. Backup during normal operations Recovery steps from line fault (after a line fault occurs) Steps for use of RV data in ATgroup after a disaster Restoration of data from MV to RV when recovering from backup site to main site 169

178 Chapter 5 Operation Examples 5.1 Backup During normal operations, DynamicDataReplication and tape backup are performed periodically at the main site and backup site. When recovering business operations at a backup site after a disaster, RVs in the ATgroup are used for data recovery. The backup data is used for data recovery from a partial fault at the main site, or for data recovery as part of business recovery after a fault has occurred at a backup site. For example, DynamicDataReplication or backup data on tape (backed up at the main site) is used when a fault has occurred among some of the logical disks at the main site. The following describes a configuration example in which the tape backup steps include the use of DynamicDataReplication at various sites. 170

179 Chapter 5 Operation Examples (1) Overview of backup steps The following configuration example is used to describe a set of backup steps. During normal operations, the ATgroup sync state is replicate synchronous (Rpl/sync) and the copy mode is semi synchronous copy mode of order guarantee. DDR pairs are set up at the main site and backup site, and during normal operations the activity state of each site s DDR pairs is separate and the sync state is separated. Backup consists of tape backup that is performed after the DDR pause point data has been secured by replication control. During these steps, business operations are temporarily paused in order to secure pause points. Main site Backup site Backup server Application server Standby server Backup server Tape Tape DB_Group DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array [Status of ATgroup and DDR pair before backup] ATgroup s copy mode: Semi synchronous copy mode of order guarantee MV access after atomic-break: MV access continue ATgroup consistency state: Atomic ATgroup sync state: Replicate synchronous (Rpl/sync) Link state: Normal State of DDR pair: Separated Figure 5-1 Configuration Example of Backup Steps 171

180 Chapter 5 Operation Examples (2) Flow of Backup Steps [Operations at main site] [Operations at backup site] Step 1a. server) Unmount DDR pair s RV (backup Step 1b. server) Unmount DDR pair s RV (backup Step 2a. Replicate DDR pairs (application server) Step 2b. Replicate DDR pairs (standby server) Step 3. Secure pause point (application server) Step 4. Separate ATgroup (application server) Step 5a. Separate DDR pair (application server) Step 5. Separate DDR pair (standby server) Step 6. Replicate ATgroup (application server) Step 7. Resume business tasks (application server) Step 8a. Mount DDR pair s RVs (backup server) Step 8b. Mount DDR pair s RVs (backup server) Step 9a. Backup to tape (backup server) Step 9b. Backup to tape (backup server) * The flow of these steps and the following operation steps are provided as an example. Be sure to adjust the specific operations for the particular environment. Figure 5-2 Flow of Backup Steps 172

181 Chapter 5 Operation Examples (3) Operation Steps Step 1. Unmount DDR pair s RV (main site and backup site s backup server) (Windows) At this step, the file system buffers are cleared in the respective DDR pair s RVs which are recognized by the main site and backup site s backup server. The server that is already accessing the target volume is stopped, the file system is flushed and unmounted, and then the file system s buffers are cleared. (i) Stop the service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system and clear the file system buffers. Since the volumes to be unmounted are RVs, their drive path (mounted to drive letter or NTFS folder) is automatically deleted so that these RVs cannot be accessed. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (Linux) The respective DDR pair s RVs, which are recognized by the main site and backup site s backup server, are unmounted. Since the RV information is overwritten over the MV information, if an LVM is used, the volume group information is temporarily deleted. (i) Unmount the file system. (Command example) umount /db umount /log (ii) Either set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 vxdisk offline sde vxdisk offline sdf 173

182 Chapter 5 Operation Examples Step 2. Replicate DDR pairs (application server and standby server) The main site and backup site DDR pairs are replicated. Since the backup site s DDR pairs are serially configured pairs that are simultaneously replicated, the -nowait option is set and replications performed during background copy mode. (Command example at main site) ismrc_replicate -file pair_ddr.txt -wait * pair_ddr.txt setting example #Type:MV Type:RV ld:db_mv ld:db_bk_local ld:log_mv ld:log_bk_local (Command example at backup site) ismrc_replicate -file pair_ddr_r.txt -cpmode bg -nowait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote Step 3. Secure pause point (application server) (Windows) A pause point is secured by the application server. Specifically, business operations and other aspects of the current service that access the target volume are stopped, the business volume s file system is flushed and unmounted, and a pause point is secured. (i) Stop business operations. (ii) Stop service. (Command example) NET STOP ESMCommonService (iii) Flush and unmount the file system. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: ismrc_umount -drv F: (Linux) A pause point is secured by the application server. Specifically, business operations are stopped, the business volume is unmounted, either the volume group is set as inactive or the disk group is deported, and a pause point is secured. (i) Stop business operations. (ii) Unmount the file system. (Command example) umount /db 174

183 Chapter 5 Operation Examples (iii) umount /log The volume group is set as inactive or the disk group is deported. If LVM is used, set the volume group as inactive. If VxVM is used, deport the disk group. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 Step 4. Separate ATgroup (application server) The ATgroup is separated in order to secure pause point data in the ATgroup s RVs at the backup site. (Command example) ismrc_separate -atg DB_Group -wait Step 5. Separate DDR pair (application server and standby server) A DDR pair is separated to secure pause point data in the RV of the DDR pair where backup data will be secured. (Command example at main site) ismrc_separate -file pair_ddr.txt -wait * pair_ddr.txt setting example #Type:MV Type:RV ld:db_mv ld:db_bk_local ld:log_mv ld:log_bk_local (Command example at backup site) ismrc_separate -file pair_ddr_r.txt -wait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote Step 6. Replicate ATgroup (application server) ATgroup is replicated and then the ATgroup is set to normal consistency state. (Command example) ismrc_replicate -atg DB_Group -cpmode semi -wait 175

184 Chapter 5 Operation Examples Step 7. Resume business tasks (application server) (Windows) All business tasks that were stopped to secure a pause point are resumed. Specifically, the business volume is mounted, stopped service is restarted, and then business tasks are resumed. (i) Mount the file system. (Command example) ismrc_mount -drv E: ismrc_mount -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (iii) Resume business tasks. (Linux) Business tasks that were stopped to secure a pause point are resumed. Specifically, the volume group is set as active or the disk group is imported, the business volume is mounted, and then business tasks are resumed. (i) Activate the volume group or import the disk group. If LVM is used, the volume group is set as active. If VxVM is used, the disk is imported. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdg import dg01 vxdg import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Mount the file system. (Command example when using LVM and mounting VxFS s file system) mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when using VxVM and mounting VxFS s file system) mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log (iii) Resume business tasks. 176

185 Chapter 5 Operation Examples Step 8. Mount DDR pair s RVs (main site and backup site s backup server) (Windows) DDR pair RVs that are recognized by the backup server are mounted. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RVs. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (Linux) DDR pair RVs that are recognized by the backup server are set as an active volume group or the disk group is imported and then mounted. If LVM is used, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online sde vxdisk online sdf vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Mount the file system. (Command example when using LVM and mounting VxFS s file system) mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when using VxVM and mounting VxFS s file system) mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log 177

186 Chapter 5 Operation Examples Step 9. Backup to tape (main site and backup site s backup server) Backup software is used to back up to tape data that was previously copied to an RV. 178

187 Chapter 5 Operation Examples 5.2 Line Fault Recovery When a line fault occurs, a link fault is detected by ism. Once a link fault has been detected, a log with the message ID ism10402 from ism is recorded. For details on the ism logs, refer to the Messages Handbook. If a line fault makes it impossible to guarantee the order when writing to RVs in an ATgroup, an atomic-break is executed. After recovery from a line fault, the ATgroup is replicated and the consistency state is returned to normal consistency state. However, consistency in the ATgroup RVs is temporarily lost during the period from replication start until replicating (i.e., from when the ATgroup sync state is replicate start (Rpl/start) until it is replicating (Rpl/exec). If the risk of a disaster occurring during ATgroup replication must be considered, then before restarting ATgroup replication any RV that has data consistency during execution of an atomic-break must be backed up, such as by using DynamicDataReplication. Even for the ATgroup for which the automatic resynchronization function is enabled, the atomic break that caused by a line fault cannot be recovered by the automatic resynchronization function. Fix the line fault, and then restart the automatic resynchronization function. The following description is based on a configuration of line fault recovery steps. Figure 5-3 Link Fault Display by ism when Line Fault has Occurred 179

188 Chapter 5 Operation Examples (1) Overview of Line Fault Recovery Steps The following configuration example is used to describe line fault recovery steps. In this example, an atomic-break is assumed to be executed when a line fault occurs during operations where the ATgroup sync state is replicate synchronous (Rpl/sync) and the ATgroup s copy mode is semi synchronous copy mode of order guarantee. After recovering from the line fault, consistent data that was saved immediately after the line fault was saved from the backup site to a DDR RV (in this case, the logical disk name is DB_BK_REMOTE, LOG_BK_REMOTE), is used to again replicate the ATgroup so that recovery is to the original consistency state. Main site Backup site Backup server Application server Standby server Backup server Tape Tape DB_Group DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array [ATgroup and DDR pair states during line fault recovery] ATgroup s copy mode: Semi synchronous copy mode of order guarantee MV access after atomic-break: MV access continue ATgroup consistency state: Atomic Concentrator s ATgroup sync state: Fault separation (Fault) Distributor s ATgroup sync state: Replicate synchronous (Rpl/sync) Link state: Fault DDR pair state: Separated Figure 5-4 Example of Line Fault Recovery Steps 180

189 Chapter 5 Operation Examples (2) Flow of Line Fault Recovery Steps (example) [Operations at main site] [Operations at backup site] Step 1. Execute Forced separation of ATgroup (standby server) Step 2. Unmount DDR pair s RV (backup site s backup server Step 3. Replicate DDR pair (standby server) Step 4. Separate DDR pair (standby server) Step 5a. Replicate ATgroup (application server) Step 5b. Mount DDR pair s RV (backup site s backup server) * This flow of steps and the operation steps described below are just examples. Be sure to perform operations and steps that are suited to the actual environment. Figure 5-5 Flow of Line Fault Recovery Steps (Example) 181

190 Chapter 5 Operation Examples (3) Operation Steps Step 1. Execute Forced separation of ATgroup (standby server) First, separate the RVS in the ATgroup in order to save consistent data immediately after the line fault occurs. Because the link state is a fault condition, forced separation must be performed to separate the RVs in the ATgroup. (Command example) ismrc_separate -atg DB_Group -force -wait This can also be done using ism at the backup site. Figure 5-6 Execution of Forced separation of ATgroup by ism (Example) Step 2. Unmount DDR pair s RV (backup site s backup server) (Windows) In order to save consistent data immediately after the line fault to a DDR RV, the file system buffers are cleared in the respective DDR pair s RVs which are recognized by the backup site s backup server, to prepare for replication of DDR pairs at the backup site. Specifically, the current service that accesses the target volume is stopped, the file system is flushed and unmounted, and then the file system buffers are cleared. (i) Stop service. (Command example) NET STOP ESMCommonService (ii) File system is flushed and unmounted and file system buffers are cleared. Since the volumes to be unmounted are RVs, their drive path (mounted to drive letter or NTFS folder) is 182

191 Chapter 5 Operation Examples automatically deleted so that these RVs cannot be accessed. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (Linux) In order to save consistent data immediately to a DDR RV after the line fault, respective DDR pair s RVs, which are recognized by the main site and backup site s backup server, are unmounted to prepare for replication of DDR pairs at the backup site. Since the RV information is overwritten over the MV information, if LVM is used, the volume group information is deleted. (i) The file system is unmounted. (Command example) umount /db umount /log (ii) The volume group is set as inactive or the disk group is deported. If LVM is used, volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 vxdisk offline sde vxdisk offline sdf 183

192 Chapter 5 Operation Examples Step 3. Replicate DDR pair (standby server) The backup site s DDR pairs are replicated, and the data is then copied. In this case, it is assumed that the ATgroup sync state of the host ATgroup s distributor side (RV side) has been changed by ism to fault separation. (Command example) ismrc_replicate -file pair_ddr_r.txt -wait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote This can also be done using ism. Figure 5-7 Execution of DDR Pair Replication by ism (Example) 184

193 Chapter 5 Operation Examples Step 4. Separate DDR pair (standby server) The DDR pair is separated in order to save consistent data to the DDR pair s RV immediately after a line fault. (Command example) ismrc_separate -file pair_ddr_r.txt -wait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote This can also be done using ism. Figure 5-8 Execution of DDR Pair Separation by ism (Example) 185

194 Chapter 5 Operation Examples Step 5a. Replicate ATgroup (application server) ATgroup is replicated and then the ATgroup is set to normal consistency state. Consistent of RV data in the ATgroup is lost until the ATgroup sync state becomes replicate synchronous (Rpl/sync). If a disaster occurs during this period, the data is recovered from the DDR pair s RV that was secured at Step 4. (Command example) ismrc_replicate -atg DB_Group -cpmode semi -wait This can also be done using ism. Figure 5-9 Execution of ATgroup Replication by ism (Example) 186

195 Chapter 5 Operation Examples Step 5b. Mount DDR pair s RV (backup site s backup server) (Windows) DDR pair RVs that are recognized by the backup server is mounted and normal consistency state is restored. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RVs. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (Linux) DDR pair RVs that are recognized by the backup site s backup server are set as an active volume group or the disk group is imported and then mounted, after which the normal consistency state is restored. This DDR pair s RV data is data that is copied after being mounted in the application server. Accordingly, after the volume group is set as active or the disk group is imported, the volume s mount bits must be cleared. When using LVM, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) (ii) Activate the volume group or import the disk group. When using LVM, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online sde vxdisk online sdf vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall Clear the volume s mount bits. (Command example when clearing mount bits while using LVM and VxFS file system) fsck -t vxfs /dev/vg01/lvol1 187

196 Chapter 5 Operation Examples (iii) fsck -t vxfs /dev/vg02/lvol1 (Command example when clearing mount bits while using VxVM and VxFS file system) fsck -t vxfs /dev/vx/dsk/dg01/vol1 fsck -t vxfs /dev/vx/dsk/dg02/vol1 Mount the file system. (Command example when mounting VxFS s file system while using LVM) mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when mounting VxFS s file system while using VxVM) mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log 188

197 Chapter 5 Operation Examples 5.3 Use of RV when Disaster Occurs When a disaster occurs at the main site and is followed by a disk array fault or line fault, a link fault is detected by the backup site s ism. Once a link fault has been detected, a log with the message ID ism10402 from ism is recorded. For details on the ism logs, refer to the Messages Handbook. When a disaster occurs at the main site, an RV in the ATgroup is used to recover business operations at a backup site. When using an RV in the ATgroup, first check the ATgroup state to determine whether or not the RV data in the ATgroup is consistent. If the ATgroup consistency state is Atomic, it means the RV data in the ATgroup can be confirmed as consistent and can be recovered. If the ATgroup consistency state is not Atomic, it means the RV data in the ATgroup cannot be confirmed as consistent, so backup data must be used for recovery. When using an RV in the ATgroup, the ATgroup sync state must be separated. To set the ATgroup sync state as separated, first execute separate ATgroup, then ATgroup forced separate, and finally recovery from fault separation. After restarting business operations at the backup site, if the ATgroup s settings do not have to be preserved, the ATgroup should be deleted and its pairs should be cancelled to prevent future operation errors. For the ATgroup for which the automatic resynchronization function is enabled, if the resynchronization failed, the consistent data may be stored in the SV or drv for automatic resynchronization. When both of the following conditions are met, restore the data from the SV or drv for automatic resynchronization to the RV and then recover the business operations on the backup site. (1) The ATgroup consistency state is Non-atomic. (2) The automatic resynchronization state is synchronizing, failure, or resuming. The following describes, based on a configuration example, the steps when using an RV after a fault occurs. 189

198 Chapter 5 Operation Examples Figure 5-10 Link Fault Display by ism After a Disaster 190

199 Chapter 5 Operation Examples (1) Overview of RV use steps after a disaster (example) A configuration example used to describe RV use steps after a disaster is illustrated below. Before the disaster, data was copied while the ATgroup sync state was replicate synchronous (Rpl/sync) and the copy mode was semi synchronous copy mode of order guarantee. When a disaster occurs at the main site, ATgroup state at the backup site is determined, after which RVs in the ATgroup are used. Main site Backup site Backup server Application server Standby server Backup server Tape Tape DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E Disaster occurs DB_MV Mount /dev/vg01 and dg01 in /db or drive E DB_Group DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array [Status of ATgroup and DDR pair before disaster] ATgroup s copy mode: Semi synchronous copy mode of order guarantee MV access after atomic-break: MV access continue ATgroup consistency state: Atomic ATgroup sync state Replicate synchronous (Rpl/sync) Link state: Fault State of DDR pair: Separated Figure 5-11 Configuration Example for RV Use Steps After a Disaster 191

200 Chapter 5 Operation Examples (2) Confirmation of ATgroup State When a disaster has occurred at the main site, prompting use of RVs in the ATgroup at a backup site to restore business operations, it is determined whether or not RVs in the ATgroup can be used, and then the steps for using RVs in the ATgroup are confirmed. These matters are determined by checking the ATgroup consistency state, ATgroup sync state, and link state. the ism replication management functions, ATgroup functions, or ReplicationControl/DisasterRecovery s commands can be used to check the ATgroup consistency state, ATgroup sync state, and link state. The details on these confirmation steps are described below. (i) Confirmation of ATgroup consistency state This step confirms the ATgroup consistency state. When the ATgroup consistency state is Atomic, then the RV data in the ATgroup can be judged as consistent and can be used for recovery. If the ATgroup consistency state is Non-atomic, the link state is normal, and replication control can be executed, then the ATgroup is replicated, separated, and restored in that order. If replication control cannot be executed or if the link state is abnormal, recovery is not possible unless a pause point has been secured, the ATgroup has been separated, and the ATgroup sync state is not separated. (Example of confirmation by command) ismrc_query -atg DB_Group * The ATG State in the command s results is used to confirm the ATgroup consistency state. This can also be confirmed via ism. Figure 5-12 Example of Confirmation of ATgroup Consistency State by ism 192

201 Chapter 5 Operation Examples (ii) Confirmation of link state This step confirms the link state. If the link state is confirmed as normal, the ATgroup is replicated and then separated, so that data from just before a disaster can be used to prevent any data loss. However, when an ATgroup is replicated, if a fault occurs during the period from replication start until replicating (i.e., from when the ATgroup sync state is replicate start (Rpl/start) until it is replicating (Rpl/exec), RVs in the ATgroup can no longer be used. Consequently, before replicating an ATgroup, software such as DynamicDataReplication must be used to back up the RVs in the ATgroup. When the link state is abnormal, if the copy mode is synchronous mode and the MV access after atomic-break setting is MV access stop, the latest data will be retained in the ATgroup s RVs. Some data loss may occur if these settings are different. (Example of confirmation by command) ismrc_query -atg DB_Group * The ATG Path State in the command s results is used to confirm the link state. This can also be confirmed via ism. Figure 5-13 Example of Link State Confirmation by ism 193

202 Chapter 5 Operation Examples (iii) Confirmation of ATgroup sync state This step confirms the ATgroup sync state. The ATgroup sync state is used to determine the recovery steps for using RVs in the ATgroup. (Example of confirmation by command) ismrc_query -atg DB_Group * The ATG State in the command s results is used to confirm the ATgroup sync state. This can also be confirmed via ism. Figure 5-14 Example of ATgroup Sync State Confirmation by ism 194

203 Chapter 5 Operation Examples (3) Steps for Using RV in ATgroup When using RVs in the ATgroup, the ATgroup sync state must be separated. To set the ATgroup sync state as separated, first execute separate ATgroup, ATgroup forced separate, and recovery from fault separation. The following describes the steps for using RVs in an ATgroup when the ATgroup consistency state, link state, and ATgroup sync state are determined to be as described below when confirmed from a backup site. These states are set up in anticipation of a scenario in which a fault has occurred in a server, disk array, or line at the main site. [States confirmed from backup site] ATgroup consistency state: Atomic Link state: Fault ATgroup sync state: Replicate synchronous (Rpl/sync) State of DDR pair at backup site: Separated * Since a disk array fault has occurred, states cannot be checked from the concentrator side (MV side), so the ATgroup sync state checked from a backup site is retained before any disaster occurs. Flow of Steps for Using RV in ATgroup Step 1. Execute forced separation (standby server) Step 2. Execute recovery from fault separation (standby server) Step 3. Mount RVs in ATgroup (standby server) Step 4. Prepare to resume business operations, and resume business operations (standby server) Step 5. Forced deletion of ATgroup and forced unpairing of pair (standby server * Step 5 can be omitted if the main site s disk array is operating normally or if the ATgroup must be set by a host application. * Be sure to execute step 5 for the ATgroup for which the automatic resynchronization function is enabled. * This flow of steps and the operation steps described below are just examples. Be sure to perform operations and steps that are suited to the actual environment. Figure 5-15 Flow of Steps for Using RV in ATgroup 195

204 Chapter 5 Operation Examples Operation Steps Step 1. Execute forced separation (standby server) Once the ATgroup consistency state is Atomic and the ATgroup sync state is replicate synchronous (Rpl/sync) for the ATgroup s distributor side (RV side) which recovers business operations, the ATgroup sync state set as separated. Since the link state is fault, a forced separation is executed first to separate the RVs in the ATgroup. When forced separation is executed, the ATgroup sync state becomes fault separation. (Command example) ismrc_separate -atg DB_Group -force -wait This can also be done using ism at the backup site. Figure 5-16 Execution of Forced separation of ATgroup by ism (Example) 196

205 Chapter 5 Operation Examples Step 2. Execute recovery from fault separation (standby server) Once the ATgroup consistency state is Atomic and the ATgroup sync state is fault separation on the ATgroup s distributor side (RV side) where business operations are performed, recovery from fault separation is executed and the distributor side (RV side) ATgroup s sync state becomes separated. (Command example) ismrc_change -atg DB_Group -force separated This can also be done using ism at the backup site. Figure 5-17 Execution of Recovery from Fault Separation by ism (Example) 197

206 Chapter 5 Operation Examples Step 3. Mount RV in ATgroup (standby server) (Windows) The RVs that were used by the standby server to recover from fault separation are mounted as RVs in the ATgroup. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RVs. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (Linux) The RVs that were used by the standby server to recover from fault separation are mounted as RVs in the ATgroup, and the volume group is set as active or the disk group is imported. The RV data in the ATgroup is data that is copied after being mounted in the application server. Consequently, after the volume group is set as active or the disk group is imported, the volume s mount bits must be cleared. When using LVM on servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on Linux servers, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online sde vxdisk online sdf vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Clear the volume s mount bits. (Command example when clearing mount bits while using LVM and VxFS file system) fsck -t vxfs /dev/vg01/lvol1 fsck -t vxfs /dev/vg02/lvol1 198

207 Chapter 5 Operation Examples (iii) (Command example when clearing mount bits while using VxVM and VxFS file system) fsck -t vxfs /dev/vx/dsk/dg01/vol1 fsck -t vxfs /dev/vx/dsk/dg02/vol1 Mount the file system. (Command example when mounting VxFS s file system while using LVM) mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when mounting VxFS s file system while using VxVM) mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log Step 4. Prepare to resume business operations, and resume business operations (standby server) The standby server is used to recover the cluster configuration, confirm data in the mounted RVs in the ATgroup, revise the network configuration that it uses for business operations, prepare for resuming business operations, and to resume business operations. Step 5. Forced deletion of ATgroup and forced deletion of pair (standby server) After resuming business operations, forced deletion is executed for the ATgroup and forced unpairing for the pair, and then settings that are no longer needed are deleted to prevent operation errors. However, if an ATgroup must be set when the main site s disk array is operating normally, then neither forced deletion of the ATgroup nor forced unpairing is executed. (Command example of Forced deletion of ATgroup) ismrc_atg -delete -atg DB_Group -arrayname REMOTE_ARRAY -force (Command example of forced unpairing of pair) ismrc_pair -unpair -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld -force rv ismrc_pair -unpair -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld -force rv or ismrc_pair -unpair -file pair.txt -force rv * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv This can also be done using ism at the backup site. 199

208 Chapter 5 Operation Examples Figure 5-18 Execution of Forced Deletion of ATgroup by ism (Example) Figure 5-19 Execution of Forced Unpairing of Pairs by ism (Example) 200

209 Chapter 5 Operation Examples 5.4 Copy Back Mode from RV to MV The following describes how, after a disaster has caused business operations to be switched to a backup site, the fault at the main site is repaired and data is restored from an RV to the MV so that business operations can be switched back to the main site. Factors such as the fault state at the main site and the time required for recovery are used to determine whether to simply restore data from an RV to the MV or to swap the pair configuration before restoring data from an RV to the MV Restoration for Copy Back Mode The following describes the method used to restore data from RVs to MVs. During this restoration, business operations can be confirmed and restarted via the main site where such operations they be resumed. However, access to the restored volume (MV) depends on the line speed. In addition, if a line fault occurs before this restoration has been completed, it may become impossible to access normal data on the restored volume (MV). Consequently, business operations are usually merely checked during the restore operation, and once restoration has been completed business operations are resumed at the main site. Note that during restoration business operations cannot be performed at the backup site that is the source for reverting data. Consequently, differences in information between disk arrays might occur, such as when data has been lost in a disk array, so this is not suitable for full-range copying of data from an RV to the MV. Restoring data from an RV to an MV is suitable when differential data between disk arrays has not been lost or when ample time for reverting business operations can be secured. Below, a configuration example is provided for describing the restoration steps for Copy Back Mode. 201

210 Chapter 5 Operation Examples (1) Overview of Restoration Steps for Copy Back Mode The restoration steps for Copy Back Mode are described below. After a disaster occurs, business operations that are being performed via a standby server at a backup site are reverted to the main site s application server and data is copied back. Main site Backup site Application server Standby server Backup server Backup server Tape Tape DB_Group DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E Restore DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F Restore LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array Copy Back Mode by restoration [ATgroup settings after Copy Back Mode] ATgroup s copy mode: Semi synchronous copy mode of order guarantee MV access after atomic-break: MV access continue Figure 5-20 Configuration of Restoration for Copy Back Mode (Example) 202

211 Chapter 5 Operation Examples (2) Flow of Restoration Steps for Copy Back Mode (example) [Operations at main site] [Operations at backup site] Step 1. Repair fault at main site (main site) Step 2a. Create volume list (application server and backup server) Step 2b. Create volume list (standby server and backup server) Step 3. Create new ATgroup (application server) Step 4. Stop business operations (standby server) Step 5a. Unmount restoration volume (MV in ATgroup) (application server) Step 5b. Unmount business volume (RVs in ATgroup) (standby server) Step 6. Restore ATgroup (application server) Step 7. Mount restoration volume (MV in ATgroup) (application server) Step 8. Restore and check business operations (application server) Step 9a. Prepare Copy Back Mode of business operations (main site) Step 9b. Prepare Copy Back Mode of business operations (backup site) Step 10. Wait for completion of ATgroup restoration (application server) Step 11. Replicate ATgroup (application server) Step 12. Resume business operations (application server) * The flow of these steps and the following operation steps are provided as an example. Be sure to adjust the specific operations for the particular environment. Figure 5-21 Flow of Restoration Steps for Copy Back Mode (Example) 203

212 Chapter 5 Operation Examples For steps for Copy Back Mode of RemoteCluster, refer to the RemoteCluster with MC/ServiceGuard User s Manual. (3) Operation Steps Step 1. Repair fault at main site (main site) The fault at the main site is repaired at this step. Software installation and environment settings are also required. Step 2. Create volume list (application server, standby server, and backup server) If the main site s disk array is connected to a backup site s disk array, a volume list is created to include all servers connected to these disk arrays. If the server in the volume list is Linux server running ism, ism must be temporarily stopped. At this step and afterward, the main site s disk array must be connected to a backup site s disk array. (Command example for Windows) ismvollist -cr (Command example for Linux) ismvollist -r * Do not perform the disk signature writing, partition creation, file system creation, and drive letter setting when making the application server recognize the rebuilt MV for the operation using a GPT disk. Step 3. Create new ATgroup (application server) If an ATgroup has been deleted, such as in order to prevent operation errors or as a result of repairing faults in volumes, a new ATgroup is created at this step. If the target ATgroup is valid, this step can be omitted. (Command example for pair settings) ismrc_pair -pair -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_pair -pair -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld or ismrc_pair -pair -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv 204

213 Chapter 5 Operation Examples (Command example of ATgroup creation) ismrc_atg -create -atg DB_Group -arrayname LOCAL_ARRAY -linkarrayname REMOTE_ARRAY (Command example of pair registration to ATgroup) ismrc_atg -add -atg DB_Group -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_atg -add -atg DB_Group -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld or ismrc_atg -add -atg DB_Group -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv These settings can also be made via the main site s ism. Figure 5-22 Pair Settings via ism (Example) 205

214 Chapter 5 Operation Examples Figure 5-23 ATgroup Creation via ism (Example) Figure 5-24 Registration of Pair to ATgroup by ism (Example) 206

215 Chapter 5 Operation Examples Step 4. Stop business operations (standby server) Business operations being handled at the backup site s standby server are stopped to enable their Copy Back Mode. Step 5. Unmount volume (MV or RV in ATgroup) (application server and standby server) (Windows) At this step, file system buffers are cleared in the main site s restored volume (MV in ATgroup) and at the backup site s business volume (RV in ATgroup). The current service that is accessing the target volume is stopped, the file system is flushed and unmounted, and then file system s buffers are cleared. Once these buffers have been cleared, the stopped service can be resumed since the standby server s file system has already been unmounted. (i) Stop the service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system, then clear the file system s buffers. At the standby server, where the target volume for unmounting is an RV, the drive path (mounted to drive letter or NTFS folder) is automatically deleted to disable access to this RV. (Command example for application server) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: ismrc_umount -drv F: (Command example for standby server) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (iii) Start service at standby server. (Command example) NET START ESMCommonService * This step is not required on the application server side for the operation using a GPT disk. (Linux) To prepare for restoration of the ATgroup, the main site s restoration volume (MV in ATgroup) and the backup site s business volume (RV in ATgroup) are unmounted. When restoring the information in the restoration volume (MV in ATgroup), information in the business volumes (RV in ATgroup) is overwritten during the replication that follows information restoration. (i) Unmount the file system. (Command example) umount /db 207

216 Chapter 5 Operation Examples (ii) umount /log Set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 vxdisk offline sde vxdisk offline sdf * If the volume has already been unmounted, such as when repairing a server fault or logical disk fault, this step can be omitted. 208

217 Chapter 5 Operation Examples Step 6. Restore ATgroup (application server) Once the data has been restored, the ATgroup is restored at this step. When restoring the ATgroup, business operations can be confirmed without waiting for restoration to be completed by specifying this command s -nowait option. (Command example) ismrc_restore -atg DB_Group -nowait This can also be done using ism. Figure 5-25 Execution of ATgroup Restoration by ism (Example) For the operation using a GPT disk, the following steps must be executed on the application server before executing Step 7: Restarting the application server Restart the application server to write the disk signature for MV. Recognizing the MV-side disk Start the [Disk Management] of Windows, and assign a drive letter to the disk. 209

218 Chapter 5 Operation Examples Re-creating the volume list Since the drive letter has been newly assigned to MV, re-create the volume list by using the ReplicationControl ismvollist command on the application server. Step 7. Mount restoration volume (MV in ATgroup) (application server) (Windows) At this step, the restoration volume (MV in ATgroup) that will be used after reverting business operations is mounted in the main site s application server, and the stopped service is resumed. (i) Mount the file system. (Command example) ismrc_mount -drv E: ismrc_mount -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (Linux) At the main site s application server, the volume group is set as active or the disk group is imported for the restoration volume (MV in ATgroup) that will be used after reverting business operations, which is then mounted. When using LVM on servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Mount the file system. (Command example when using LVM and mounting VxFS s file system) mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when using VxVM and mounting VxFS s file system) mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log 210

219 Chapter 5 Operation Examples Step 8. Restore and check business operations (application server) At this step, the restoration volume (MV in ATgroup) is used to recover the cluster configuration, check the mounted data, and to recover and check business operations in the main site s application server. When using LVM on Linux servers, after business operations have been recovered and checked, be sure to fetch and store the volume group setting information at the main site s application server. This volume group setting information should be backed up once business operations have been recovered and checked. In some cases, the volume group setting information is used for fault repair. (Command example for fetching volume group setting information file) vgcfgbackup /dev/vg01 vgcfgbackup /dev/vg02 Step 9. Prepare Copy Back Mode of business operations (main site and backup site) Once the recovered business operations have been checked in the main site s application server, the network configuration is changed to enable Copy Back Mode of business operations to the main site. 211

220 Chapter 5 Operation Examples Step 10. Wait for completion of ATgroup restoration (application server) At this step, a wait is set until data copying has been completed as part of ATgroup restoration. (Command example of wait for completion of ATgroup restoration) ismrc_wait -atg DB_Group -cond sep Completion of ATgroup restoration can also be checked by ism. * Restoration is completed when the ATgroup sync state changes from restoring to separated. Figure 5-26 Confirmation of ATgroup Restoration by ism (Example) 212

221 Chapter 5 Operation Examples Step 11. Replicate ATgroup (application server) After data copying has been completed as part of ATgroup restoration, the ATgroup is replicated and then restored to normal state. (Command example) ismrc_replicate -atg DB_Group -cpmode semi -wait This can also be done using ism. Figure 5-27 Execution of ATgroup Replication by ism (Example) Step 12. Resume business operations (application server) Business operations are resumed at the main site s application server. 213

222 Chapter 5 Operation Examples Copy Back Mode by Pair Configuration Substitutions This is a Copy Back Mode method that includes temporarily substituting pair configurations by registering a newly created ATgroup in which a backup site s volume is copied back to the MV while the main site s volume is copied back to an RV, after which this ATgroup is replicated. While business operations are continued at the backup site, this ATgroup created for Copy Back Mode is replicated and the data is copied back. Once the Copy Back Mode is completed, business operations are switched back. Since this method enables data to be copied back while business operations are continued, business operations can be switched with very little downtime. However, when pair configurations are substituted in this way, ATgroup replication requires full-range copying, which means that some time is needed to reach the replicated state (Rpl/sync) after business operations are switched. Until the replicated state (Rpl/sync) is reached, data in the ATgroup s RVs is not consistent. Accordingly, DynamicDataReplication (or other software) is used to back up the copied back data after business operations are switched and before ATgroup replication. Copy Back Mode by pair substitution is therefore a method that is appropriate in cases such as when there is not enough time to switch over business operations for Copy Back Mode by restoration. The following describes Copy Back Mode by pair substitution based on a configuration example. 214

223 Chapter 5 Operation Examples (1) Overview of Copy Back Mode by Pair Configuration Substitutions The example used to describe the steps of Copy Back Mode by pair configuration substitution is shown below. When a disaster occurs, business operations that had been handled by a backup site s standby server are switched to the main site s application server. The backup site s volume is temporarily copied to the MV and the main site s volume is copied to an RV (hence a pair substitution), while a new ATgroup created for Copy Back Mode is registered and replicated, after which the data is copied back. Main site Backup site Application server Standby server Backup server Backup server Tape Tape DB_Group DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E Replicate DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E LOG_BK_ LOCAL Mount /dev/vg02 and dg02 in /log or drive F LOG_MV Mount /dev/vg02 and dg02 in /log or drive F Replicate LOG_RV Mount /dev/vg02 and dg02 in /log or drive F LOG_BK_ REMOTE Mount /dev/vg02 and dg02 in /log or drive F LOCAL_ARRAY Disk array REMOTE_ARRAY Disk array Copy Back Mode by substitution of MV and RV pairs [ATgroup settings after Copy Back Mode] ATgroup s copy mode: Semi synchronous copy mode of order guarantee MV access after atomic-break: MV access continue Figure 5-28 Copy Back Mode by Pair Configuration Substitutions (Example) 215

224 Chapter 5 Operation Examples (2) Flow of steps in Copy Back Mode by pair configuration substitution [Operations at main site] [Operations at backup site] Step 1. Repair fault at main site (main site) Step 2a. Create volume list (application server and backup server) Step 2b. Create volume list (standby server and backup server) Step 3. Create new ATgroup for Copy Back Mode during MV and RV pair configuration substitution (standby server) Step 4. Unmount restoration volume (RV in the ATgroup used for Copy Back Mode) (application server) Step 5. Replicate ATgroup used for Copy Back Mode (standby server) Step 6. Secure pause point (standby server) Step 7. Separate ATgroup used for Copy Back Mode (standby server) Step 8. Resume business operations (application server) Step 9. Mount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) Step 10. Restore and check business operations (application server) Step 11. Unmount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) Step 12. Replicate ATgroup used for Copy Back Mode (standby server) 216

225 Chapter 5 Operation Examples Step 13. Stop business operations (standby server) Step 14. Unmount business volume (MV in the ATgroup used for Copy Back Mode) (standby server) Step 15. Separate ATgroup used for Copy Back Mode (standby server) Step 16. Mount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) Step 17a. Prepare to switch over business operations (main site) Step 17b. Prepare to switch over business operations (backup site) Step 18. Resume business operations (application server) Step 19. Delete ATgroup used for Copy Back Mode where pair configuration of MV and RV has been switched (standby server) Step 20. Unmount DDR pair s RV (backup server) Step 21. Replicate DDR pair (standby server) Step 22. Separate DDR pair (standby server) Step 23. Mount DDR pair s RV (backup server) Step 24. Reset ATgroup (application server) Step 25. Unmount ATgroup s RV (standby server: if a Windows server) Step 26. Replicate ATgroup (application server) 217

226 Chapter 5 Operation Examples * When business operations have been stopped, if it is possible to confirm whether or not business operations can be resumed at the main site s application server, then steps 6 to 12 are not necessary. * If a Linux server is used, step 25 is not necessary. * After restoring the main site fault from steps 13 through 26, you can also copy back data using the swap function as described later. * The flow of these steps and the following operation steps are provided as an example. Be sure to adjust the specific operations for the particular environment. Figure 5-29 Flow of Steps in Copy Back Mode by Pair Configuration Substitution (Example) For steps for Copy Back Mode of RemoteCluster, refer to the RemoteCluster with MC/ServiceGuard User s Manual. (3) Operation Steps Step 1. Repair fault at main site (main site) At this step, the main site s fault is repaired. Software installation and environment settings are also required. Step 2. Create volume list (application server, standby server, and backup server) If the main site s disk array is connected to a backup site s disk array, a volume list is created to include all servers connected to these disk arrays. At this step and afterward, main site s disk array must be connected to a backup site s disk array. (Command example for Windows) ismvollist -cr (Command example for Linux) ismvollist -r * This step is not required on the application server side for the operation using a GPT disk. Make the application server recognize MV rebuilt in Step 8, and re-create the volume list (on the application server) after checking the drive letter. 218

227 Chapter 5 Operation Examples Step 3. Create new ATgroup for Copy Back Mode during MV and RV pair configuration substitution (standby server) When ATgroup replication is used to perform Copy Back Mode, a new ATgroup in which of the backup site s volume is the MV and of the main site s volume is the RV. If any previously set ATgroup remains, it is deleted beforehand. (Command example for pair settings) ismrc_pair -pair -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld ismrc_pair -pair -mv LOG_RV -mvflg ld -rv LOG_MV -rvflg ld or ismrc_pair -pair -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv ld:log_rv ld:log_mv (Command example for creating ATgroup) ismrc_atg -create -atg DB_Group_swap -arrayname REMOTE_ARRAY -linkarrayname REMOTE_ARRAY (Command example for registering pair to ATgroup) ismrc_atg -add -atg DB_Group_swap -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld ismrc_atg -add -atg DB_Group_swap -mv LOG_RV -mvflg ld -rv LOG_MV -rvflg ld or ismrc_atg -add -atg DB_Group_swap -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv ld:log_rv ld:log_mv * Make sure that the backup site s volume is the MV and the main site s volume is the RV. * Before creating the ATgroup, make sure that the backup site s disk array has been specified. * The new ATgroup used for Copy Back Mode is temporarily created as an ATgroup named DB_Group_swap. These settings can also be made via ism. 219

228 Chapter 5 Operation Examples Figure 5-30 Settings in ism for Substitution of MV and RV Pair Configuration (Example) Figure 5-31 Settings in ism for Creation of New ATgroup Used for Substitution of MV and RV Pair Configuration (Example) 220

229 Chapter 5 Operation Examples Figure 5-32 Registration of Pair to ATgroup by ism (Example) 221

230 Chapter 5 Operation Examples Step 4. Unmount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) (Windows) The file system buffers are cleared in the main site s Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) in order to copy back data to the main site. Specifically, the current service that accesses the target volume is stopped, the file system is flushed and unmounted, and then the file system buffers are cleared. (i) Stop the service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system, then clear the file system s buffers. Since the volume to be unmounted is an RV, the drive path (mounted to drive letter or NTFS folder) is automatically deleted to disable access to this RV. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline * This step is not required on the application server side for the operation using a GPT disk. (Linux) To prepare for Copy Back Mode of data to the main site, the main site s restoration volume (RV in the ATgroup used for Copy Back Mode) is unmount. The information in the restoration volume (RV in the ATgroup used for Copy Back Mode) overwrites the information in the business volume (MV in the ATgroup used for Copy Back Mode). (i) Unmount the file system. (Command example) umount /db umount /log (ii) Set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 vxdisk offline sde vxdisk offline sdf * If the volume has already been unmounted, such as when repairing a server fault or logical disk fault, this step can be omitted. 222

231 Chapter 5 Operation Examples Step 5. Replicate ATgroup used for Copy Back Mode (standby server) At this step, the ATgroup used for Copy Back Mode is replicated and data is copied to the copy back volume (RV in ATgroup used for Copy Back Mode). (Command example) ismrc_replicate -atg DB_Group_swap -cpmode semi -cprange full -wait * Since this is done immediately after pair reconfiguration, full range copying is performed explicitly at this step. This can also be done using ism. Figure 5-33 Replication by ism of ATgroup Used for Copy Back Mode (Example) 223

232 Chapter 5 Operation Examples Step 6. Secure pause point (standby server) (Windows) A pause point is reserved in the business operations at the backup site in order to retrieve data used to confirm the switchover of business operations to the Copy Back Mode volume (the RV in the ATgroup used for Copy Back Mode). Specifically, business operations and other aspects of the current service that access the target volume are stopped, the business volume s file system is flushed and unmounted, and a pause point is secured. (i) Stop business operations. (ii) Stop service. (Command example) NET STOP ESMCommonService (iii) Flush and unmount the file system. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: ismrc_umount -drv F: (Linux) A pause point is reserved in the business operations at the backup site in order to retrieve data used to confirm the switchover of business operations to the Copy Back Mode volume (the RV in the ATgroup used for Copy Back Mode). Specifically, business operations are stopped, the business volume is unmounted, either the volume group is set as inactive or the disk group is deported, and a pause point is secured. (i) Stop business operations at backup site. (ii) Unmount backup site s business volume. (Command example) umount /db umount /log (iii) Set the volume group as inactive or deport the disk group. If LVM is used, set the volume group as inactive for the backup site s business volume. If VxVM is used, deport the business volume s disk group at the backup site. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 224

233 Chapter 5 Operation Examples Step 7. Separate ATgroup used for Copy Back Mode (standby server) Separate the ATgroup used for Copy Back Mode and save to the Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) the data that will be checked switching over business operations. (Command example) ismrc_separate -atg DB_Group_swap -wait This can also be done using ism. Figure 5-34 Separation by ism of ATgroup Used for Copy Back (Example) For the operation using a GPT disk, the following steps must be executed on the application server before executing Step 8: Making the application server recognize the rebuilt MV Make the application server recognize the rebuilt MV by setting the AccessControl. Recognizing the MV-side drive letter Start the [Disk Management] of Windows, and assign a drive letter to the disk. Re-creating the volume list Since the drive letter has been newly assigned to MV, re-create the volume list by using the ReplicationControl ismvollist command on the application server. 225

234 Chapter 5 Operation Examples Step 8. Resume business operations (standby server) (Windows) Business operations are resumed in order to secure a pause point. At this point, the business volume is mounted, then the stopped service is started and business operations are resumed. (i) Mount the file system. (Command example) ismrc_mount -drv E: ismrc_mount -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (iii) Resume business operations. (Linux) Business operations are resumed in order to secure a pause point. At this point, the volume group is set as active or the disk group is imported, and then the business volume is mounted and business operations are resumed. (i) Activate the volume group or import the disk group for the backup site s business volume. If LVM is used, set the backup site s business volume group as active. If VxVM is used, import the backup site s business volume s disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdg import dg01 vxdg import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Mount the business volume. (Command example of mounting VxFS s file system when using LVM) mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example of mounting VxFS s file system when using VxVM) mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log (iii) Resume business operations. 226

235 Chapter 5 Operation Examples Step 9. Mount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) (Windows) The Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) is mounted to enable switchover of business operations to the main site s application server. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RV. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (Linux) The Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) is set as in an active volume group or the disk group is imported and mounted to enable switchover of business operations to the main site s application server. When using LVM on servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on servers, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online sde vxdisk online sdf vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Mount the file system. (Command example when using LVM and mounting VxFS s file system) mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when using VxVM and mounting VxFS s file system) 227

236 Chapter 5 Operation Examples mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log Step 10. Restore and check business operations (application server) The Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) is used to confirm recovery of the cluster configuration, and mounted data, as well as to confirm recovery of business operations at the main site s application server. Step 11. Unmount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) (Windows) The file system buffers are cleared in the main site s Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) in order to copy differential data to the Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) during confirmation of business operations recovery. Specifically, the current service that accesses the target volume is stopped, the file system is flushed and unmounted, and then the file system buffers are cleared. (i) Stop service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system and clear the file system buffers. Since the volume to be unmounted is an RV, its drive path (mounted to drive letter or NTFS folder) is automatically deleted so that these it cannot be accessed. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (Linux) The main site s Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) is unmounted in order to copy differential data to the Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) during confirmation of business operations recovery. Since information in the restoration volume (RV in the ATgroup used for Copy Back Mode) is overwritten by information from the business volume (MV in ATgroup used for Copy Back Mode). (i) Unmount the file system. (Command example) umount /db umount /log (ii) Set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) vgchange -a n /dev/vg01 228

237 Chapter 5 Operation Examples vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 vxdisk offline sde vxdisk offline sdf Step 12. Replicate ATgroup used for Copy Back Mode (standby server) At this step, the ATgroup used for Copy Back Mode is replicated and differential data during confirmation of business operations recovery is copied to the restoration volume (RV in the ATgroup used for Copy Back Mode). (Command example) ismrc_replicate -atg DB_Group_swap -cpmode semi -wait This can also be done using ism. Figure 5-35 Replication of ATgroup Used for Copy Back ism (Example) 229

238 Chapter 5 Operation Examples Step 13. Stop business operations (standby server) Business operations are stopped at the backup site s standby server to enable their switchover. Step 14. Unmount business volume (MV in the ATgroup used for Copy Back Mode) (standby server) (Windows) At this step, the file system in the business volume (MV in ATgroup used for Copy Back Mode) is unmounted. Specifically, the current service that accesses the target volume is stopped and the business volume s file system is flushed and unmounted. (i) Stop the service. (Command example) NET STOP ESMCommonService (ii) Flush and unmount the file system. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: ismrc_umount -drv F: (Linux) At this step, the business volume (MV in ATgroup used for Copy Back Mode) is unmounted. Since the information in the business volume (MV in ATgroup used for Copy Back Mode) is overwritten during replication that follows switchover of business operations. (i) Unmount the file system. (Command example) umount /db umount /log (ii) Set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. If VxVM is used, deport the disk group. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 vxdg deport dg02 230

239 Chapter 5 Operation Examples Step 15. Separate ATgroup used for Copy Back Mode (standby server) At this step, the ATgroup is separated in order to preserve the differential data to the volume used for Copy Back Mode (RV in the ATgroup used for Copy Back Mode). (Command example) ismrc_separate -atg DB_Group_swap -wait This can also be done using ism. Figure 5-36 Separation of ATgroup Used for Copy Back Mode by ism (Example) 231

240 Chapter 5 Operation Examples Step 16. Mount Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) (application server) (Windows) The Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) that is used after switching over business operations in the main site s application server is mounted. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RV. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (Linux) The Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) that is used after switching over business operations at the main site s application server is set as an active volume group or the disk group is imported and mounted. When using LVM on servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on servers, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online sde vxdisk online sdf vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Mount the file system. (Command example when using LVM and mounting VxFS s file system) mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log 232

241 Chapter 5 Operation Examples (Command example when using VxVM and mounting VxFS s file system) mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log When using LVM on Linux servers, after business operations have been recovered and checked, be sure to fetch and store the volume group setting information at the main site s application server. This volume group setting information should be backed up once business operations have been recovered and checked. In some cases, the volume group setting information is used during switchover of business operations or during fault repair. (Example of commands to fetch volume group setting information file) vgcfgbackup /dev/vg01 vgcfgbackup /dev/vg02 Step 17. Prepare to switch over business operations (main site and backup site) At this step, the Copy Back Mode volume (RV in the ATgroup used for Copy Back Mode) is used to recover the cluster configuration and change the network configuration in order to enable the resumption of business operations at the main site. Step 18. Resume business operations (application server) Business operations are resumed in the main site s application server. Step 19. Delete ATgroup used for Copy Back Mode where pair configuration of MV and RV has been switched (standby server) At this step, the ATgroup that was created for Copy Back Mode is deleted. (Command example of unpairing of pair registered to ATgroup) ismrc_atg -remove -atg DB_Group_swap -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld ismrc_atg -remove -atg DB_Group_swap -mv LOG_RV -mvflg ld -rv LOG_MV -rvflg ld or ismrc_atg -remove -atg DB_Group_swap -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv ld:log_rv ld:log_mv (Command example of ATgroup deletion) ismrc_atg -delete -atg DB_Group_swap -arrayname REMOTE_ARRAY 233

242 Chapter 5 Operation Examples (Command example of unpairing) ismrc_pair -unpair -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld ismrc_pair -unpair -mv LOG_RV -mvflg ld -rv LOG_MV -rvflg ld or ismrc_pair -unpair -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv ld:log_rv ld:log_mv These deletions can also be performed via ism. Figure 5-37 Unpairing by ism of Pairs Registered to ATgroup Used for Copy Back Mode (Example) 234

243 Chapter 5 Operation Examples Figure 5-38 Deletion by ism of ATgroup Used for Copy Back Mode (Example) Figure 5-39 Unpairing by ism of Pairs Used for Copy Back Mode (Example) 235

244 Chapter 5 Operation Examples Step 20. Unmount DDR pair s RV (backup site s backup server) (Windows) Replication of an ATgroup after recovery of business operations requires full-range copying, which means that some time is needed to reach the replicated state (Rpl/sync) after business operations are switched. Until the replicated state (Rpl/sync) is reached, data in the ATgroup s RVs is not consistent, so the data to be used for Copy Back Mode is backed up to a DDR pair s RV at a backup site. In preparation for backup, buffers are cleared in the respective DDR pair s RV which is recognized by the backup site s backup server. The server that is already accessing the target volume is stopped, the file system is flushed and unmounted, and then the file system s buffers are cleared. (i) (ii) Stop the service. (Command example) NET STOP ESMCommonService Flush and unmount the file system and clear the file system buffers. Since the volume to be unmounted is an RV, its drive path (mounted to drive letter or NTFS folder) is automatically deleted so that the RV cannot be accessed. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (Linux) Replication of an ATgroup after recovery of business operations requires full-range copying, which means that some time is needed to reach the replicated state (Rpl/sync) after business operations are switched. Until the replicated state (Rpl/sync) is reached, data in the ATgroup s RVs is not consistent, so the data to be used for Copy Back Mode is backed up to a DDR pair s RV at a backup site. In preparation for backup, the DDR pair s RV that is recognized by the backup site s backup server is unmounted. Since the RV information is overwritten over the MV information. (i) Unmount the file system. (Command example) umount /db umount /log (ii) Either set the volume group as inactive or deport the disk group. If LVM is used, the volume group is set as inactive. If VxVM is used, deport the disk group, and make the RV disks offline. (Example of command when using LVM) vgchange -a n /dev/vg01 vgchange -a n /dev/vg02 (Example of command when using VxVM) vxdg deport dg01 236

245 Chapter 5 Operation Examples vxdg deport dg02 vxdisk offline sde vxdisk offline sdf Step 21. Replicate DDR pair (standby server) At this step, the backup site s DDR pair is replicated so that the data to be used for Copy Back Mode can be copied to the DDR pair s RV at the backup site. (Command example) ismrc_replicate -file pair_ddr_r.txt -wait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote This can also be done using ism. Figure 5-40 Replication by ism (Example) 237

246 Chapter 5 Operation Examples Step 22. Separate DDR pair (standby server) At this step, a DDR pair is separated so that the data to be used for Copy Back Mode can be preserved in the DDR s RV at the backup site. (Command example) ismrc_separate -file pair_ddr_r.txt -wait * pair_ddr_r.txt setting example #Type:MV Type:RV ld:db_rv ld:db_bk_remote ld:log_rv ld:log_bk_remote This can also be done using ism. Figure 5-41 Separation by ism (Example) 238

247 Chapter 5 Operation Examples Step 23. Mount DDR pair s RV (backup site s backup server) (Windows) At this step, the DDR pair s RV that is recognized by the backup site s backup server is mounted. Since the drive path (mounted to drive letter or NTFS folder) has been deleted, the drive path (mounted to drive letter or NTFS folder) option is used to specify a drive path before mounting the RVs. Also, the current service that was stopped for unmounting is restarted. (i) Specify a value in the drive path (mounted to drive letter or NTFS folder) option and mount the file system. (Command example) ismrc_mount -mvol \\?\Volume{023b5ee0-bf03-11d6-ad1b-806d f} -drv E: ismrc_mount -mvol \\?\Volume{023b5ee1-bf03-11d6-ad1b-806d f} -drv F: (ii) Restart the service. (Command example) NET START ESMCommonService (Linux) The DDR pair s RV that is recognized by the backup site s backup server is mounted in the volume group is set as active or the disk group is imported and the DDR pair normal consistency state is restored. When using LVM on servers, only activation of the volume group is performed. If VxVM is used, the -C option is used when importing. (i) Activate the volume group or import the disk group. When using LVM on servers, only activation of the volume group is performed. If VxVM is used, after making the RV disks online, the -C option is used when clearing and importing the disk group. (Example of command when using LVM) vgchange -a y /dev/vg01 vgchange -a y /dev/vg02 (Example of command when using VxVM) vxdisk online sde vxdisk online sdf vxdg -C import dg01 vxdg -C import dg02 vxvol -g dg01 startall vxvol -g dg02 startall (ii) Mount the file system. (Command example when using LVM and mounting VxFS s file system) mount -t vxfs /dev/vg01/lvol1 /db mount -t vxfs /dev/vg02/lvol1 /log (Command example when using VxVM and mounting VxFS s file system) mount -t vxfs /dev/vx/dsk/dg01/vol1 /db mount -t vxfs /dev/vx/dsk/dg02/vol1 /log 239

248 Chapter 5 Operation Examples Step 24. Reset ATgroup (application server) Reset the ATgroup so that the main site s volume is the MV and the backup site s volume is the RV, which is the configuration for normal operation. (Command example of pair setting) ismrc_pair -pair -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_pair -pair -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld or ismrc_pair -pair -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv (Command example of ATgroup creation) ismrc_atg -create -atg DB_Group -arrayname LOCAL_ARRAY linkarrayname REMOTE_ARRAY (Command example of registration of pair to ATgroup) ismrc_atg -add -atg DB_Group -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld ismrc_atg -add -atg DB_Group -mv LOG_MV -mvflg ld -rv LOG_RV -rvflg ld or ismrc_atg -add -atg DB_Group -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv ld:log_mv ld:log_rv These settings can also be entered via ism. 240

249 Chapter 5 Operation Examples Figure 5-42 Pair Resetting by ism (Example) Figure 5-43 Re-creation of New ATgroup by ism (Example) 241

250 Chapter 5 Operation Examples Figure 5-44 Re-registration of Pair to ATgroup by ism (Example) Step 25. Unmount ATgroup s RV (when standby server is a Windows server) (Windows) To prepare for replication of the ATgroup after business operations have been restored, the file system buffers are cleared once again in the ATgroup s RV, and the RV cannot be accessed. Once the buffers have been cleared, the file system remains unmounted, so the stopped service is restarted. (i) Flush and unmount the file system and clear the file system s buffers once again. Since the volume to be unmounted is an RV, the drive path (mounted to drive letter or NTFS folder) is automatically deleted and the RV cannot be accessed. (Command example) ismrc_flush -drv E: ismrc_flush -drv F: ismrc_umount -drv E: -offline ismrc_umount -drv F: -offline (ii) Restart the service. (Command example) NET START ESMCommonService 242

251 Chapter 5 Operation Examples Step 26. Replicate ATgroup (application server) At this step, the ATgroup is replicated and the normal configuration for operations is restored. (Command example) ismrc_replicate -atg DB_Group -cpmode semi -cprange full -wait * Since this is done immediately after pair reconfiguration, full range copying is performed explicitly at this step. Figure 5-45 Replication by ism of ATgroup (Example) 243

252 Chapter 5 Operation Examples 5.5 Copy Back Mode through Swap for Single RDR Pair How to copy back data from RV to MV by pair configuration substitutions is described in Copy Back Mode by Pair Configuration Substitutions. In this RDR pair substitution, you perform unpairing in Step 19. Delete ATgroup used for Copy Back Mode where pair configuration of MV and RV has been switched (standby server) and reset the pair in Step 24. Reset ATgroup (application server). Although the supporting disk arrays are limited, the swap function for RDR pair enables you to immediately switch the roles of MV and RV, avoid full copy during the replication operation after the processing, and maintain the MV data redundancy by RV. However, when you use the swap function, the operation restarting time and the failure recovery procedures are different from the conventional operation. This section describes the points to be noted using a single RDR pair. Then, application to the ATgroup is described in the next section. 244

253 Chapter 5 Operation Examples Notes on Copy Back Mode Procedure through Swap for Single RDR Pair Notes on Copy Back Mode procedure using the swap for a single RDR pair is described based on the configuration diagram below. Main site Backup site Backup server Application server Standby server Backup server Tape Tape DB_BK_ LOCAL Mount /dev/vg01 and dg01 in /db or drive E DB_MV Mount /dev/vg01 and dg01 in /db or drive E Replicate DB_RV Mount /dev/vg01 and dg01 in /db or drive E DB_BK_ REMOTE Mount /dev/vg01 and dg01 in /db or drive E dg02 LOCAL_ARRAY REMOTE_ARRAY Disk array Copy Back Mode through swap for RDR pair Figure 5-46 Configuration Example of Data Switching Procedure through Swap for RDR Pair 245

254 Chapter 5 Operation Examples (1) Conventional Pair Configuration Substitutions (Unpairing and Resetting the Pair) (i) Conventional Procedure: Unpair the pair for Copy Back Mode where pair configuration of MV and RV has been switched (standby server) Unpair the RDR pair created for Copy Back Mode. (Unpairing example from the command line) ismrc_pair -unpair -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld or ismrc_pair -unpair -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv (ii) Conventional Procedure: Reset the pair (application server) Reset the pair using MV as the main site volume and RV as the backup site volume and configure normal operation configuration. (Pair setting example from the command line) ismrc_pair -pair -mv DB_MV -mvflg ld -rv DB_RV -rvflg ld or ismrc_pair -pair -file pair.txt * pair.txt setting example #Type:MV Type:RV ld:db_mv ld:db_rv (2) Pair Configuration Substitutions by Swap Procedure by swap: Swap the pair for Copy Back Mode (standby server) Swap the pair for Copy Back Mode and make normal operation configuration. (Command example) ismrc_swap -mv DB_RV -mvflg ld -rv DB_MV -rvflg ld or ismrc_swap -file pair_swap.txt * pair_swap.txt setting example #Type:MV Type:RV ld:db_rv ld:db_mv 246

255 Chapter 5 Operation Examples * Note that the main site volume returns to MV and the backup site volume returns to RV after performing the swap. However, in pair swap operation, the time when the operation is restarted at the main site application server and the time when the parallel and serial DDR pairs are backed up are different as shown in the table below compared with when you unpair and reset the pair. Table 5-2 Differences between Unpairing/Resetting Operation and Pair Swap Operation Differences Unpairing/resetting operation Pair swap operation MV and RV mounting states Target pair s replication state Parallel/serial pair s replication state It is not required to unmount MV and RV when unpairing or resetting the pair. You can unpair and reset the pair at the main site application server after the operation is restarted. The pair must be in the separated state to perform unpairing. However, if data is different between MV and RV when the pair is unpaired or set, it is not a problem. Because the RV at the main site is separated, the operation can be restarted and data can be different between MV and RV. The parallel/serial pair s replication state is not specified. You can unpair and reset the pair at the main site application server by copying data to DDR after the operation is restarted. You need to unmount both MV and RV in advance because their update is prohibited. It is not possible to restart the operation before pair swapping because the RV at the main site cannot be updated when swapping the pair. MV and RV must be in the replicated state (Rpl/sync) where the data in both volumes is the same when performing the swap. It is not possible to restart the operation before swapping the pair because the RV at the main site is being replicated. Because MV and RV cannot be updated, the parallel/serial pair must be in the replicated or separated state where copy operations to MV or RV are not executed (The restored or separating state is not usable). To ensure the MV redundancy at the backup site, you need to complete copying and separation of the backup to the parallel DDR pair before performing the swap. 247

256 Chapter 5 Operation Examples Recovery Procedure at Pair Swap Failure This section describes the recovery procedure when pair swap is abnormally terminated during Copy Back Mode through swap for RDR pair. (1) Determination of Pair State and Recovery Method When Pair Swap Is Abnormally Terminated Determine the recovery method based on the pair state in the table below if the swap is abnormally terminated. Table 5-3 Pair State and Recovery Method When Pair Swap is Abnormally Terminated Pair state State Recovery method Switching Sync state MV/RV update determination state prevention state result MV and RV roles not swapped Replicate synchronous Replicate synchronous Separate Non-update prevention state MV and/or RV is in the update prevention state. MV and RV are in the update prevention state. Swap not yet performed Failure during preprocessing before swap During swap Remove the cause of failure and then proceed to the recovery procedure. (1) Perform the swap operation again. Remove the cause of failure and then proceed to the recovery procedure. (1) Perform forced cancellation of update prevention to MV or RV that is in the update prevention state. (2) Perform the swap operation again. Remove the cause of failure and then proceed to the recovery procedure. (1) Perform forced cancellation of update prevention to MV and RV. (If the message ID ism13994 * error occurs, use the pair switching by unpairing and resetting. Perform the steps (4) to (6) below.) (2) Replicate the pair. (3) Execute the swap command again. MV and RV roles swapped Separate Separate MV and/or RV is in the update prevention state. Update non-prevention state Failure during post processing after swap Swap completed (Execute the following steps if he message ID ism13994 * error occurs.) (4) Perform forced unpairing of MV and RV. (5) Perform forced cancellation of update prevention to the Copy Back Mode and business volumes. (6) Set the pair in the configuration (normal operation configuration) to be switched. Remove the cause of failure and then proceed to the recovery procedure. (1) Perform forced cancellation of update prevention to MV or RV that is in the update prevention state. This pair has been swapped and the recovery process is not needed. * For the error message with message ID ism13994, refer to the Messages Handbook. 248

257 Chapter 5 Operation Examples (2) Checking the Pair State Using the Command (i) Link status check Check that the link state is normal. ismrc_arayinfo -arrayname REMOTE_ARRAY -linfo * Check the link state by referring to LOCAL_ARRAY and Path State in the command execution result. (ii) Pair switching state check Check that the roles of MV and RV in the pair have been switched by the swap operation. You can determine the recovery procedure based on this state. (Example using the command for confirmation) Execute this command for the volume that was MV before the swap. ismrc_sense -vol DB_RV -volflg ld -attr * If Attribute in the command execution result shows MV, MV and RV roles have not yet been switched. When this value is RV or RV/MV, MV and RV roles have been switched. (iii) Pair sync state check Check the pair sync state. You can determine the recovery procedure based on this state. (Example using the command for confirmation) ismrc_query -mv BD_RV -mvflg ld * Check the pair sync state by referring to Sync State in the command execution result. (iv) MV and RV update prevention state check Check the MV and RV update prevention state. You can determine the recovery procedure based on this state. (Command example) or ismrc_sense-vol DB_RV -volflg ld -updprevent ismrc_sense -vol DB_MV -volflg ld -updprevent ismrc_sense -file volume.txt -updprevent * volume.txt setting example #Type:LD ld:db_rv ld:db_mv * Check the MV and RV update prevention state by referring to Update Prevention State in the command execution result. 249

258 Chapter 5 Operation Examples (3) Pair State Check from the ism Screen You can check the pair state and determine the recovery procedure from the ism screen. (i) Link state check You can check this state in the [Path State] field on the Disk Array Link Information tab in the replication management screen. Figure 5-47 Link State Check Example with ism (ii) Pair switching state, pair sync state and MV/RV update prevention state check You can check these states in the [Type], [Sync State], and [Update Prevention State] fields on the Replication Information tab in the replication management screen. 250

259 Chapter 5 Operation Examples Figure 5-48 Example of Checking Pair and Other States with ism (4) Removing Cause of Failure If the pair swap operation is abnormally terminated, refer to the output error message in Messages Handbook and take appropriate action against the error. 251

260 Chapter 5 Operation Examples (5) Example of Recovery Procedure at Pair Swap Failure In this section, a failure that occurred during the swap is used as example (if other pair states are caused by failure, perform one of the steps (1) to (3) to complete the swap). [Operations at main site] [Operations at backup site] Step 1. Perform forced cancellation of update prevention to MV and RV in the pair used for Copy Back Mode(standby server) Step 2. Replicate the pair used for Copy Back Mode (standby server) Step 3. Swap the pair used for Copy Back Mode (standby server) Step 4a. Perform forced unpair of RV in the pair used for Copy Back Mode (application server) Step 4b. Perform forced unpair of MV in the pair used for Copy Back Mode (standby server) Step 5a. Perform forced cancellation of update prevention to the Copy Back Mode volume (application server) Step 5b. Perform forced cancellation of update prevention to the business volume (standby server) Step 6. Set the pair for the configuration to be switched (configuration for normal operation) (application server) * If the error with the message ID ism13994 occurs at step 1, perform steps 4 to 6. * If recovery from swap error is completed at step 3, steps 4 to 6 are unnecessary. Figure 5-49 Flow Example of Recovery Procedure at Pair Swap Failure 252

261 Chapter 5 Operation Examples Step 1. Perform forced cancellation of update prevention to MV and RV in the pair used for Copy Back Mode (standby server) Perform forced cancellation of update prevention for MV and RV in the pair used for Copy Back Mode. (Command example) or ismrc_updprevent -vol DB_RV -volflg ld -mode cancel -force ismrc_updprevent -vol DB_MV -volflg ld -mode cancel -force ismrc_updprevent -file volume.txt -mode cancel -force * volume.txt setting example #Type:LD ld:db_rv ld:db_mv * If the error with the message ID ism13994 occurs at step 1, perform steps 4 to 6. For the error message with message ID ism13994, refer to the Messages Handbook. You can perform this operation from ism by selecting the pair on the replication management screen and performing the forced cancellation of update prevention operation. Figure 5-50 Example of Forced Cancellation of Update Prevention by ism 253