10 The next chapter of this Web Based Training module describe the two different Remote Equivalent Transfer Modes; synchronous and asynchronous.

Transcription

1 Slide 0 Welcome to this Web Based Training session providing an introduction to the fundamentals of Remote Equivalent Copy. This is one of the ETERNUS DX Advanced Copy Functions available in Fujitsu ETERNUS DX systems from DX90 upwards. Advanced Copy functionality is embedded in the firmware of the DX system and is managed with the ETERNUS SF management applications: either with AdvancedCopy Manager or SF Express. For the use of ACM there are different purchasable licenses for the different Advanced Copy Functions. SF Express with basic functionality is available free of charge for the DX Entry models, but a Remote Copy license is required for the use of Remote Equivalent Copy. 1 After this Web Based Training module you will be familiar with the Advanced Copy Feature Remote Equivalent Copy, often abbreviated as REC. In some contexts during this training module it is also pronounced as REC. We will start with an overall description of some general terms associated with REC. The chapter REC Modes explains in brief the two different interconnect types. The chapter Synchronous Split and Recovery Sub-modes explains the configuration parameters and their use in case of a connection failure. Next chapter explains the different REC transmission modes and their configuration parameters for the asynchronous mode. The next section explains in more detail the functionality of a REC session in asynchronous mode. A comparison and a summary of the previously learned REC topics follow in the chapter Differences Between REC Modes. The last two chapters of this Web Based Training show how to shorten the initial synchronization time in an ETERNUS DX Disaster Recovery configuration and how to combine the different local and remote ETERNUS DX Advanced Copy features. 2 Let's start with the REC overview. First we will have a look at the two different main usages of REC. Next we will see what is important in a Disaster Configuration from the business continuity point of view. In chapters Concept Diagram and Suspend and Resume we will look in detail at the different REC administration steps. Chapter REC Process summarizes the different phases of REC replication, after which a brief overview of the REC Concurrent Suspend mode rounds out this first chapter. 3 Remote Equivalent Copy is always about having two ETERNUS DX systems interconnected for data mirroring purposes. Single DX system can be connected with more than one other system but each REC connection is always a pair. The distance between the two systems can be anything between meters and thousands of kilometers - physically close systems provide typically High Availability whereas interconnected systems in two continents typically speak for Disaster Recovery functionality.

2 Typically for data High Availability REC is used within the same data center or between two data centers that are located in nearby buildings. One REC session connects two ETERNUS DX systems and with short distances the interconnection lines can be set up using Fiber Channel or iscsi protocol. With short distances data High Availability is rather easy to achieve by simply mirroring the data between two ETERNUS DX systems. Consequently the server or servers have to have access to both ETERNUS systems. Should the other system fail the servers continue to have access to the data available either at site A or at site B. The other option is to have a configuration with active business related applications at both locations. In this case the production and backup data are split between the two locations. For example applications running on a server located at site B have its backup data in site A - and vice versa. The interconnection lines need to be dedicated for REC and to be independent from the host data lines. This means that a minimum of one - for redundancy two - host ports of a Controller Module are used only for REC. 4 The term Remote Equivalent Copy implies data mirroring between two ETERNUS DX systems. REC is often used for Disaster Recovery functionality. As data availability has become increasingly critical for the business continuity of a company, the importance of ensuring - or quickly recovering - access to data has increased. The physical distance between the two ETERNUS systems can be anything from next room or building to different continent - the configuration is defined by the security concept of the customer. However, with distances exceeding 10 kilometers special converter hardware needs to be used to convert from Fiber Channel to Wide Area Network or iscsi protocol. Like on the previous slide, also over greater distances it is still possible to use the backup site as a second productive environment with full access to the replicated data. In such a constellation the main site holds its backup data at the backup site and the backup site respectively has its backup data at the main site. A more common configuration is to use the backup site only for the replicated backup. If the main site goes down the data can be accessed in the backup site. Remote Equivalent Copy uses ETERNUS DX built-in copy functionality and therefore functions completely serverless. For data replication the two systems must be connected with dedicated interconnection lines that are separate from normal host data interfaces and thus used for REC only. 5 With this slide we will have a look at some parameters associated with data recovery. Two parameters are commonly used as planning basis when a company is considering implementing a Disaster Recovery scheme; RPO and RTO. These are key parameters when defining company's business continuity processes. The abbreviation RPO is short for Recovery Point Objective and RTO for Recovery Time Objective. This slide explains in brief the meanings of these two terms. Let's use the blue time line as a basis. At one point in time a disaster - anything from a data base corruption, hardware failure or a data center fire - occurs and impacts the company's business. The result is an unexpected application downtime.

3 The first question is how much data has been lost? The answer associates with the first key parameter - RPO - in a Disaster Recovery constellation. In our time line that means: was the last known good data backed up 24 hours ago or was it backed up - let's say - only a few hours ago? During this time window of RPO all modifications are lost. Now the other aspect of Disaster Recovery comes into play, RTO that answers the question: How soon after the crash is the application available again? During this time window the following three objectives have to be achieved: The Analyze phase determines how much data is lost? When did the last data backup take place? What are the next steps to restore the data? Next, in the Restore phase, all the necessary tasks have to be executed to bring back the data online. These tasks were specified in the previous "Analyze" phase. After the data restore phase the Recovery phase follows. That brings back the data to the state prior to the crash. These tasks can include activities like computing and executing different type of log files; intent logs, redo logs and the like. Obviously, the shorter both parameters are, the better the company's business is protected against disasters; REC can help bring these values down to minutes and even seconds. 6 This diagram demonstrates a generic REC concept overview. The upper line shows the Source Volume in site A and the lower line in site B the Destination - or Target - Volume. Both reside in different ETERNUS DX systems. Using ACM or SF Express the Remote Equivalent Copy session is started by a GUI or CLI. The whole copy process runs serverless in the background and sequentially transfers the data from the Source to the Target Volume. What happens now with a write request to the source LUN? As REC is a kind of a mirror process, the writing of the data block is executed on both Volumes - Source and Target. After the complete Source Volume data is copied over, the initial copy phase is finished and the REC status is Paired. In Paired status the data is mirrored meaning all new data is always written to both Volumes which guarantees the data equivalence. During this time the Destination Volume is visible but not accessible to the server Operating System or its applications even it is mapped; it can be read but a write attempt results in an error. The system administrator can stop the REC session using ACM or SF Express. In an emergency situation this can also be done with the ETERNUS Web GUI. After the split of the remote mirror, the two Volumes are autonomous and can be accessed independently. 7 This next slide shows the difference and the pros and cons of tasks "Cancel" and "Suspend and Resume" of an REC session. In both cases a replication pair - comprising a Source and Destination Volume - must have reached the Equivalent state before the mirroring can be interrupted. Let's have first look at the "Cancel" interruption. Consequently the two Volumes are no longer linked together. Next we look what happens when we write a black and green block to the Source Volume and a red data block to the Destination Volume.

4 After cancelation, a REC session has to be started again to bring it back into Equivalent state. The initial synchronization copy process is started. All data of the Source Volume will be copied to the Destination Volume so that the former "red" data block at the Destination Volume is now overwritten by the original, unmodified Source data block. The REC session finishes the initial synchronization after all Source Volume data blocks are copied over to the Destination Volume and the state is Equivalent. Next we will have a look at the Suspend and Resume interruption and start with the same initial point: a replication pair - consisting a Source and Destination Volume - has reached state Equivalent. After the Suspend command, the two replication Volumes are no longer actively mirrored. Again we change the Source Volume with a black and green data block. But now the modification at the Source is tracked by a bitmap and can be stored in a dedicated REC buffer cache, depending on the REC usage mode. This is explained in detail later in this Web Based Training module. And also the modifications of the Destination Volume are tracked by a bitmap and can be stored in a REC Buffer Cache that resides in the Controller Module of the ETERNUS system. A REC session that is in Suspend state can be re-started with a Resume command only. In our example the bitmap shows that we have to copy only three data blocks from the Source to the Destination Volume to reach equivalent state with the remote replication. According to the bitmap, first the black and green data blocks are copied over to the Destination Volume and then the red block of the Destination Volume is overwritten by the corresponding original data block from the Source. The copy process of these three data blocks brings the replication pair to Equivalent state. 8 Let's summarize now the steps of a Remote Equivalent Copy session. When a REC session starts, a block level copy from the local copy Source to the remote Destination is executed as a background task by the ETERNUS DX. The ETERNUS internal copy priority can be set up with the parameter settings with the ETERNUS Web GUI or via CLI. For a detailed explanation of the use of these parameters please join an ACM or SF Express classroom training. After the REC replication is started with ACM or SF Express or CLI the Source Volume is normally accessible by the server operating system and application. Note, when REC is started the Destination Volume is locked by the ETERNUS DX and no longer write accessible by the host server. The initial copy process has an impact on the overall I/O performance of the storage system. After the Equivalent state is achieved the performance impact is no longer recognizable by the server or its applications. The next process step begins after 100% synchronization is achieved. After that all write tasks are executed in both volumes - the Source and Destination - by the ETERNUS system. This guarantees that the replication pair remains in Equivalent state. To discontinue the Paired status of an REC the administrator issues either a Cancel or Suspend command.

5 It is also strongly recommended to guarantee data consistency before the REC session is interrupted with the Suspend command; data writes that take place when ETERNUS is changing from Equivalent state to suspended are written on the Source disk and consequently tracked by the bitmap or REC buffer, but would not be written to the Target disk. That would result in inconsistency between the Source and Target data. Consistency can be guaranteed by stopping or freezing all host access to the Source LUN. ACM comes with sample pre- and post-scripts that can be used to guarantee data consistency by stopping or freezing all host access to the Source LUN. It is strongly recommended to use these scripts that are automatically executed by ACM prior and after the Suspend command. Now both Volumes are read and write accessible as independent LUNs. If Suspend was issued then the following modifications will be recorded in a bitmap. 9 Concurrent suspend is a function that ensures consistency between several parallel replication pairs; for example between different database components. Applications like databases use multi-volume access and if these Volumes belong to a REC configuration, it is a challenge to guarantee data consistency between REC sessions when they are suspended. This is practically impossible with the standard sequential command execution. To guarantee consistency, the associated REC replication pairs have to be grouped within ACM or SF Express. To invoke the Concurrent Suspend task for all REC members of the same group they all must have Equivalent state as an indication that the initial copy process is completed. All REC members of the same group can be stopped with the Concurrent Suspend at the same point in time. A Resume command can be issued in sequential order for each individual replication pair without jeopardizing data consistency. 10 The next chapter of this Web Based Training module describe the two different Remote Equivalent Transfer Modes; synchronous and asynchronous. 11 Prior to starting an REC session it is necessary to select the right transfer mode. This slide explains in brief the operating principle of the REC synchronous mode. We have again a Source Volume and a Target Volume in sites A and B. The REC session has reached Equivalent state at this point in time. Now an application writes a data block to the Source Volume and waits for an acknowledgement from ETERNUS DX. In synchronous REC mode the ETERNUS with the Source Volume transfers the data block to the Target Volume residing in the remote ETERNUS system. After the local ETERNUS system receives a write acknowledgement from the remote ETERNUS system, the local ETERNUS system forwards the acknowledgement for this data block to the host server and its application. That means that synchronous transfer mode guarantees that the copy process is fully completed in both systems before the write is acknowledged. But this guarantee has its price; data transmission latencies between the two systems may become a performance issue for the application.

6 Therefore ETERNUS DX offers also an asynchronous REC mode, more about that on the following slide. 12 If the write acknowledgement response times become an issue with synchronous mirroring mode, asynchronous mode is the better option, albeit without acknowledgement that the data has been actually written on the Target Volume. The Source Volume and Target Volume reside again in sites A and B, the replication pair has reached Equivalent state. To demonstrate the characteristics of asynchronous replication there is a second replication pair with Source and Target Volumes. Now the application writes a data block to the first Source Volume and waits for the acknowledgement before proceeding. One major difference between synchronous and asynchronous mode is the REC Queue. This tracks the modifications or the new written data in a bitmap or in a dedicated REC Buffer cache. After a write to the Source, the local ETERNUS DX system sends an acknowledgement back to the server within a normally expected response time. The animation continues with the second application sending new data to the second Source Volume. Also this data block is queued in the local ETERNUS system and the application gets an acknowledgement almost immediately. It is obvious that this mechanism has a big advantage from the application's point of view; the application can immediately proceed with the next write without having to wait a long time for a successful write acknowledgement. The local ETERNUS system uses its own internal algorithm for transmitting the data blocks to the target system. The sending order of the data blocks is determined by the internal algorithm, too. This example demonstrates the disadvantage of the REC asynchronous mode; the data is queued prior to being transferred to the remote system. Should site A experience a hardware failure that prevents it from sending the data out from the queue, the queued data may get permanently lost and consequently the remote site data is inconsistent. 13 REC synchronous transfer mode provides sub-modes that need to be assigned during the REC start invocation. The following chapter explains the Automatic and Manual Split as well as the Automatic and Manual Recovery options used for synchronous transfer mode only. 14 Transfer mode Automatic Split is aimed to manage potential data path failures between the REC systems. This animation illustrates data writes on the Source Volume and the consequent copy functions to the Target Volume. The next three steps show the write requests to the Source and its respective copy step to the Target. If now the data path between the two ETERNUS DX systems fails completely, an automatic disconnect of all REC sessions is performed. With the Automatic Split flag enabled the application is still able to write to the Source Volume while the REC session is interrupted with the status Hardware Suspend.

7 As long as the data path remains interrupted the server continues to write to the Source Volume only, but all modifications are tracked by the bitmap mechanism. After the data path is restored, a re-synchronization with a Resume function has to be performed to bring the Volumes back to Equivalent state. 15 This slide explains how Manual Split differs from an Automatic Split during a synchronous REC session. The animation shows write and copy activities for the Source and Target Volume. The next three steps show again the write requests to the Source and the respective copies on the Target. Now again a complete transmission path error comes about between the two ETERNUS DX systems and all REC sessions are immediately stopped. With the "Manual Split" flag enabled the application is no longer able to write to the Source Volume because ETERNUS has locked the Volume pairs of all REC sessions. As long as this path failure exists the server is not able to write to the Source, the Volumes remain in Equivalent state and consequently no bitmap is needed to track changes. In the case of an actual disaster this sub-mode setting ensures that the Source and Target remain fully synchronized. Issuing a Suspend command releases the Source Volume for write access and initializes the tracking bitmap to record Source Volume changes so that the Target Volume can be again synchronized as soon as the data path is restored. 16 The two recovery options Automatic Recovery and Manual Recovery become important after the data paths have been restored. Automatic Recovery means that the temporarily suspended REC sessions are resumed automatically without any intervention from the administrator. Manual Recovery option means that the administrator must resume the temporarily suspended REC sessions manually. While the replication session is on hold it is possible to create a backup - with OPC or EC - from both Volumes to ensure consistent data backup before new application data and replication data is applied on the Volumes. After the backup copies are made the Resume command for the REC session can be manually invoked. It is strongly recommended that these two manual mode options - Manual Split and Manual Recovery - are enabled and used only by experienced users because they require manual or scripted intervention. 17 Also for the REC asynchronous transfer mode there are a number of options that can be assigned during the REC start invocation. This chapter explains the functionality and usage of the Stack Mode and Consistency Mode. The necessary parameters that are required for the Consistency REC Buffer configuration are explained as well. This is followed by some important REC Buffer transfer parameters. The last topic explains the usage of the Consistency Mode in relation to the different buffer cache configurations.

8 18 The asynchronous REC mode can work either in Stack Mode or in Consistency Mode. Let's start by having a look at the Stack Mode. It uses a bitmap located in the cache memory of the ETERNUS DX Controller Module. This example starts with a situation where one data block has already been updated in the Source Volume. When the next data block in the local Source Volume is updated the corresponding bit in the bitmap is changed to flag the modification. The actual physical data is not cached at this time. After that ETERNUS sends an acknowledgement to the server to acknowledge successful data reception. ETERNUS internal transfer engine scans the bitmap from time to time, thus the write order at the remote destination cannot be guaranteed. Asynchronous REC Stack Mode is useful when the bandwidth of the REC interconnection is limited and consequently large amounts of data can accumulate at the transfer engine. 19 The transfer of the last modification of the same source data block to the remote destination site is a disadvantage of the REC Asynchronous Stack Mode. Let's have look at an example to see why: A write request sends a black data block to the local Source Volume and the corresponding bit in the bitmap is changed. A second write request to the same block address sends the blue content but the corresponding bit in the bitmap is already set to Modified. The ETERNUS DX internal transfer engine scans now the bitmap and finds the set bits for the updated data blocks. The transfer engine cannot recognize from the bitmap that the respective data block was modified twice so it transfers the blue content to the remote destination. REC Asynchronous Stack Mode can lead to data inconsistencies if a malfunction occurs at the ETERNUS source system because only a reference to the data is recorded, not the data itself. REC Asynchronous mode is typically used as a preparation for a disk based remote data backup. The remote data can also be used as a source for a quick data restore or used as the data source should the local source data get lost or corrupted. 20 This slide gives two examples for the use of REC Asynchronous Stack Mode. The first example shows the use of a REC multi copy configuration. Two independent REC sessions use the same Source Volume. Each session is toggled between suspend and resume. This example provides a consistent data set at a particular point in time. The second example shows cascading two ETERNUS DX Advanced Copy functions. Two local Copy functions like OPC or EC are set up for the remote system. In this example both local copy sessions use alternating the same Source Volume. This Source Volume is also the destination or target volume of the REC Asynchronous Stack session. Here again the toggle operation of the two local copy sessions provides a consistent data set at a particular point in time.

9 In case of an access problem with the source data you can fall back on these data sets and restore data from one of the last local disk backups at the destination system. Asynchronous Stack Mode will never reach 100% synchronization for which reason it cannot be normally suspended. Suspend mode is however required for a reverse copy or for data restoration. Suspend functionality can be simulated by interrupting the Stack Mode session and then changing the REC transfer mode to Through Mode. Consequently the un-copied data blocks are copied over according to the bitmap. Now the data is consistent and in the last step a Resume can be invoked and immediately thereafter a Suspend command. Consequently the destination data can be used for a restore. The Through Mode is suitable for some very specific situations and we will come back to it in more detail later in this Web Based Training module. 21 The next three slides show more details for the REC Asynchronous Consistency Mode. This first slide provides a general introduction. The Asynchronous Consistency Mode uses a buffer pair located in both ETERNUS DX systems; one for sending and one for receiving data. One of the advantages of the Consistency Mode is to guarantee the write order across multiple REC sessions. So the data will be sent to the destination site in the same order as they arrived in the send buffer. Data modifications to multiple LUNs are accumulated in the REC Buffer Cache of the source ETERNUS before they are transferred in groups to the destination ETERNUS system. This provides a better efficiency and utilization of the REC interconnection link. 22 This slide gives a more detailed description of the internal usage of the two REC Buffers. As mentioned on the previous slide, Asynchronous Consistency Mode does not use bitmap pointers instead it uses a REC buffer pair that is set up with the ETERNUS DX Manager GUI or CLI. The following animation shows the source site at the top and the destination site in the bottom. At the source site REC Consistency Mode uses a REC Send Buffer and this buffer is automatically divided into two equally sized parts. The same is valid for the REC Receiver Buffer at the destination site. When the application writes data to the Source Volume these data contents are also written in the REC Send Buffer. The administrator can define checkpoints to define at what interval the write process to the active part of the buffer cache is stopped. This takes approximately 100 micro seconds and generates an overhead of 0.1% - that has typically no measurable impact for the application. A checkpoint interval timer is set with the ETERNUS Manager GUI or CLI; typical values being for example one, two or four seconds. The value of this parameter has an impact on the Disaster Recovery key parameter "RPO" because it relates indirectly to how much data can get lost. Indirectly for the reason that the maximum amount of data that can be lost is actually defined by the size of the buffer; during heavy update load the buffer may get full in less than the interval time. At the given intervals - or when the buffer gets full - the data blocks are copied into the second part of the REC Send Buffer.

10 The data blocks in the first part of the Send Buffer are passed over in parallel to the target ETERNUS system and will be stored equally in the first part of the Receive Buffer Cache. Both copy processes are running in parallel; the local copy process to the Send Buffer and the remote copy process to the Receive Buffer. The two ETERNUS systems confirm the completeness of the buffer transfer and only then the remote site copies the data from the Receive Buffer to the Destination Volume of the REC replication. 23 This last of the three slides explains in brief the three REC Buffer parameters that have to be setup in advance and that have a direct impact on the accumulated data and its transfer to the target site. The parameter Forwarding Interval can be defined from 1 second up to 120 seconds - one second being the default value. In the previous slide we referred to this parameter as the check point value; the parameter specifies the time interval for copying the data to the destination, independent from the buffer space usage. The second REC Buffer parameter is called Watch - or Monitor - Timer and is selectable in a range between zero and fifteen minutes. Here the default value is fifteen minutes. If the write load to the send buffer is too high it might become a bottleneck and if the REC Send Buffer remains in this high-load state for the time specified by the Watch Timer the REC session status is changed from Paired to Halt. If set to zero the buffer overload monitoring is disabled. The last REC Buffer parameter is the Halt Wait Timer that can be set between zero and 15 seconds; zero giving the host I/O always highest priority, 15 seconds being the recommended value. This specifies the maximum no-response time in seconds that the ETERNUS system waits before responding to a write request from the host server. During the Send Buffer high-load phase ETERNUS delays the host I/O write acknowledgements to give the ETERNUS internal mechanism enough time to transfer data from the Send Buffer to the remote target ETERNUS. This host "wait I/O phase" is monitored by the ETERNUS and when the host waiting time exceeds the pre-defined threshold of the Halt Wait Timer the ETERNUS session changes its status from Paired to Halt automatically. 24 This slide summarizes some of the REC Buffer Cache configuration parameters. The send and receive buffer parameters are assigned with the ETERNUS DX Manager GUI or CLI. Before using REC the administrator has to define at the ETERNUS source site a Send Buffer and at the ETERNUS destination site a Receive Buffer. Please note that the same ETERNUS system can operate both as a Source and as a Target. The maximum number of REC Buffer definitions is limited to eight. What does this mean in practice? Let's assume there are eight ETERNUS REC Buffer pairs configured. Each buffer pair consists of a send and receive buffer at each site. This configuration allows forward copy sessions from the Source to the Destination site.

11 As such, restore would not be possible but for that purpose ETERNUS SF AdvancedCopy Manager provides a command that changes the role of the REC Buffer Cache. Please note that the REC Buffer Cache is located within the Controller Module cache memory. As the maximal cache size is model dependent, please refer to the respective reference documentation for further recommendations. An alternative functionality is provided by a number of ETERNUS DX models. With these systems it is possible to prevent the REC session from running into a REC Buffer shortage by setting up a REC Disk Buffer instead. 25 This slide explains in more detail the use of multiple REC Buffers. Up to eight REC Buffers in an ETERNUS DX system provides the possibility to establish multiple REC connections between multiple ETERNUS DX systems. That means for example that one ETERNUS system can be at the same time both source and target and that one ETERNUS can be a source for several target systems. A REC Send Buffer located in site A is paired with a REC Receive Buffer in site B. Each copy pair is always unidirectional - from Send Buffer to Receive Buffer - for bidirectional copying two pairs must be established. Bidirectional data transfer is needed for example to be able to restore backed up data from the target back to the source or when both sites have active business applications that require local data access but backups are made in the other site. However it is also allowed to configure a REC bidirectional data transfer between the two cabinets. Please note that it is not possible to set up two parallel send-receive pairs that run in the same direction between the same two systems. 26 The last two slides of this chapter shows couple of practical samples for the use of Asynchronous Consistency Mode. A REC configuration consists of two ETERNUS systems connected by one or two unidirectional data paths. The applications of the server A are using for instance the REC consistence pair number one. The copy direction is from main site A to backup site B. Another potential configuration is to have business applications running in server C in site B using a second REC consistency pair. Its copy direction is from site B to site A. Remote Equivalent Copy and its Buffer mechanism support consistency pairs that are shared by multiple applications. In this example the applications at server B share the REC consistency pair number one with the applications running at server A. Each REC consistency pair can be individually controlled. For example, a disk backup and a data restore session can run parallel between two systems. The application one of server C uses the blue REC session and parallel to that the application two of the same server is recovering its data using the red REC session. Both processes use different REC Buffer pairs to transfer its data. 27 As already mentioned in the beginning of this introduction, Asynchronous Consistency Mode preserves the write order with the help of the REC Buffer mechanism.

12 During an active REC copy session the different data modifications are stored in different REC Buffer sets and stored in the cache memory of the ETERNUS DX Controller Module. The Buffer sets are transmitted to the target in one transaction. The receiving ETERNUS system confirms the reception after the last data block is written to its REC Receive Buffer cache. The sending ETERNUS system holds the data in its Send Buffer cache until a confirmation is received from the receiving system. That guarantees data consistency in case of a transmission failure. 28 The next chapter deals with the details of the Suspend and Resume invocation during an REC session and explains how a line failure in an Asynchronous Mode is internally executed as well as the functionality and usage of the Asynchronous Through Mode. 29 The REC Asynchronous Mode uses a bitmap to track changes when the replication pair is suspended. After the REC session is resumed the bitmap entries are deleted in the target ETERNUS site. The upper part of the picture shows the Source Volume in site A and the lower part shows the target Volume in site B. The session has Paired status and the application writes data to the Source Volume. This new data block is copied to the remote Destination Volume. Next the session is suspended using ACM or SF Express GUI or CLI. When the status Suspend is established, both ETERNUS systems create local bitmaps to record data block modifications. If new data is written to the Source Volume ETERNUS flags the corresponding bit in the bitmap. When new data arrives to the Destination Volume also this ETERNUS system flags the modification by setting the corresponding bit in its bitmap. As the REC session resumes with the command execution from ACM or SF Express GUI or CLI, the two bitmaps are merged. Consequently the Destination Volume is no longer accessible for the server. Next the REC resynchronization process starts to copy the data blocks from Source to the Destination Volume according to the bitmap. Any data block that was updated in the Destination Volume during suspend is overwritten by the respective data block of the Source Volume. After the resynchronization process is completed the session resumes the normal copy process between the two ETERNUS systems and mirrors all Source data changes in the Target Volume. 30 This animation shows the bitmap merge after a Resume is invoked in an REC Asynchronous transfer mode. In suspend mode both Volumes are independently accessible and the first modification is written to the Source Volume. Consequently the corresponding bit is flagged in the bitmap. The next modification takes place in the Destination Volume and also here the corresponding bit is marked in the bitmap.

13 Two consecutive modifications take place in the Source Volume and are tracked by the bitmap. The last data update takes place in the Destination Volume and gets tracked by the local bitmap. Let's see what happens when the REC session resumes. The Destination Volume is locked and the two bitmaps are merged. The first bit indicates that the first data block of the Source Volume was modified. The third bit signals a modification in the Destination Volume. This information is needed so that the original data block from the Source Volume gets copied to the Destination. The fifth and sixth bit indicate respectively the modified data blocks in the Source Volume. After the bitmap merge is completed the synchronization copy process starts by copying the first data block to the Destination Volume, followed by the fifth data block and continued with the third and the sixth data block. Now both disk contents are equivalent and the REC session changes its status to Paired. 31 This slide illustrates what happens in a REC Asynchronous session when an interconnection failure occurs. During the line error the ETERNUS DX system in the source site records all modifications in a bitmap. The upper part shows the Source Volume in site A and the lower part the Destination Volume in site B. The session is active and has a Paired status so all Source data writes are copied to the Destination. As a line error occurs both ETERNUS systems change their status to Halt. The Source Volume remains accessible. The first write after REC path failure returns sense information to the server and consequently the bitmap is flagged. After the interconnect path is restored the Halt status ends. During the Halt status all Source Volume changes were tracked by the bitmap. Next begins the re-synchronization process. According to the bitmap contents - but now only the source bitmap is available - the resynchronization copies all flagged data blocks to the Destination Volume, without maintaining the write order of the changes. After the re-synchronization the session returns to Paired status. 32 REC Asynchronous Mode can run in three different modes; "Stack", "Consistency" and "Through Mode", also known as "Sequential Transfer Mode". With the following two slides we will have a closer look at this third mode. This mode is used to synchronize not yet copied data blocks of the Source Volume if a "Stack" or "Consistency" mode session is stopped or suspended and cannot be resumed or restarted. This mode should be used when an REC Asynchronous session is stopped or suspended and after that an interconnect path failure occurs. That can result in a situation where modified data should be copied from Source to Destination. After the path is restored the current Asynchronous mode must be changed from "Stack" or "Consistency" to "Through Mode" so that the pending copy process can start.

14 Remember the earlier example with "Stack Mode" where a "Stack Mode" session was forcibly stopped and the mode was changed to "Through Mode" to guarantee data consistency at both sides. 33 This slide describes the functionality of the REC Asynchronous Through Mode or Sequential Transfer Mode. The animation illustrates the data transfer process between the server, ETERNUS DX Copy Source and the ETERNUS DX Copy Destination. The server wants to write data in the ETERNUS Copy Source and gets an acknowledgement for this first data. In the next step the Copy Source system copies the data to the Copy Destination system. Now the server sends the next write request to ETERNUS. At this moment the Copy Source system is still waiting for an acknowledgement from the Copy Destination system for the first write and only after having received the first acknowledgement it acknowledges the second write. That means the Copy Source ETERNUS has to wait for each previous remote acknowledgment before it can response to the next pending write process. As a result the data transfer performance equals that of the "Synchronous Mode". That is why you should use this special Asynchronous "Though Mode" only under the circumstances described in the previous slide. 34 The following three slides explain REC network bandwidth requirements, explain the differences between the REC modes and provide an overview of the different REC modes. 35 This slide helps to estimate the needed transfer capacity for the one-way interconnect path between two systems. For the Synchronous or Asynchronous Consistency mode the amount of data updates in the peak time should match with the maximum line bandwidth. When considering the bandwidth requirements please bear also in mind the performance of the SAN and WAN network components - and any potential converters along the network path. For the Asynchronous Stack mode the volume of data updates should match with the average load that is expected in the data update. Especially over longer distances - to keep the transfer line costs optimal - the required bandwidth is allowed occasionally to exceed the available line bandwidth. 36 This table provides a comparison between the different REC Modes. The leftmost column shows the features that are compared between the Synchronous Mode and Asynchronous Stack and Consistency Modes. Let's start with the topic "Main Usage". "Synchronous Mode" is used for local configurations, for instance within the same building or data center. The data mirroring between two ETERNUS systems provides High Availability of the storage - with or without cluster configuration - or can be used for pure data backup only.

15 The Asynchronous Stack Mode is used for remote installations, where the amount of updated data is reasonable. That could be for example a remote data backup as a Disaster Recovery solution. The Asynchronous Consistency Mode is the right option for remote installations where the amount of updated data is relatively high. How does the mode selection affect the host server responses? In case of the Synchronous Mode after each write the server has to wait for the acknowledgement from the source ETERNUS system. The impact can be very big if the data transfer latency is high - caused for example by low performance of the interconnect path between the systems. With Stack Mode the impact of the write acknowledgment is normally lower because the source ETERNUS response times depend on the local network performance only. The same impact exists also for the Consistency Mode, albeit the REC buffer cache mechanism provides a mechanism that reduces the performance impact. Correct data sequence during the transfer phase can be guaranteed in the Synchronous Mode because the next data block is sent to the remote system only after the previous write has been acknowledged. The Asynchronous Stack Mode does not guarantee the order of writes because the transfer sequence is driven by a control bitmap that ignores the write order. With the help of the REC Buffer cache the correct order of the data transfer is guaranteed in the Asynchronous Consistency Mode. Next we have a look at the difference between the commands "stop" and "suspend" in the different transfer modes. In the Synchronous Mode it is possible to invoke these commands at any time when the REC session has Equivalent state. After Asynchronous REC session is suspended or stopped it cannot be simply resumed; before resume the transfer mode must be changed to Asynchronous Through Mode to guarantee that all changed data gets transferred to the remote system. After the Volumes achieve Equivalent state the Through Mode can be stopped or suspended. What is the maximum number of ETERNUS system can be connected with REC? Synchronous Mode configuration supports one source ETERNUS system and up to sixteen remote ETERNUS systems. The same is valid for Asynchronous Stack Mode. Because of the REC Buffer cache pairs, up to eight remote ETERNUS systems are supported with the Asynchronous Consistency Mode. Multiple copy destinations can be specified to obtain more than one copy of the original data, allowing copies of the same Source data for different purposes, such as backup and testing. This multi copy function is supported by all three transfer modes. Which controller topology can be used for REC connectivity? For the Synchronous Mode only the Fibre Channel is released. The two Asynchronous Modes can be run with either Fibre Channel or the REC iscsi controller. Any unused FC port can be used by changing the port usage flag to Remote Adapter. Earlier DX models required a separate Remote Adapter for replication over iscsi, with S2 models the same physical port can be configured either for host or replication usage. Remember that a port configured for REC is no longer usable to map open volumes to the host server.

16 The last table entry concerns the Asynchronous Consistency Mode only and reminds about the necessity to set up the parameters for the buffer caches using the ETERNUS Web GUI or CLI. 37 This slide provides a summary of the main characteristics of the three main REC modes. Synchronous REC Mode guarantees the write order of the data but the application has to wait for the confirmation that the data is written in both ETERNUS DX systems. This has a high impact on the server write performance but guarantees that no data can get lost. The Split Mode defines how the system behaves in a case of an interconnect path failure; the REC session needs to be interrupted to allow application access to the Source Volume, this can happen either automatically or manually by the administrator. The Asynchronous Stack Mode does not guarantee the data transfer order. Application receives write acknowledgements with low latency meaning this mode provides good server write performance. As a downside data writes in the remote system cannot be guaranteed and therefore this REC mode is vulnerable for data losses. Equal to Synchronous REC Mode also the Asynchronous Consistency Mode guarantees data transfer order. Data transfers to the remote system can be managed with check point parameters but data writes to the remote system cannot be guaranteed. 38 This next chapter provides a brief introduction and practical advice for the use of REC Start+Suspend and Resume+Remain functions that can be used for the replication pair to skip the initial copy process. 39 In a Disaster Recovery configuration where the two ETERNUS DX systems are physically far apart from each other - for example in Munich and Tokyo - the first initial copy phase can take a very long time. To avoid this there is a special command available that skips the initial copy sequence. On the left hand side of the animation there is a Source Volume in site A and Target - or Destination - Volume in site B. The REC session is not yet started so at the moment both Volumes are independent from each other and open for data write. Normally when the REC session is started the Destination Volume is locked and the initial copy phase starts. That can be avoided by using the start option Start+Suspend. That means the REC session is started but immediately suspended and therewith also the initial copy process is suppressed. After Start+Suspend a block level backup of the Copy Source is made using common backup software. After that normal operation at the business site can be started while the REC replication remains suspended. From now onwards the application can write to the Source Volume and the modifications are recorded by the bitmap. Next step is to restore the backup of the Source data to the Destination Volume.

17 Because the replication pair has the "suspend" status the Destination Volume is not locked and therefore normally accessible. After the complete Source data is manually restored, the REC session can be resumed, but now with the other special command option "Resume+ Remain". Consequently the REC session proceeds with a normal resume operation and uses the bitmap to copy over the modified Source data to the Destination site. After the resynchronization process is completed the REC session runs in normal mode and mirrors all Source Volume changes to the Destination Volume by using the interconnect path. 40 Here is some additional advice for using the "skip initial copy" feature. If possible, the Copy Source data should be frozen or idled from application point of view after the Start+Suspend option is executed. This ensures that the re-synchronization phase is as short as possible. Before issuing the "Resume+Remain" command it has to be guaranteed that the Destination Volume is 100% identical with the Source Volume as it was at the time of the "Start+Suspend" invocation. Also note that the ETERNUS DX systems will not check the data consistency after the "Resume+Remain" invocation. That means that the REC session shows Equivalent state even though the data may in reality be inconsistent. 41 The last chapter of this Web Base Training module deals with Cascade Copy and Multi Copy configurations. 42 This slide demonstrates two examples for cascading several ETERNUS DX Advanced Copy sessions in conjunction with the REC. In such constellations a Volume can have two roles; first it can be a Destination of a copy session and at the same time it can be the Source Volume of a second, separate copy session. ETERNUS system number one holds two Volumes. These Volumes can be for example members of a local Advanced Copy Function like OPC, QuickOPC or EC. In this configuration the Volume on the right hand side is the Destination Volume of the copy session number one. The second system - ETERNUS DX number two - has also two volumes. Now a REC session - for example in Stack Mode - can be set up with the right most Volume of the DX number one as the Source and the left most Volume of DX two as the Destination. Furthermore, a third ETERNUS DX system also with two Volumes could be used to cascade a further copy session. For example a REC session - in Synchronous, Consistency or Stack Mode - could mirror the Source Volume of the DX system two with a Destination Volume in DX system three. This last REC session could be cascaded with a further local copy session within the third ETERNUS DX. The left most Volume plays two roles: in one hand it is the Destination Volume of a REC session and secondly it is also the Source Volume for a local Advanced Copy session. This local copy session could be any one of the available types: OPC, QuickOPC, SnapOPC or EC.