EMC CLOUD-ENABLED INFRASTRUCTURE FOR SAP BUSINESS CONTINUITY SERIES: DISASTER RECOVERY BUNDLE VNX

Transcription

1 White Paper EMC CLOUD-ENABLED INFRASTRUCTURE FOR SAP BUSINESS CONTINUITY SERIES: DISASTER RECOVERY BUNDLE VNX EMC RecoverPoint, EMC next-generation VNX, VMware vcenter Site Recovery Manager, VMware vcloud Suite, VMware PowerCLI Protection of mission-critical SAP landscapes in the cloud Cloud infrastructure resiliency Automated cloud infrastructure disaster recovery EMC Solutions Abstract This white paper focuses on the disaster recovery add-on bundle of the Cloudenabled Infrastructure for SAP. It explains the transformation of a single data center solution into a mission-critical business continuity solution, which protects data from full site disaster and increases resiliency of operations. The solution is enabled by EMC RecoverPoint, EMC next-generation VNX, VMware vcloud Suite, VMware vcenter Site Recovery Manager, and VMware PowerCLI. December 2013

2 Copyright 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H

3 Table of contents Executive summary... 6 Business case... 6 Solution overview... 7 Key benefits... 7 Introduction... 8 Purpose... 8 Scope... 8 Audience... 8 Terminology... 8 Technology overview Introduction Physical architecture Hardware resources Software resources Disaster recovery for Cloud-Enabled Infrastructure Introduction Disaster recovery logical architecture Recovery modes DR operational cycle Failover process Workflow Failback process Workflow Setting up disaster recovery for CEI Virtual Data Center Introduction Configuring VMware vcloud Director Configuring the network Configuring infrastructure Chargeback Manager EMC storage infrastructure Introduction Configuring VNX storage EMC RecoverPoint Introduction Configuring RecoverPoint consistency groups VMware vcenter Site Recovery Manager

4 Introduction Integrating SRM with RecoverPoint Configuring VMware SRM SAP system architecture Introduction SAP ERP SAP system architecture Key SAP design considerations Design best practices Data center RecoverPoint VNX storage Cloud SAP Test and validation Overview Notes Test Scenario 1: Failover after disaster Test objectives Test procedure Results and analysis Test Scenario 2: Resynchronizing to Data Center A Test objectives Test procedure Results and analysis Test Scenario 3: Planned migration Test objectives Test procedure Results and analysis Test Scenario 4: Reprotection for Data Center A Test objectives Results and analysis Conclusion Summary Findings References EMC documents

5 VMware documents SAP documents Appendix Configuring consistency group Configuring VMware SRM Site pairing Array-based replication Resource mapping Protection groups Recovery plans DR design for three data centers

6 Executive summary Business case For most businesses, the focus on the customer s experience has led to more opportunities and competition. Businesses have expanded beyond the traditional geographical boundaries and markets, interacting directly with consumers. Capitalizing on this opportunity requires: Responsiveness Consumers have choices. If they are not satisfied with the service level they receive from one source, they go elsewhere to find what they need. Market and consumer intelligence Availability of the most current data and information is critical to understanding customer behavior and preference, which in turn helps to maintain a competitive advantage. Supply, demand, and order fulfillment management What may seem to be a simple transaction to an ecommerce consumer in reality necessitates the seamless availability of a federated landscape of interdependent business applications. As IT systems embark on their journey to the cloud, the importance of business continuity has become increasingly critical to the operations of an enterprise. Only one extended downtime may cause irrecoverable loss for many businesses. However, proper preparation typically involves a significant investment of time and cost to keep up with a company s recovery time objectives (RTOs) and recovery point objectives (RPOs) during a disaster. Designing disaster recovery (DR) solutions poses many challenges, including: Ensuring RTO and RPO for varying SAP applications in different organizations in a cloud environment Testing the recovery plan periodically Reconfiguration effort needed to prepare another site for DR Human intervention effort needed to carry out DR for hundreds of systems Data protection by using backup and restore is designed to protect the business data from isolated system failure. However, in a full data center disaster, backup strategies no longer meet these challenges. Besides, backup and restore strategies for restoring hundreds of SAP systems to DR within the restriction of RTOs expose system administrators to human error. This white paper, specifically designed to protect SAP mission-critical applications in a cloud environment, introduces an EMC solution that addresses all these challenges. 6

7 Solution overview This solution provides simplified and reliable DR for SAP systems and management infrastructures. We 1 built the solution on top of the Cloud-enabled Infrastructure (CEI) for SAP and used EMC RecoverPoint and VMware vcenter Site Recovery Manager (SRM) as the primary enabling technologies. EMC RecoverPoint is a high-performing, reliable, and flexible replication solution designed for enterprise-class DR. VMware SRM, when integrated with RecoverPoint, can fully orchestrate the entire recovery cycle, from the failover until the SAP systems are ready for user access. This white paper demonstrates how the following technologies support the solution: The EMC next-generation VNX flash-optimized unified storage platform delivers innovation and enterprise capabilities for file, block, and object storage in a single, scalable, and easy-to-use solution. Ideal for mixed workloads in physical or virtual environments, VNX combines powerful and flexible hardware with advanced efficiency, management, and protection software to meet the demanding needs of today s virtualized application environments. EMC RecoverPoint provides a robust replication mechanism between the VNX storage arrays located in production and at the recovery data centers. Its data compression technology reduces bandwidth requirements to minimize overhead. VMware SRM integrates seamlessly with the RecoverPoint system and VNX storage to orchestrate the DR process. VMware vcloud Director applies the principles of pooling, abstraction, and automation to all data center services, such as compute, storage, networking, and security. It enables the multitenancy business models for IT operation and provides the portal for self-service provisioning. Key benefits This solution adds DR functionality to the CEI for the SAP foundation bundle by providing: Increased resiliency during disasters by automating the failover and failback sequences for both management applications and SAP systems in the cloud environment Well-organized and highly coordinated recovery plans for a trouble-free failover The convenience of triggering the failover from any surviving site by a few clicks of a button Minimized network overhead during replication 1 In this white paper, "we" and our refers to the EMC Solutions engineering team that validated the solution 7

8 Introduction Purpose This white paper describes a solution that provides DR for SAP systems running on the EMC CEI for the SAP foundation bundle. This solution provides automated DR with seamless integration of RecoverPoint and VMware SRM in a vcloud environment. This white paper: Introduces the key enabling technologies Describes the solution architecture and design Describes how the key components are configured Presents the test results demonstrating the increased availability of the environment Identifies the key business benefits of the solution Scope Audience Terminology The solution discussed in this paper focuses mainly on its integration with a standalone foundation bundle on a VNX platform. However, the solution is fully compatible with any other CEI add-on bundles that may already be in place, such as the high availability (HA) and application mobility bundle and data protection bundle. This paper does not discuss additional configurations required to work with the HA bundle. This white paper is intended for SAP BASIS administrators, storage administrators, IT architects, and DR managers responsible for designing, creating, and managing business continuity, especially for customers who have implemented CEI for SAP. Understanding this paper requires a basic understanding of the concepts and workflows used by the CEI for SAP foundation bundle. We highly encourage you to read the EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle VNX White Paper first before continuing. This white paper includes the terminology in Table 1. Table 1. Terminology Term AAS ABAP ASCS Asynchronous replication Description SAP Additional Application Server (previously known as Dialog instances). SAP Advanced Business Application Programming. ABAP System Central Services. The process of committing data writes when the data has been sent out for replication without waiting for acknowledgement from the receiving side. Asynchronous replication provides crash-consistent protection and recovery to specific points-in-time, with a small RPO. 8

9 Term Consistency group Copy Deduplication ERP IDES Journal volume OvDC PAS Protection Group PvDC Repository volume RPO RTO Description RecoverPoint configuration method that logically groups data sets so that they are consistent with one another. The consistency groups ensure that the updates to the production volumes are written to the replicas in a consistent write order and that the replicas can always be used to continue working or to restore the production source. A logical RecoverPoint entity that constitutes all volumes of a consistency group. A specialized data reduction technique that eliminates duplicate copies of repeating data. Enterprise Resource Planning. SAP Internet Demonstration and Evaluation System. An SAP system with preconfigured settings and sample data. A RecoverPoint volume that stores time-stamped application writes for later recovery to selected points-intime. Organization Virtual Data Center. A vdc abstraction, which is partitioned from a PvDC that maps resources to an organization. SAP Primary Application Server. A collection of virtual machines and templates that you protect together by using VMware SRM. Provider Virtual Data Center. A vdc abstraction that combines the computer and memory resources of a single vcenter Server resource pool with the storage resources of one or more datastores connected to that resource pool. A volume that stores configuration information about RPAs and consistency groups. This volume enables a surviving RPA to take over the replication activities of a failed RPA from the same cluster. Recovery point objective. The maximum tolerable period in which data might be lost from an IT service. Recovery time objective. The maximum amount of time allowed to resolve an issue. 9

10 Technology overview Introduction EMC, VMware, and SAP have joined forces to design a complete hardware and software solution that supports an on-premises private cloud. This initiative is now known as the CEI. Its architecture is divided into six different functionalities, delivered as separate bundles, as illustrated in Figure 1. Within the CEI, customers can choose the cloud functionalities they want to enable first without losing sight of their end goal an efficient and reliable private cloud. Figure 1. Cloud-enabled Infrastructure for SAP The CEI is based on a mandatory component called the foundation bundle. Table 2 outlines the functions and benefits of the CEI foundation bundle. EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle VNX White Paper provides more information about the foundation bundle. Customers can then develop their own roadmaps for what to build next. Table 2. On-premises EMC Cloud-enabled Infrastructure for SAP solution: Foundation bundle Function Benefits Technology Virtual data center for SAP Infrastructure chargeback Autonomy of business units and application operations Service catalogs Service level agreements (SLAs) Management of vcloud tenants Resource pooling Cost measurement, analysis, and reporting of the use of compute, network, storage, and backup resources (in combination with data protection add-on bundle) VMware vcloud Suite Enterprise VMware vcenter Chargeback 10

11 Function Benefits Technology Integrated cloud management and performance analysis Storage tiering Cloud networking and security for SAP Manage availability, capacity, performance, and health in the SAP landscape Automatically get the right data to the right place at the right time Cloud-enabled Infrastructure security framework Authorization concepts Compliance and non-compliance tracking VMware vcenter Operations Manager EMC Storage Resource Management EMC Storage Analytics (ESA) EMC Virtual Storage Integrator (VSI) EMC VNX EMC Fully Automated Storage Tiering (FAST ) Suite VMware vcloud networking and security Table 3 describes the add-on bundles that enable additional cloud functionality and data center enhancements. Table 3. EMC Cloud-enabled Infrastructure for SAP: Add-on bundles IT strategy Bundle Benefits Technology Business continuity HA and application mobility for SAP High availability within one data center and across two data centers with application awareness for the complete SAP landscape Nondisruptive movement of applications from one data center to another data center EMC VPLEX Metro vsphere HA VMware vsphere Distributed Resource Scheduler (DRS) Symantec ApplicationHA Improved resource utilization across data centers Minimize downtime for infrastructure maintenance DR for SAP Provide disaster recovery protection for cloud management applications and SAP systems VMware SRM EMC RecoverPoint Data protection for SAP Provide data protection to cover the cloud management applications and SAP systems Backup and recovery at all data centers with remote replication of backup sets for offsite protection EMC Avamar EMC Data Domain EMC Data Protection Advisor 11

12 IT strategy Bundle Benefits Technology Service support/ provisioning Automation and operations for SAP SAP application virtualization enables any service any time on any server SAP NetWeaver Landscape Virtualization Management Provision SAP systems on demand with automated endto-end process Reduce downtime window during maintenance leveraging mass operations Enhanced security Enhanced security and compliance for SAP Efficient, collaborative enterprise governance, risk and compliance (egrc) program across IT, finance, operations, and legal domains RSA Archer egrc RSA Data Loss Prevention RSA SecurID Data loss prevention Secure user to network authentication The stand-alone foundation bundle combined with all five add-on bundles form the fully realized CEI for SAP. Physical architecture The full physical architecture of the CEI for SAP consists of three data centers. Production Data Center A and Data Center B are co-located within approximately 100 kilometers (km), with cross-site IP/SAN connectivity and Layer 2 network. Data Center A and Data Center B together provide active/active access, cross-data-center high availability, application mobility, and maximum resource utilization. In addition, Data Center A and Data Center B are equipped with RecoverPoint to provide DR, while Data Center B is also equipped with Avamar and Data Domain to provide data protection. A third data center (Data Center C) is situated remotely in a different failure domain, serving as a standby recovery data center that contains replicated datastores and backup copies. Data Center C also houses a VPLEX witness cluster to monitor the status of the two production sites. EMC Cloud-Enabled Infrastructure for SAP VNX Reference Architecture provides more details. In this white paper, we focus on a two-site scenario (Data Center A and Data Center C), where the DR bundle is implemented on a foundation bundle without a dedicated high-availability data center (Data Center B). The DR configuration for Data Center B is similar. See DR design for three data centers in the Appendix for the design overview for three data centers. In this solution, we built Data Center A for the cloud services upon which all the management and SAP systems run. We built Data Center C as a remote standby data center to add DR capabilities. Data Center C is located far enough from Data Center A to avoid a natural disaster from affecting both sites. The hardware resource and network topology in Data Center C and Data Center A are quite similar, as shown in Figure 2. 12

13 Figure 2. Physical architecture The ESXi servers and storage are totally reserved for DR. Proper sizing should be taken into account before providing DR capability. For example, if the total resource for all tenants subscribing to DR capability in Data Center A is 200 GHZ CPU, 2048 GB memory, and 50 TB storage, the resource reserved in Data Center C should be no less than this capacity. Hardware resources Table 4 lists the hardware resources for this solution and includes the minimum requirements for validating the solution. Table 4. Solution hardware resources Purpose Quantity Configuration Storage (Data Center A) 1 EMC VNX8000 Storage (Data Center C) 1 EMC VNX5600 Storage replication 4 Two RecoverPoint Appliance (RPA) clusters: Two RPAs in Data Center A Two RPAs in Data Center C VMware ESXi servers for the management cluster 4 Two eight-core CPUs, 256 GB RAM: Two ESXi servers in Data Center A Two ESXi servers in Data Center C 13

14 Purpose Quantity Configuration VMware ESXi servers for the resource cluster SAN-switching and networkswitching/routing platform 4 Two eight-core CPUs, 256 GB RAM: Two ESXi servers in Data Center A Two ESXi servers in Data Center C 4 Backbone SAN switches 2 Backbone Layer 3 IP switches 2 Routers 4 10 GbE Layer 2 IP switches Software resources Table 5 details the software resources for the solution. Note: Other operating system and database combinations supported by the EMC Solutions Support Matrix (SSM) can also be used. Table 5. Solution software resources Software Version Purpose EMC Block OE Operating environment for VNX Active Directory Windows 2008 Central hostname and user management EMC PowerPath /VE 5.8 Multipathing software EMC Unisphere VNX management software EMC RecoverPoint 4.0 P1 RecoverPoint software EMC RecoverPoint Storage Replication Adaptor 2.1 EMC RecoverPoint integration with VMware SRM SAP ERP 6.0 EHP 5 IDES/Unicode Reference system Microsoft SQL Server 2012 Used for all SAP systems Microsoft Windows Server 2008 R2 SP1 Operating system (OS) for SAP VMware vcenter Chargeback Manager Chargeback software VMware vcenter Server 5.1 Management and resource clusters Microsoft SQL Server 2008 R2 Used by VMware SRM, VMware vcenter, VMware vcenter Chargeback Manager, and vcloud Director VMware vcloud Networking and Security 5.1 Networking and security software for vcloud Director VMware vcloud Director 5.1 Cloud management tool for provisioning virtual data center VMware vsphere 5.1 Hypervisor hosting all virtual machines 14

15 Software Version Purpose VMware vsphere PowerCLI 5.1 Release 2 Windows PowerShell plug-in for managing vsphere VMware vsphere PowerCLI for Tenants 5.1 Release 2 Windows PowerShell plug-in for managing vcloud Director 15

16 Disaster recovery for Cloud-Enabled Infrastructure Introduction Disaster recovery logical architecture A DR solution is typically complex because a cloud landscape often involves multitenancy and large-scale environments. This solution integrates products from VMware and EMC to automate the DR process and reduce the complexity of the recovery plan. This section includes the following topics: DR logical architecture Recovery modes Operational cycle Failover process Failback process Figure 3 shows the logical architecture for DR. Figure 3. Disaster recovery logical architecture 16

17 In this solution, Data Center A is the production site while Data Center C is the remote recovery site where hardware resources are reserved for DR. We focus on an active/standby DR scenario where hosts at the recovery site are unused under normal conditions. We separated the DR process into two phases: Management group DR The virtual machines in the management group are not managed by vcloud Director. You can fully automate the DR process with native support of VMware SRM. VMware vcenter servers are deployed on both data centers. They manage their own ESXi hosts individually. VMware SRM servers are also deployed on both data centers. They continuously synchronize data with each other so that the failover and failback operations can be executed on any of the available data centers. The DNS server in Data Center C is configured to operate in slave mode. All DNS records updated in the master DNS server in Data Center A are replicated to the slave DNS server in Data Center C to ensure consistent DNS records across the entire infrastructure. Because of their unique replication technique, the DNS, VMware vcenter, and VMware SRM servers no longer need to be replicated at the storage level by EMC RecoverPoint. We highly recommend deploying a backup and restore strategy using the data protection bundle to protect these management applications against OS corruption or other logical errors. Resource group DR vcloud Director manages the SAP systems in the resource group. We used the RecoverPoint CLI and VMware PowerCLI to automate DR for this phase 2. The VMware SRM server executes the scripts after successful recovery of the management group. ESXi hosts in both Data Center A and Data Center C form a cross-data-center DRS cluster called the advanced cluster. This cluster is managed by a single vcenter server, as shown in Figure 3. ESXi hosts in Data Center C remain in maintenance mode so that the workload does not run in the recovery site under normal circumstances. When a failover occurs on virtual machines belonging to the resource group, the VMware PowerCLI script we developed shifts the ESXi servers from maintenance mode to active mode, ready for failover. We used the same VMware SRM to orchestrate both sequences. 2 vcenter Site Recovery Manager 5.1 (or earlier) does not support the protection of SAP workloads in vcloud Director (resource group) at this time. VMware vcloud Director Infrastructure Resiliency Case Study provides details about the integration between VMware SRM and vcloud Director. 17

18 Recovery modes This solution, through VMware SRM, allows two modes of recovery: Planned Migration and Disaster Recovery, as shown in Figure 4. Figure 4. Types of recovery in VMware SRM Both types attempt to replicate recent changes to the recovery site, but they differ in error tolerance. The Disaster Recovery option is used during the failover process. Real-life disasters often present uncontrollable factors and unforeseen consequences. However, the prime objective of DR is system availability; that is, to reinstate as many services as possible within the RTO and RPO. If the outage breaches the maximum tolerable downtime (MTD), the business is at critical risk of irrecoverable damage. In this case, the organization would rather resume the service first to relieve the operations from immediate danger, and deal with the smaller problems later. This is why the Disaster Recovery option is intended for actual site-wide impact. The Planned Migration option is used as a step in the failback process. Planned migrations only succeed when no errors are encountered during their execution. This guarantees a trouble-free failback normal operations are finally resumed. The following section demonstrates how to apply the recovery options in the operational cycle of DR. 18

19 DR operational cycle All protected systems, from failover to failback, undergo a series of six states. Figure 5 shows transitions from one state to another. DC stands for data center and S1 through S6 represent the protection states at the data centers. Figure 5. Operational states transitions 19

20 Table 6 describes the states during the failover and failback processes. Table 6. State S1 S2 S3 S4 S5 S6 Back to S1 Failover and failback transition states Description Both resource group and management group workloads run in Data Center A. Data Center C serves as the recovery site where data are replicated. A disaster causes Data Center A to fail. The entire infrastructure becomes unavailable, including the ESXi servers, network, storage, and RPA. IT administrator triggers a DR in VMware SRM at Data Center C. Data Center C becomes the production site and all virtual machines in both the resource group and the management group become active on Data Center C. Service is restored at Data Center C while Data Center A is being returned to an operational state. Until IT restores the original data center (Data Center A), remote protection is no longer possible. Data Center A recovers and becomes functional. The VMware SRM server and vcenter server in Data Center A are back online and resynchronized with Data Center C. Storage failback has not yet commenced. The effort required to rebuild and configure Data Center A depends on the disaster s impact on the site. At worst case, where all hardware devices are irrecoverably damaged and a complete data center rebuild is necessary, you need to reconfigure VNX, RecoverPoint, VMware SRM, and VMware vcloud Suite. The detailed steps are described in the following sections: Virtual Data Center EMC storage infrastructure EMC RecoverPoint VMware vcenter Site Recovery Manager IT administrator triggers a reprotect operation in VMware SRM to reverse the original replication direction (that is, from C to A). After reprotection, RPA protects Data Center C and Data Center A becomes the DR data center. A downtime maintenance is scheduled. IT administrator triggers a planned migration. After migration, Data Center A becomes the production data center once again, but is yet to reestablish replication as a final post procedure. Finally, reprotection is triggered again to restore the replication from Data Center A to Data Center C. Afterward, Data Center A becomes protected once again and the environment is now considered to be fully restored to its normal state. 20

21 Table 7 summarizes the major elements of the six states during the failover and failback processes. Table 7. Failover and failback process states States Activity Role of Data Center A Role of Data Center C Replication Workloads run on S1 Normal activity Production Recovery A -> C Data Center A S2 S3 S4 S5 S6 S1 A full data center outage occurs after a disaster Workloads fail over to Data Center C IT rebuilds Data Center A Resynchronize replication Migrate workload back to Data Center A Resynchronize replication Outage Standby None None Outage Production None Data Center C Standby Production None Data Center C Recovery Production C -> A Data Center C Production Standby None Data Center A Production Recovery A > C Data Center A Failover process The failover operation consists of states S1 through S3 in Figure 5. The entire failover process includes: 1. Preparing the storage in Data Center C which consists of the RPA operations. 2. Failing over the management and resource groups: VMware SRM automates the management group failover. Custom PowerCLI scripts automate the resource group failover. Workflow After a successful management group failover, you can initiate the resource group failover. Figure 6 shows the key steps involved in failing over the resource group. 21

22 Figure 6. Resource group failover workflow To execute the failover process, perform the following steps, as shown in Figure 6: 1. Enable image access for remote copy in Data Center C. This makes the volumes in Data Center C accessible to the ESXi hosts. 2. Fail over the volume to the remote copy. This turns the remote copy into the production copy while the original copy is not available. 3. Disable VMware High Availability (VMHA) for the DRS cluster. This ensures that VMHA does not restart the SAP virtual machines. We started vapps in Step 8 by using a script so that the correct startup sequence within the vapp will be followed. 4. Rescan the host bus adapter (HBA) and virtual machine file system (VMFS) volumes on the ESXi hosts in Data Center C. All virtual machines in the resource group appear Inactive in the vcenter server inventory after rescanning. Note: Resolving the unresolved volume issue arising from a universally unique identifier (UUID) conflict is essential. You must perform this process with caution to ensure that the mount process does not result in resignaturing volumes. Critical to the success of the approach described in this solution is that no MoRef or UUID changes occur. 5. Perform Exit maintenance mode for ESXi servers after you mount the datastores. 6. Register the virtual machines to the ESXi hosts because VMHA is disabled to prevent the virtual machines from automatically restarting. 7. Start the vshield Edge gateways before starting the SAP systems to ensure that the networking, security settings, and rules are ready. 8. Start all the vapps in vcloud Director using vcloud PowerCLI scripts. This ensures proper start order within the vapp. In this solution, ABAP System Central Services (ASCS) instance is started first, followed in order by the database instance and then the application instances. 22

23 Failback process The failback operation includes states S4 through S6 in Figure 5. The failback process requires the following VMware SRM operations: Execute the reprotect process of Data Center C. This operation reestablishes replication with the direction reversed, that is, from Data Center C to Data Center A. Perform a planned migration from Data Center C to Data Center A. This operation requires downtime for both the management group and resource group. The required downtime depends on the number of ESXi servers, datastores, and virtual machines. Test Scenario 3: Planned migration demonstrates an estimation of downtime based on our environment. During a planned migration, all applications and storage data are reinstated to Data Center A. After this state, the business operations can be resumed, but Data Center A remains unprotected until storage replication is reestablished. Execute the reprotect process of Data Center A. This reestablishes the RecoverPoint replication from Data Center A to Data Center C, returning both data centers to their normal operating state. Data Center A then becomes protected against another disaster. Workflow Figure 7 shows the failback workflow. Figure 7. Resource group failback workflow 23

24 To execute the failback process, perform the following steps, as shown in Figure 7: 1. Restart replication in RecoverPoint once Data Center A is fully functional. VMware SRM can restart the replication for the datastores used by the management group in the reprotect operation. You must execute the RPA script for the resource group on VMware SRM server. A successful reprotection ensures replication of datastores from Data Center C to Data Center A. Thus, the later planned migration can be executed with consistent data. 2. Shut down all the workloads including vshield Edge gateways resource group before planned migration. In this solution, the shutdown operations are included in the scripts. 3. Unmount the datastores from all ESXi hosts in Data Center C. 4. Enable image access for remote copy in Data Center A. This allows ESXi hosts access to volumes in Data Center A. 5. Fail over the volume to the remote copy. This turns the remote copy into the production copy while the original copy becomes replica. 6. Disable VMHA for the DRS cluster. This ensures that VMHA does not restart the SAP virtual machines. We started the vapps in Step 10 by using a script so that the correct startup sequence within the vapp will be followed. 7. Rescan the HBA and VMFS volumes on the ESXi hosts in Data Center A. 8. Enable Exit maintenance mode for the ESXi servers in Data Center A after mounting the datastores. 9. Register the virtual machines to the ESXi hosts, because VMHA is disabled to prevent the virtual machines from automatically restarting. 10. Start the vshield Edge gateways before the SAP systems to ensure that the networking, security settings, and rules are ready. 11. Start all the vapps in vcloud Director using vcloud PowerCLI scripts. This ensures the proper start order within the vapp. In this solution, we started the ASCS instance first, the database instance second, and the application instances last. 12. Enable replication from Data Center A to Data Center C again. 24

25 Setting up disaster recovery for CEI Virtual Data Center Introduction Virtual Data Center is the key enabling technology introduced in the EMC Cloud- Enabled Infrastructure for SAP Foundation Bundle VNX White Paper. This solution reuses the existing architecture in the foundation bundle and configures the DR capability as an add-on. This section describes how to configure the following components: VMware vcloud Director Networking VMware vcenter Chargeback Configuring VMware vcloud Director This section details the vcloud Director settings needed to fully integrate the DR bundle with the CEI for SAP. Figure 8 illustrates the cloud architecture with the DR add-on. Figure 8. Cloud architecture with the DR bundle 25

26 You must configure the following components to facilitate the DR capability: Storage Provider vdc Organization vdc Configuring storage We created new datastores together with storage DRS clusters and storage profiles to enable DR. Table 8 and Table 9 list the storage settings needed in vcloud Director, configured in both data centers. Table 8. Storage configuration in vcloud Director for Data Center A Consistency group Management group Storage DRS cluster Storage profile Datastore N/A N/A DS_MGMT_DR_Protected (2 TB) Yes DR MGMT SRM Infrastructure N/A DS_MGMT_DR_SRM (2 TB) No Replicated N/A N/A DS_MGMT_Placeholder (10 GB) No Resource group DR Gold Tier DR Gold DS_DR_Gold_001 (4 TB) DS_DR_Gold_002 (4 TB) Yes DR Silver Tier DR Silver DS_DR_Silver_001 (4 TB) Yes Table 9. Storage configuration in vcloud Director for Data Center C Consistency group Management group Storage DRS cluster DR MGMT SRM Infrastructure Storage profile Datastore N/A DS_MGMT_DR_SRM (2 TB) No Replicated N/A N/A DS_MGMT_Placeholder (10 GB) No For the management group, we created a 2 TB datastore DS_MGMT_DR_Protected in Data Center A to serve as the RecoverPoint replication source for the management virtual machines. We then placed the following virtual machines on this datastore: Active Directory (AD) server Microsoft SQL Server for vcenter, vcloud Director, and vcenter Chargeback vcenter server for the resource group vcloud Director vshield Manager vcenter Chargeback 26

27 For the resource group, we created two storage DRS clusters in Data Center A to group datastores with different performance levels. RecoverPoint replicates the following datastores: DR Gold Tier A storage DRS cluster for placing datastores set to the highest performance DR Silver Tier A storage DRS cluster for placing datastores set to high performance We assigned two 4 TB datastores to the DR Gold Tier and one 4 TB datastore to the DR Silver Tier, and then created two new storage profiles to characterize the datastores with DR capabilities: DR Gold DR Silver We then assigned storage profiles to the corresponding datastores in the storage DRS clusters and configured datastores in the DR Gold Tier with storage profile DR Gold and datastores in the Silver Tier with storage profile DR Silver. As listed in Table 8 and Table 9, two datastores are marked as not replicated by RecoverPoint in both data centers as follows: VMware SRM infrastructure DNS, VMware vcenter server for management group, VMware SRM server, and their Microsoft SQL Server database server are deployed on this datastore. Since an independent VMware SRM server is already deployed in Data Center C, if a disaster occurs in Data Center A, we can trigger a failover from the VMware SRM server directly in Data Center C instead. Therefore, the containing datastore does not need to be replicated anymore. Placeholder Required for VMware SRM setup. Placeholder virtual machines consist of a few configuration files deployed on these datastores, including a VMX file. However virtual machine disk (VMDK) files are not placed on these datastores. Therefore, these datastores can be configured with a smaller size. Configuring the provider vdc In the EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle VNX White Paper, we defined two provider vdcs (PvDCs): Advanced and Standard. In this solution, we configured only the Advanced PvDC with the DR capability. We based this decision on the design consideration that the Standard PvDC is intended for development, test, sandbox, and POC systems. These may not require a fast and automated DR solution. Instead, we can recover these systems in Data Center C with the backup/restore solution. To add the DR capability to the Advanced PvDC, add the storage profiles DR Gold and DR Silver to the PvDC storage profile tab. Configuring the organization vdc After adding the DR storage profiles to PvDC, the DR storage profiles are also available in each existing organization vdc (OvDC) built on the Advanced PvDC. However, EMC recommends creating new OvDCs dedicated for DR storage profiles to group together the vapps with DR capabilities. This simplifies the management efforts during a failover or failback operation. 27

28 In this solution, we created new OvDCs in two organizations: OrgA_Dedicated_DR in OrgA and OrgB_Committed_DR in OrgB. Configuring the network The IP network environment varies in every cloud provider s data center. This solution introduces a network design that requires minimum change efforts during the failover and failback processes. Figure 9 illustrates the network design. Figure 9. Network design For the management group, VLAN 10 in Data Center A and VLAN 100 in Data Center C are designed to handle network activities of management virtual machines needed for the VMware SRM server. VLAN 11 is designed to handle the activities of management virtual machines needed to be protected. In case of a disaster, the protected virtual machines will be restarted in Data Center C on the same VLAN tag (VLAN 11). All the IP addresses of protected virtual machines remain unchanged so that no DNS records updates are required. For the resource group, VLAN 20 is designed to be an external network for tenants front-end access to SAP systems. Within vcloud Director, an organization network is created for each organization. A vshield Edge gateway is also created for each organization network to act as router and firewall. 28

29 We recommend establishing a secure WAN connection between two data centers. In this solution we used a site-to-site VPN to secure the connection. This solution has three types of cross-data center communications: Communication between the vcenter servers for the resource group in Data Center A and ESXi servers in Data Center C. Communication between the VMware SRM servers in both data centers. Communication between RPA clusters in both data centers. RecoverPoint replicates data over WAN. During the failover and failback processes, management virtual machines connected to VLAN 11 and SAP systems connected to VLAN 20 need to be transferred to another data center. Thus, the route table must be updated prior to a failover or failback action. This is a mandatory manual step and prerequisite on routes or any other network devices for a successful failover or failback. Table 10 summarizes the networks required in this solution. Table 10. Network configurations Network Purpose IP address VLAN 10 (Data Center A only) VLAN 11 For DNS, VMware SRM and vcenter for management with their SQL Server For management virtual machines to be protected / /24 VLAN 20 External network in vcloud /24 VLAN 100 (Data Center C only) For DNS, VMware SRM and vcenter for management with their SQL Server /24 OrgNet_OrgA Organization network for OrgA in vcloud /24 OrgNet_OrgB Organization network for OrgB in vcloud /24 Configuring infrastructure Chargeback Manager vcenter Chargeback Manager provides the virtualized infrastructure metering functionality. It integrates with Cloud Director seamlessly, providing the service provider with the ability to chargeback the resource consumed by tenants, and generates cost and utilization reports periodically or on demand. 29

30 The EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle VNX White Paper defines several pricing models to differentiate OvDC with different allocation models and storage profiles with different storage performance levels. Several approaches are available to reflect the chargeback of DR capability. You can create and bind a pricing model to a new OvDC with DR capability, with reasonable charges for infrastructure including CPU, memory, and storage. In this way, you can accurately charge the infrastructure and account for the resource reserved in Data Center C. Another simple approach is to set a rate factor to the storage profiles DR Gold and DR Silver to reflect a charge for the DR capability. You only need to set it once and all the vapps deployed on the storage profiles DR Gold and DR Silver will be charged accordingly with base rate times rate factor. That simplifies the management efforts needed to create pricing models for each OvDC compared to the previously described approach. Figure 10 shows how to set the rate factor in vcenter Chargeback Manager. Figure 10. Setting the factor rate for DR storage profiles EMC storage infrastructure Introduction The EMC VNX flash-optimized unified storage platform delivers innovation and enterprise capabilities for file, block, and object storage in a single, scalable, and easy-to-use solution. VNX includes many features and enhancements designed and built on the success of the first generation. From the performance and scalability perspective, these features and enhancements include: More capacity with optimized multicore optimization (MCx) with Multicore Cache, Multicore RAID, and Multicore FAST Cache Greater efficiency with a flash-optimized hybrid array The VNX array-based splitter seamlessly integrates with EMC RecoverPoint, providing reliable replication. 30

31 Configuring VNX storage For this solution, we configured all storage into VNX Fully Automated Storage Tiering for Virtual Pools (FAST VP) storage pools. We bound thin LUNs to the storage pools and present thin LUNs to the ESXi servers using masking views. Standard thin LUN sizes of 2 TB and 4 TB are presented from the VNX storage array to the ESXi servers. Both VNX systems at Data Centers A and C have similar configurations. Table 11 details the storage layout for the solution. Table 11. VNX storage layout Items Data Center A Data Center C Management group Resource group 2 x 2 TB thin LUNs: 1 thin LUN for management virtual machines that are not replicated to Data Center C, including VMware vcenter, VMware SRM, and its Microsoft SQL Server database server 1 thin LUN for management virtual machines that are replicated to Data Center C 3 x 4 thin LUNs: 2 thin LUNs associated with FAST VP policy Gold 1 thin LUN associated with FAST VP policy Silver 2 x 2 TB thin LUNs: 1 thin LUN for management virtual machines including VMware vcenter, VMware SRM, and its Microsoft SQL Server database server 1 thin LUN as the remote copy 3 x 4 TB thin LUNs: 2 thin LUNs associated with FAST VP policy Gold as the remote copy 1 thin LUN associated with FAST VP policy Silver as the remote copy Journal volume 3 x 800 GB thin LUNs 3 x 800 GB thin LUNs Repository volume Placeholder volume 1 x 8 GB thin LUN 1 x 8 GB thin LUN 1 x 10 GB thin LUN 1 x 10 GB thin LUN EMC RecoverPoint Introduction EMC RecoverPoint is an advanced enterprise-class DR solution designed with the performance, reliability, and flexibility required for enterprise applications. It provides integrated continuous data protection and continuous remote replication to recover applications to any point-in-time (PiT). RecoverPoint systems enable reliable replication locally with one RPA cluster to up to four remote RPA clusters, and over any distance. RecoverPoint systems support data replication for applications writing to SAN-attached storage using existing Fibre Channel (FC) and Internet SCSI (iscsi) infrastructures to integrate seamlessly with existing host applications and data storage subsystems. For remote replication, the systems use existing connections to send the replicated data over FC and WAN, and use FC and IP infrastructures to replicate data asynchronously or synchronously. The 31

32 systems provide failover of operations to another cluster in the event of a disaster at the production cluster. RecoverPoint variants The RecoverPoint family consists of three variants: RecoverPoint/CL for EMC and non-emc storage platforms RecoverPoint/EX for EMC storage platforms (Block), licensed by storage array RecoverPoint/SE for VNX series, EMC CLARiiON CX3 and CX4 series, and EMC Celerra unified storage environments In this solution, we used array-based RecoverPoint/SE for VNX. RecoverPoint 4.0 features RecoverPoint 4.0 offers the following new features and enhancements to significantly increase capacity and improve performance, helping to build a more flexible and reliable DR solution: Support for multisite replication RecoverPoint can replicate a primary site to up to four remote sites (1:4) for increased data availability. It can also protect data across branch offices with multisite replication from the branches to a central site (4:1). Any PiT recovery to VMware SRM operations Asynchronous throughput up to 400 MB/s using distributed consistency groups Replicated capacity from up to 2 PB per RecoverPoint cluster Support for synchronous replication over IP (versus FC-only in previous versions) A new, easy-to-use Unisphere interface for RecoverPoint management RecoverPoint Appliance The RPA is the RecoverPoint system's intelligent data protection appliance. RPAs manage all aspects of reliable data replication including physical hardware or virtual machines. RPAs are used to attain availability. If an RPA fails, another RPA assumes the workload. A group of two to eight physical (or virtual) RPAs at the same geographic location working together to replicate and protect data is called an RPA cluster. An RPA cluster supports immediate switchover to another appliance if one of the RPA nodes in the cluster fails. There can be up to five RPA clusters in a RecoverPoint system. To scale up and support a higher throughput rate, you can add up to eight RPAs to an RPA cluster. RPAs use powerful bandwidth reduction technologies, including deduplication and compression, to minimize the use of bandwidth and dramatically reduce the time lag between the source and target sites when writing data to storage. RecoverPoint splitter The RecoverPoint splitter is proprietary software that you can install on storage subsystems, and is built into the VNX, Symmetrix VMAX, and VPLEX storage systems. 32

33 The RecoverPoint splitter is used to split the application writes so that they are sent first to the RPA and then to their normally designated storage volumes. Splitters perform this activity efficiently, with little impact on host performance because the RPA performs all CPU-intensive processing necessary for replication. This feature can also be upgraded without disrupting storage access. Unisphere for RecoverPoint The RecoverPoint 4.0 user interface was developed using Adobe Flex technology and promotes ease of use and a common user experience between the RecoverPoint, VNX, VPLEX, and Symmetrix products. RecoverPoint includes a dashboard that provides an overview of the system s health, and all of the information displayed through Unisphere for RecoverPoint is consistently auto-refreshed. RecoverPoint replication options RecoverPoint supports three modes of replication: Continuous Local Replication (CLR) RecoverPoint continuously captures and stores data modifications locally, enabling local recovery from any PiT, with no data loss. CLR supports both synchronous and asynchronous replications. Continuous Remote Replication (CRR) RecoverPoint captures local data modifications and replicates data to remote sites through IP or FC connections. CRR supports both synchronous and asynchronous replications. Continuous Local and Remote Replication (CLRR) A combination of CLR and CRR that provides both local and remote data replication. CLR is normally used for PiT recovery when business data or application logic has been severely compromised. CRR is normally used for DR. CLRR thus implies that two replicas are made for every protected volume (local and remote) to protect the business from both scenarios. This solution demonstrates a CRR with asynchronous replication. Configuring RecoverPoint consistency groups RecoverPoint protects volumes using consistency groups. If two data sets are dependent on one another (such as a database and a database log), they should be part of the same consistency group. When creating a consistency group, define the following elements: Production copy The production copy consists of all source volumes for a consistency group replication, as well as the production journal volumes. Local copy A local copy consists of all target volumes for the replication in local storage for a specific consistency group and the copy journal volumes. Remote copy A remote copy consists of all target volumes of the replication in the remote storage for a specific consistency group and the copy journal volumes. 33

34 Defining consistency groups A consistency group can contain a local copy, remote copy, or both, which facilitates CLR, CRR, and CLRR respectively. Besides containing a mandatory production copy, the consistency group can also contain a local copy, a remote copy, or both copies, depending on the replicating mode it facilitates. In this solution, we created two consistency groups with remote copy: The management group, which consists of all the volumes for managing the virtual machines The resource group, which consists of all the volumes for SAP systems from different tenants Table 12 details the consistency groups in this solution. Table 12. RecoverPoint consistency groups Consistency group Data Center A Data Center C Management group Resource group Production copy volumes: DR_Management_1 (2 TB) Journal volumes: Journal_1 (400 GB) Production copy volumes: DR_Resource_1 DR_Resource_2 DR_Resource_3 (4 TB each) Journal volumes: Journal_2 Journal_3 Journal_4 (800 GB each) Remote copy volumes: DR_Management_1 (2 TB) Journal volumes: Journal_1 (400 GB) Remote copy volumes: DR_Resource_1 DR_Resource_2 DR_Resource_3 (4 TB each) Journal volumes: Journal_2 Journal_3 Journal_4 (800 GB each) Configuring consistency groups For detailed configuration steps for defining consistency groups, see Configuring consistency group in Appendix. After the steps to configure consistency groups are completed, RecoverPoint initiates the replication. The first initialization run may take a long time if the volumes to be replicated are very large. For the management group, VMware SRM takes over the operation of RecoverPoint by using RecoverPoint Storage Replication Adaptor (SRA). Therefore, you need to configure the consistency group for the management group to be externally managed by VMware SRM. Figure 11 highlights the additional configuration required. 34

35 Figure 11. Configuring a consistency group to be managed by VMware SRM VMware vcenter Site Recovery Manager Introduction VMware SRM is the market-leading DR management product. It provides the simplest and most reliable disaster protection for all virtualized applications. VMware SRM leverages a broad set of high-performance storage-replication products to replicate virtual machines to a secondary site. VMware SRM provides a simple interface for setting up recovery plans that are coordinated across all infrastructure layers, replacing traditional error-prone human processes. You can test recovery plans nondisruptively as frequently as required to ensure that they meet your business objectives. At the time of a site failover or migration, VMware SRM automates both failover and failback processes, ensuring fast and highly predictable RPOs and RTOs. Integrating SRM with RecoverPoint VMware SRM integrates with RecoverPoint using SRA. RecoverPoint SRA supports the discovery of arrays attached to RecoverPoint and of consistency groups for the management group. It also supports VMware SRM functions such as recovery and reprotect by mapping VMware SRM recovery plans to the appropriate RecoverPoint actions. Configuring VMware SRM Most of the configuration of VMware SRM in this solution is set up for the management group. For detailed configuration steps, see Configuring VMware SRM in the Appendix. As mentioned in Disaster recovery logical architecture, we used VMware SRM to orchestrate both the management group and the resource group. We added steps in the recovery plan that invoke our customized VMware PowerCLI and RPA CLI scripts to automate the recovery process of the resource group. Figure 12 shows the configuration needed. 35

36 Figure 12. Automating recovery of resource group We added steps in different modes recovery and planned migration to perform relevant operations in RPA and vcloud Director. 36

37 SAP system architecture Introduction This section describes the SAP system architecture deployed for the solution in the two data centers. The SAP application layer uses the following SAP components: SAP ERP 6.0 Core IDES Enhancement Package 5 SAP NetWeaver 7.02 SAP ASCS SAP Database Service (database) SAP Primary Application Server (PAS) SAP Additional Application Servers (AAS) All SAP instances were deployed on vcloud environment with the Windows Server 2008 R2 SP1 operating system. Note: Other operating systems and database combinations supported by the SAP Product Availability Matrix (PAM) and the EMC Solutions Support Matrix (SSM) can also be used with this solution. SAP ERP 6.0 SAP ERP 6.0, powered by the SAP NetWeaver technology platform, is a world-class, fully integrated enterprise resource planning (ERP) application that fulfills the core business needs of midsize companies and large enterprises across all industries and market sectors. SAP ERP 6.0 delivers a comprehensive set of integrated, crossfunctional business processes and can serve as a solid business process platform that supports continued growth, innovation, and operational excellence. SAP Internet Demonstration and Evaluation System (IDES) supports demos, testing, and functional evaluation based on preconfigured data and clients. IDES contains application data for various business scenarios, with business processes that are designed to reflect real-life business requirements and have access to many realistic characteristics. This solution uses IDES to represent a model company for testing purposes. 37

38 SAP system architecture This solution implements a distributed SAP system architecture, as shown in Figure 13. Figure 13. SAP system architecture We decoupled the enqueue server and message server from the central instance and implemented them as services within the ASCS instance. We installed two AASs to provide redundant work processes such as dialog (DIA), background (BGD), update (UPD), spool (SPO), and gateway. Key SAP design considerations The SAP system deployed for the solution implements these key design features: The ASCS instance is installed with a virtual hostname (SAPVIPE) to decouple it from the virtual machine hostname. We installed and configured the SAP patches, parameters, basis settings, and load balancing settings according to the SAP Installation Guide and the SAP Notes listed in the References section. We used VMware best practices for SAP in this solution. SAP update processes (UPD/UP2) are configured on the AAS instances. SAP shared file systems, including //SAPVIPE/sapmnt/SID (available to all SAP instances), are stored on the ASCS server and mounted as share folders on the SAP virtual machines. 38

39 Design best practices Data center The hardware resources in Data Center C should be sufficient to hold virtual machines that failover from Data Center A. Do not allocate CPU and memory resources used for DR in Data Center A more than what Data Center C can provide; otherwise, failover may not be successful due to resource shortage, which may result in failure to provide service-level compliance. RecoverPoint We recommend the following design best practices for configuring RecoverPoint: Ensure that the number of RPAs in an RPA cluster is the same with all RPA clusters in a RecoverPoint system. You can manage all RPAs in an RPA cluster through a floating cluster management IP address. In other words, after you configure RecoverPoint, you can manage the entire installation from a single IP address. Enable compression on RPA Gen5 because there is almost no performance penalty when using compression. Enable deduplication for remote replications in asynchronous mode to save bandwidth on RPA Gen4 or Gen5. Use a distributed consistency group in a large-scale environment if the maximum throughput rate of a single RPA does not sufficiently sustain the write-rate or peaks of the consistency group, or if you expect that a future consistency group will require a higher throughput rate than that of a single RPA. Running a distributed consistency group on an RPA cluster with three RPAs provides only a small improvement in performance. However, a huge improvement in performance is achieved when a distributed consistency group is run on an RPA cluster with four RPAs. Ensure that the size of the journal volumes reflect the RPO. You need to determine the expected peak change in the environment to calculate the minimum size of the journal volume. The formula is: ( ) ( ) ( ) However, in a multitenancy cloud environment, estimating parameters in the formula is difficult. We recommend that you reserve at least 20 percent of the production volume capacity for the journal volume. VNX storage We recommend the following design best practices for configuring VNX storage: The majority I/O pattern of the journal volumes is write I/O. Therefore, I/O may not benefit from FAST VP technology. We recommend placing journal volumes on thin LUNs of a SAS storage pool. Avoid using FAST Cache on journal LUNs or pools. FAST Cache is suitable for workloads of high locality, small blocks, and random I/O. The sequential large block I/Os of journals benefit very little from FAST Cache. 39

40 Cloud SAP EMC recommends the following design best practices for configuring a vcloud: Do not include non-dr storage profiles in an OvDC that contains DR storage profiles. Always create new OvDCs to include storage profiles with DR capability only, or migrate all vapps to the DR storage profile in an existing OvDC and remove the non-dr storage profile. This simplifies management and script development to find vapps that are deployed on DR storage profiles. Set the Stop Action to guest-shutdown in the vapp properties so that whenever a stop action is executed (whether in the web GUI or by a script) vcloud Director shuts down all the virtual machines in the vapp instead of powering them off. Set a proper start and stop order in the vapp properties to ensure that the SAP system can be started and stopped correctly. We recommend setting the following order for a SAP system with distributed installation: First: ASCS Second: database Third: PAS and AAS instances EMC recommends creating OS scripts for starting and stopping SAP application and database instances along with the starting and stopping of the OS. This ensures minimum downtime and reduces the manual efforts needed to start or stop SAP and databases during failover and planned migration. 40

41 Test and validation Overview We initially installed and validated the environment without any preexisting DR or business-continuity protection schemes. We then transformed the environment to a mission-critical business continuity solution described in this white paper. We simulated a disaster that caused a full-production data center to become unavailable, and validated the outcome. The use cases were designed with the following key business considerations: Operations: 24 hours a day, 7 days a week RTO: 1 hour RPO: 30 minutes RTO and RPO in this paper are based on the current lab environment. The RTO in a customer s environment depends on the number of ESXi hosts, datastores, and virtual machines on DR datastores. You can calculate your RTO based on the test result. RTO and RPO are configured in the RecoverPoint Link settings. The link bandwidth is limited to 100 Mb/s to better simulate the real situation. Table 13 lists the SAP systems deployed in our environment. Table 13. SAP systems deployed in our environment Organization OvDC vapp name Virtual machines OrgA OrgA_Dedicated_DR SAP ERP 6.05 PRD ASCS: saperpascs Database: saperpdb PAS: saperppas AAS: saperpaas1 SAP ERP 6.05 DEV Centralized: saperpvm6 OrgB OrgB_Committed_DR SAP HR 6.05 PRD ASCS: saphrascs Database: saphrdb PAS: saphrpas AAS: saphraas1 SAP HR 6.05 QAS Centralized: saphrvm6 T-code SGEN is executed on SAP ERP DEV and SAP HR QAS to generate write inputs/outputs (I/Os). We ran the following tests to validate the solution and demonstrate the full cycle of DR operations: Test scenario 1: Failover after disaster Test scenario 2: Resynchronizing to Data Center A Test scenario 3: Planned migration Test scenario 4: Reprotection for Data Center A 41

42 Notes Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated. All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute. Test Scenario 1: Failover after disaster Virtual machines in the management group and SAP systems in the resource group are running in Data Center A. A disaster occurred and resulted in a site outage at Data Center A. This outage could not be recovered in Data Center A within the RTO. Therefore, a failover to Data Center C was performed to bring services back online. Test objectives The test objectives include: Failover of virtual machines in the management group must be automated by the solution in the correct sequence. Failover of the SAP systems must be triggered after a successful failover of the management group. Virtual machines such as database, ASCS, PAS, and AAS within a vapp must be started automatically after failover, following the startup order defined in the vapp. All of the SAP systems must be restored and back in service within the RTO and RPO. No errors should appear in the SAP systems. Test procedure We used the following test procedure: 1. Run the workload on the SAP system in Data Center A. Run SD benchmarks to fill VBAK with new data at a constant rate (approximately 15 seconds per entry). 2. Take a screenshot of the latest entries in the VBAK table. 3. Power off all ESXi servers in the management group and the resource group. 4. Disable the network ports of the RPAs in Data Center A. 5. Log in to a vsphere client in Data Center C. Navigate to the VMware SRM plugin and execute the recovery operation. Note the failover procedure and timestamp. 42

43 6. Log in to the SAP system SAP ERP 6.05 PRD in Org A vcenter server for the resource group. Check the VBAK table to determine data loss and note the data loss time period. 7. Log in to all SAP systems using SAP GUI and check the transactions for errors including SICK, SM51, SM50, SM21, ST22, and DB02. Results and analysis Table 14 outlines the failover test results. Table 14. Failover test results Test item Success criteria Results Total failover duration (minutes) Data loss in resource (minutes) Failover sequence Startup sequence Errors in the SAP systems Final evaluation < < 30 3 VMware SRM automates failover of the virtual machines in the management group in the correct sequence. The failover script for the resource group must be invoked by VMware SRM as part of the failover process. The SAP virtual machines such as database, ASCS, PAS, and AAS within a vapp that are started by scripts must be started automatically after failover in the startup order defined in the vapp. Successful Successful Successful Database and SAP startup scripts Successful are executed along with the OS boot. 0 0 PASSED As shown in Table 14, the total recovery time is 34 minutes and the total data loss happens within three minutes, with limited bandwidth of 100 Mb/s, which successfully meets the RTO and RPO. There are no errors in the SAP system, which means the data is consistent. Figure 14 shows the data before disaster and the data after failover in table VBAK of the SAP system. 43

44 Figure 14. RPO in the SAP system Test Scenario 2: Resynchronizing to Data Center A After Data Center A is rebuilt, the customer plans to resynchronize the virtual machines in the management group and resource group that are running in Data Center C. The replication direction is reversed. Data Center C becomes the production site and Data Center A becomes the recovery site. Test objectives The test objectives include: Resynchronization of virtual machines in the management group must be automated. The replication direction of the virtual machines in the resource group must be reversed. No service interruption of SAP systems should occur. Test procedure We used the following test procedure: 1. Check connectivity of RPA and VMware SRM between Data Center A and Data Center C. 2. Run the workload on the SAP system in Data Center C. Run the SD benchmark to fill VBAK with the new data at a constant rate (approximately 15 seconds per entry). 3. Log in to the vsphere client in Data Center C. Navigate to the VMware SRM plug-in and execute the reprotect operation. If errors occurred, fix the errors and rerun reprotect. 44

45 4. Log in to the VMware SRM server OS and execute the RecoverPoint CLI script to reverse the replication for the resource group During the resynchronization process, continuously check to see if the SAP GUI is terminated. Results and analysis Table 15 outlines the resynchronization test results. Table 15. Resynchronization test results for Data Center C Test item Success criteria Results Duration of resynchronization for management (minutes) Duration of resynchronization for resource (minutes) Direction of replication must be reversed completely. Direction of replication must be reversed completely. Successful (33 minutes) Successful (248 minutes) Service interruption in SAP system Final evaluation 0 0 PASSED The total duration of resynchronization depends on the total capacity of the volumes in the consistency groups and WAN bandwidth between two data centers. RecoverPoint initiated a full sweep because the link between RPA clusters in Data Center A and Data Center C was broken for a long time during the outage of Data Center A. The services of SAP systems are not interrupted during the resynchronization process. Test Scenario 3: Planned migration After Data Center C is resynchronized, the customer plans to migrate virtual machines running in the management group and SAP systems running in the resource group from Data Center C back to Data Center A. A planned downtime is required. Test objectives The test objectives include: Migration of virtual machines in the management group must be automated by VMware SRM in a predefined order. Migration of SAP systems in the resource group must be triggered automatically after successful migration of the management group. All vshield Edge gateways must be gracefully shut down by scripts. Virtual machines such as database, ASCS, PAS, and AAS within a vapp must be gracefully shut down by scripts. The stop order defined in vapp must be followed. 3 Reprotection of virtual machines in the resource group is a manual process in RPA and can be automated by an RPA script. However, the VMware SRM server does not support adding a step in the reprotect process. A manual execution of the RPA scripts is required. 45

46 All the SAP systems should be migrated and brought back to service within the specified RTO and RPO. No errors should be shown in SAP systems. Test procedure We used the following test procedure: 1. Check the connectivity of RPA and SRM between Data Center A and Data Center C. 2. Stop the SD benchmark and take a screenshot of the latest entries in the VBAK table. 3. Log in to the vsphere client in Data Center A. Navigate to the VMware SRM plug-in and execute the planned migration operation. Note the migration procedure and timestamp. 4. Log in to the SAP system using the SAP GUI and check the transactions for any errors including SICK, SM51, SM50, SM21, ST22, and DB Take a screenshot of the VBAK table and compare it with the screenshot taken before the disaster. Note the period of data loss. Results and analysis Table 16 details the planned migration test results. Table 16. Planned migration test results Test item Success criteria Results Total failover duration (minutes) < Data loss in resource (minutes) < 30 0 Automation The script must be invoked by VMware SRM as part of the entire failover process for the resource group. Successful Errors in SAP systems 0 0 Final evaluation Passed As shown in Table 16, the total migration time is 40 minutes and the total data loss is zero, which successfully meets the RTO and RPO. There are no errors in the SAP system, which means the data is consistent. Test Scenario 4: Reprotection for Data Center A The virtual machines migrated back to Data Center A must now be reprotected using VMware SRM and RPA. Test objectives The test objectives and procedure are the same as those in Test Scenario 2. 46

47 Results and analysis Table 17 details the reprotection test results for Data Center A. Table 17. Reprotection test results for Data Center A Test item Success criteria Results Duration of reprotect for management (minutes) Direction of replication must be reversed completely. Successful (3 minutes) Duration of Reprotect for resource (minutes) Direction of replication must be reversed completely. Successful (2 minutes) Service interruption in SAP system 0 0 Final evaluation Passed As listed in Table 17, the reprotection duration for both the management group and the resource group are shorter than that of Scenario 2. The full sweep is not performed because the replication between the RPAs at both sites is active before reprotection. We recommend triggering reprotection right after a successful planned migration. 47

48 Conclusion Summary This solution offers effective, affordable, and efficient DR protection for CEIs running critical business functions. This strategy significantly improves business agility and scales out for SAP production systems with varying needs. This solution demonstrates the DR bundle that is added on top of the foundation bundle solution by combining products and components from EMC and VMware to: Build active/standby data centers for SAP environments by using the integrated RecoverPoint splitters for VNX arrays across the two data centers. Integrate RecoverPoint and VMware SRM to automate the entire DR process. Develop customized VMware PowerCLI and RecoverPoint CLI script to minimize human intervention during DR for SAP systems managed by vcloud Director. The EMC CEI DR bundle provides: Business continuity solution for mission-critical SAP applications in a cloud environment. Cloud tenants have the option to easily deploy their most critical SAP systems on the DR infrastructures described in this solution. The expense need for DR will be reflected correctly in the chargeback report. The failover and failback processes require no additional intervention from the tenants side. Rapid and automated DR in a cloud environment powered by VMware PowerCLI and RecoverPoint CLI. The DR administrator can easily complete the entire DR process with a few clicks. This is critical to ensure RTO in a large-scale cloud environment. Findings In addition to the conclusions drawn from the test results, we identified the following points of interest during our analysis: Test Scenario 1 (failover after disaster) The failover process can be executed on Data Center C without first bringing up components like vcenter or VMware SRM in Data Center A. The start script defined in the OS level to start DB and SAP further reduces human intervention, thus ensuring the compliance of RTO. Test Scenario 2 (resynchronizing to Data Center A) The services in both the management group and the resource group are not interrupted during the reprotection process. Test Scenario 3 (planned migration) The operation cannot be successful unless all errors in the process are eliminated, ensuring the data integrity and zero RPO. Test Scenario 4 (reprotection for Data Center A) The duration of reprotection is shorter than in Test Scenario 2 because the replication link was not broken and the reprotection was triggered after a successful planned migration. 48

49 References EMC documents VMware documents SAP documents For additional information, see the following EMC documents (available on EMC.com and EMC Online Support): EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle VNX White Paper EMC Cloud-Enabled Infrastructure for SAP VNX Reference Architecture EMC RecoverPoint Installation and Deployment Guide EMC RecoverPoint 4.0 Administrator s Guide EMC RecoverPoint Adapter for VMware vcenter Site Recovery Manager Release Notes (version 2.x) EMC RecoverPoint 4.0 CLI Reference Guide EMC RecoverPoint Deploying with VNX/CLARiiON Arrays and Splitter Technical Notes EMC Transforms IT for SAP: Fully Automated Disaster Recovery White Paper For additional information, see the following VMware documents: VMware vsphere 5.1 Command-Line Documentation VMware vcenter Site Recovery Manager 5.1 Installation and Configuration VMware vcenter Site Recovery Manager 5.1 Administration VMware vcloud Director Infrastructure Resiliency Case Study SAP Solutions on VMware: Best Practices Guide For additional information, see the following SAP documents: SAP Note Windows: VMware vsphere Configuration Guidelines Installation Guide: SAP Enhancement Package 5 for SAP ERP 6.0 Based on SAP NetWeaver 7.0 including Enhancement Package 2 49

50 Appendix Configuring consistency group For the management group, we created a consistency group named Management_Group, which required the following steps: 1. Define the consistency group and select the production volumes. Figure 15 shows the parameter required for continuous configuration. Figure 15. Defining consistency group and select production volumes 2. Define a production journal as shown in Figure 16. Figure 16. Defining production journal 3. Add a remote copy. We selected a volume from Data Center C as a remote copy. The RPO parameter is defined in this step and can be changed later in the consistency group settings. We selected asynchronous replication mode because Data Center C is located in a remote site connected by a WAN. Figure 17 shows the parameters required for adding a remote copy. 50

51 Figure 17. Adding a remote copy 4. Define a copy journal as shown in Figure 18. Figure 18. Defining copy journal The same configuration steps also apply to creating the consistency group for the resource group. Configuring VMware SRM The following configuration requirements are discussed in this section: Site pairing Array-based replication Resource mapping 51

52 Protection groups Recovery plans Site pairing Site pairing establishes a connection for the VMware SRM server between the production site (Data Center A) and recovery site (Data Center C). Figure 19 shows the configuration steps. Figure 19. Site pairing We recommend installing the VMware SRM server certificate in Data Center A to the remote vcenter server in Data Center C so that the communication between two data centers is secured. 52

53 Array-based replication Configure the array-based replication after site pairing. We used SRA to establish a connection between VMware SRM and RecoverPoint. Figure 20 shows the configuration required in Data Center A. Figure 20. Configuration of array-based replication Follow the same steps to add Array Manager in Data Center C. After configuring Array Manager in both data centers, we enable array pairs in both VMware SRM servers. As a result, VMware SRM can automatically retrieve the consistency group information in RecoverPoint. Resource mapping Resource mapping is critical to ensure that the virtual machine settings, especially the network configuration, are correct and usable after failover and resynchronization. Three types of mappings must be configured: Resource pool, network, and folder. During failover and resynchronization, VMware SRM automatically changes the corresponding settings to the mapped resource. Table 18 lists the mappings between our two data centers. 53

54 Table 18. Resource mapping Mapping Data Center A Data Center C Resource pool MGMT Protected MGMT Recovery Network dvpg_vlan11 dvpg_vlan11 Folder MGMT Protected MGMT Recovery Protection groups A protection group is a collection of virtual machines and templates that you protect together using VMware SRM. In this solution, we use array-based replication; therefore, a protection group consists of one or several consistency groups defined in RecoverPoint. VMware computes the set of virtual machines in the consistency groups. Figure 21 shows the configuration steps for creating a protection group. Figure 21. Configuring a protection group All the management virtual machines that need to be protected by VMware SRM and RecoverPoint are automatically computed and included in the protection group named Management_Group. 54

55 Recovery plans A recovery plan specifies how VMware SRM recovers the virtual machines in the protection groups contained in the recovery plan. One or more protection groups can be included in each recovery plan. Figure 22 shows the configuration steps needed to create a recovery plan. Figure 22. Configuring a recovery plan After creating the recovery plan, we specify the restart priority and dependency of each management virtual machine. The priorities depend on the business requirements of each customer. In this solution, we defined the priorities as follows: First: AD Second: Microsoft SQL Server Third: vcenter Server, vshield Manager Fourth: vcloud Director, vcenter Chargeback 55

56 DR design for three data centers Figure 23 shows the DR design overview on top of the HA bundle, which involves two production data centers as a cluster enabled by VPLEX Metro. Figure 23. Configuring a recovery plan Customers who have already implemented the HA bundle can chose to deploy RPA clusters in all production data centers. VLPEX splitter is used instead of VNX splitter, enabling storages in the Metro cluster in production data center (Data Center A and Data Center B) to be replicated to Data Center C. If one of the production data centers has an outage, HA ensures management virtual machines and SAP systems are restarted in another production data center. The RPA cluster in another data center continues to protect the storage. The recovery operation is executed only when both production data centers are unavailable due to the disaster. 56