EMC CLOUD-ENABLED INFRASTRUCTURE FOR SAP - BUSINESS CONTINUITY SERIES: HIGH AVAILABILITY AND APPLICATION MOBILITY BUNDLE

Transcription

1 White Paper EMC CLOUD-ENABLED INFRASTRUCTURE FOR SAP - BUSINESS CONTINUITY SERIES: HIGH AVAILABILITY AND APPLICATION MOBILITY BUNDLE EMC VPLEX, EMC Symmetrix VMAX, VMware vcloud Suite, and VMware vsphere Metro Storage Cluster with vsphere HA and DRS Resilient mission-critical SAP deployment in a private cloud Application mobility across datacenters Active/active datacenters EMC Solutions Abstract This white paper focuses on high availability and application mobility add-on bundle of the on-premise Cloud-enabled Infrastructure for SAP. It explains the transformation of a single datacenter into a mission-critical business continuity solution with active/active datacenters. The solution is enabled by EMC VPLEX Metro, EMC Symmetrix VMAX, VMware vcloud Suite, and VMware vsphere Metro Storage Cluster with vsphere High Availability and vsphere Distributed Resource Scheduler. November 2013

2 Copyright 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. All test results contained in this report were obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a customer can or will achieve similar results. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H

3 Table of contents Executive summary... 5 Business case... 5 Solution overview... 5 Key results... 6 Introduction... 7 Purpose... 7 Objectives... 8 Audience... 8 Terminology... 8 Technology overview Introduction Solution architecture Physical architecture Logical architecture Protection layers Hardware resources Software resources Key components Overview EMC VPLEX EMC Symmetrix VMAX EMC PowerPath/VE EMC VSI VMware vcloud Suite Symantec ApplicationHA EMC VPLEX Metro infrastructure Introduction VPLEX Metro solution configuration VPLEX Witness configuration VMware virtual datacenter Introduction Configuring VMware vcloud Director Configuring vcenter Chargeback Manager VMware deployments on VPLEX Metro VMware stretched cluster configuration

4 VMware vsphere HA configuration Configuring VMware vsphere DRS EMC VSI and VPLEX Symantec ApplicationHA SAP system architecture Introduction Configuring SAP system Outage impact Key design considerations EMC storage infrastructure Introduction Symmetrix HA Configuring Symmetrix VMAX Workload generation SAP standard SD Benchmark Testing and validation Introduction Planned downtime Unplanned downtime Conclusion Summary Findings References EMC VMware SUSE SAP Symantec

5 Executive summary Business case Application and infrastructure availability is a key consideration in today s modern business continuity strategies for SAP cloud infrastructures stretched across distances. In traditional infrastructure designs for SAP high availability, workloads across physical datacenters are disrupted when any of the infrastructure components fails at the same time. These traditional high availability designs are limited and usually require special combinations of software and hardware that are operating system-specific, database specific, or both, making them complex, inflexible, costly, and difficult to implement and maintain. Often these designs require more effort to keep them running than actually providing the required protection and availability promised in the first place. This white paper introduces an EMC solution for mission-critical SAP high availability with the following objectives: High availability, which manages risks in an IT environment and plans for recovery across distances resulting from unplanned outages. Application mobility, which enables non-disruptive movement of running SAP workloads seamlessly across servers or datacenters. The main business challenges addressed by this solution include: Protect against single points of failure (SPOFs) Minimize the impact caused by planned and unplanned downtimes Reduce the infrastructure complexity and operational costs Provide an end-to-end automated resilience model Improve the utilization of resources across datacenters Solution overview This solution is an add-on to the existing EMC Cloud-enabled Infrastructure for SAP foundation bundle, and it adds the vsphere Metro Storage Cluster (vmsc) configuration commonly referred to as a stretched cluster. VMware vmsc is a VMware certified solution, which is a combination of storage array based clustering and synchronous replication provided by EMC VPLEX Metro. The main goal of this VMware vmsc solution is to extend the high availability capability provided by the local clusters within a local datacenter to a geographically separated model with two datacenters in different locations. The architecture extends what is defined as local in terms of network and storage. This enables the network and storage subsystems to span a metropolitan area, presenting a single and common base infrastructure set of resources to the VMware vsphere cluster at both datacenters. This solution provides a new level of availability with an end-to-end, out-of-the-box, integrated, simplified, and cost-effective solution that is also application-aware. 5

6 The white paper demonstrates how the following technologies create this innovative solution: EMC VPLEX Metro Provides the virtual storage layer that enables an active/active Metro datacenter and supports continuous availability, even in the event of disruption at one of the datacenters. VMware vcloud Director Applies the principles of pooling, abstraction, and automation to all datacenter services such as compute, storage, networking, and security. It enables the multi-tenancy business model for IT operation and provides the portal for self-service provisioning. EMC Symmetrix VMAX platform Provides powerful and trusted storage for multiple mission-critical applications, as well as automation and efficiency in tiered storage environments. VMware vsphere Transforms datacenters into simplified cloud computing infrastructures and enables IT organizations to deliver flexible and reliable IT services. Provides higher availability independent of hardware, operating system, and applications. Reduces planned downtime for common maintenance operations and prevents extended downtime with automatic, rapid recovery in case of failures. VMware vsphere Distributed Resource Scheduler (DRS) Aligns compute resources with business priorities and affinity rules by automatically balancing virtual machines across hosts. Symantec Application High Availability (ApplicationHA) Simplifies and centralizes application administration and management through integration with VMware vsphere and guest applications. Key results The solution builds the HA and application mobility add-on bundle by providing the following functions: Automatic restart of virtual machines and application-specific processes in the event of a datacenter, server, or process failures Simple and automated HA protection for complete SAP cloud datacenters provided by vsphereha Automated SAP application-aware protection integrated for protecting missioncritical applications Additional benefits include the following: Increased use of hardware and software assets: Improved resource utilization across datacenters Automatic load balancing between datacenters Zero downtime on planned infrastructure maintenance Reduced resources required with single clustered systems that eliminates the need for additional standby systems. Reduced management, maintenance costs and risks with full end-to-end application restart automation, minimizing human intervention 6

7 Introduction Purpose This white paper describes the HA and application mobility add-on bundle for the EMC Cloud-enabled Infrastructure for SAP foundation bundle. This solution is intended to provide enhanced application availability and application mobility for workloads across physical datacenter boundaries with EMC VPLEX, VMware vsphere HA, vsphere DRS, and Symantec ApplicationHA in a vmsc configuration. Business continuity for SAP Because SAP is a major part of the core business operations in many organizations, it is critical that these organizations have a business continuity management plan to: Safeguard the continuity of their business operations Protect revenue Recover to at least a minimum level of operation if an outage occurs. A business continuity management plan: Helps to manage risks in an IT environment Details contingency strategies of business processes following a disruption of operations Business continuity for IT comprises three components: High availability (HA) Data protection (DP) Disaster recovery (DR) One way to provide high availability is to minimize the effect of an unplanned outage by masking the outage from the end users. This requires an availability automation solution to manage application failover within the same server, or between servers, within or across datacenters to ensure that high availability measures are achieved. The IT Infrastructure Library (ITIL) is a public framework of best practices for IT service management. For more information, refer to the ITIL publication ITSC IT Service Continuity Management. The SAP Solution Management Business Continuity Best Practices document refers to ITIL and it was considered in this white paper. Types of failures Three main types of failures should be considered in IT business continuity management: Technical failure These failures range from crashes of individual hardware components to entire datacenters. They disrupt normal operations and require different solutions based on their severity to resume the business operations. Logical failure These failures can be caused by faulty or malicious software, or the incorrect use of software such as corrupt data that can disrupt business processes. Logistical failure These are failures of operational or logistical business operations such as unavailable staff or facilities. 7

8 This paper focuses on the high availability component to address technical failures and increase the availability of a cloud-enabled SAP landscape. Recovery There are two major types of recovery after an outage: System recovery Technical availability of failed systems must be reestablished. Technical availability through high availability is the main focus of this paper. Disaster recovery is another form of system recovery after a disaster event. The EMC Cloud-enabled Infrastructure for SAP disaster recovery add-on bundle addresses this kind of recovery type. Business recovery Logical or data inconsistencies must be corrected. (Logical errors may sometimes be the result of technical failures). The EMC Cloud-enabled Infrastructure for SAP data protection add-on bundle addresses the technical aspect of this recovery type. Simplifying SAP high availability This white paper describes a solution that addresses technical failures and provides high availability protection in a simplified and cost-effective manner compared to traditional HA solutions in the market. This solution was implemented in an EMC lab environment to validate the protection of an SAP system or landscape after minor and major technical failures. Objectives Audience Terminology The objectives of the white paper are to: Introduce the key enabling technologies Describe the solution architecture and design Describe how the key components are configured Present the results of the tests and validation performed Identify the key business benefits of the solution This white paper is intended for SAP Basis Administrators, storage administrators, IT architects, and technical managers responsible for designing, creating, and managing mission-critical SAP applications in 24/7 landscapes. Previous technical knowledge of cloud, virtualization, server, networking, and storage solutions are required to fully comprehend the concepts and benefits described in this paper. Table 1 defines the terms and abbreviations used in this white paper. Table 1. Term AAS ABAP ASCS Terminology Description SAP Additional Application Server (previously known as Dialog instance) SAP Advanced Business Application Programming ABAP SAP Central Services (runs the central enqueue and message server services) 8

9 Term CNA dvswitch EHP ERP HA HBA IDES ISL LACP LAG LLDP LUW MPP NL-SAS PAS RFC SAP DB SLES SPOF VCS vlag vlan VMDK VMHA VPLS VPN Description Converged network adapter vsphere distributed switch SAP Enhancement Package Enterprise resource planning, powered by the SAP NetWeaver technology platform, is a fully-integrated enterprise resource planning (ERP) application that fulfills the core business needs of midsize companies and large enterprises across all industries and market sectors. High availability Host bus adapters SAP Internet Demonstration and Evaluation System Inter-Switch Link Link Aggregation Control Protocol Link Aggregation Group Link Layer Discovery Protocol Logical unit of work Multipathing plug-in Nearline SAS (Serial Attached SCSI) SAP Primary Application Server (previously known as Central instance) Remote function call SAP Database Server that runs the Database instance SUSE Linux Enterprise Server Single point of failure Virtual cluster switch Virtual Link Aggregation Group Virtual LAN Virtual disk VMware vsphere High Availability Virtual Private LAN Service Virtual private network 9

10 Technology overview Introduction This section introduces the concept of the Cloud-enabled Infrastructure as a whole and how its key components are integrated. This section also describes the hardware and software components part of this solution. The Cloud-enabled Infrastructure for SAP solution is a result of the preferred three-way partnership between EMC, SAP, and VMware. The infrastructure is divided into several different functionalities, which are delivered as separate add-on bundles, as illustrated in Figure 1. The EMC Cloud-enabled Infrastructure for SAP solution is designed to offer flexibility so that customers can choose the required cloud functionalities (hereon referred to as add-on bundles), which they can enable without losing sight of their end goal an efficient and reliable private cloud. Figure 1. Cloud-enabled Infrastructure for SAP A journey to the private cloud starts with the foundation bundle, which is described in EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle White Paper. The EMC Cloud-enabled Infrastructure for SAP foundation bundle is a mandatory component and it is the base for all add-on bundles. Table 2 outlines the functions and benefits of the EMC Cloud-enabled Infrastructure foundation bundle. Table 2. On-premise EMC Cloud-enabled Infrastructure for SAP solution: Foundation bundle Function Benefits Technology Virtual datacenter for SAP Autonomy of business units and application operations Service catalogs Service level agreements (SLAs) Management of vcloud tenants Resource pooling VMware vcloud Suite Enterprise 10

11 Function Benefits Technology Infrastructure chargeback Cost measurement, analysis, and reporting of the use of compute, network, storage, and backup resources (in combination with data protection add-on bundle) VMware vcenter Chargeback Integrated cloud management and performance analysis Storage tiering Cloud networking and security for SAP Manage availability, capacity, performance, and health in the SAP landscape Automatically get the right data to the right place at the right time Cloud-enabled Infrastructure security framework Authorization concepts Compliance and non-compliance tracking VMware vcenter Operations Manager EMC Storage Resource Management EMC Storage Analytics (ESA) EMC Virtual Storage Integrator (VSI) EMC Symmetrix VMAX EMC FAST VP VMware vcloud networking and security For more information about the foundation bundle, refer to the EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle White Paper. Table 3 lists the five add-on bundles that enable more cloud functionalities and data center enhancements. Table 3. EMC Cloud-enabled Infrastructure for SAP solution: Add-on bundles IT strategy Bundle Benefits Technology Business continuity High availability (HA) and application mobility for SAP High availability within one datacenter and across two datacenters with application awareness for the complete SAP landscape Non-disruptive movement of applications from one datacenter to another datacenter Improved resource utilization across datacenters Minimize downtime for infrastructure maintenance EMC VPLEX Metro vsphere HA vsphere DRS Symantec ApplicationHA Disaster recovery (DR) for SAP Provide disaster recovery protection for cloud management applications and SAP systems VMware vcenter Site Recovery Manager EMC RecoverPoint 11

12 IT strategy Bundle Benefits Technology Data protection (DP) for SAP Provide data protection to cover the cloud management applications and SAP systems Backup and recovery at all datacenters with remote replication of backup sets for offsite protection EMC Avamar EMC Data Domain EMC Data Protection Advisor Service support/ provisioning Automation and operations for SAP SAP application virtualization enables any service any time on any server Provision SAP systems on demand with automated endto-end process SAP NetWeaver Landscape Virtualization Management Reduce downtime window during maintenance leveraging mass operations Enhanced security Enhanced security and compliance for SAP Efficient, collaborative enterprise governance, risk and compliance (egrc) program across IT, finance, operations, and legal domains RSA Archer egrc RSA Data Loss Prevention RSA SecurID Data loss prevention Secure user to network authentication 12

13 Solution architecture The following sections describe the additional components required for the EMC Cloudenabled Infrastructure for SAP to transition from the foundation bundle to the HA and application mobility add-on bundle, as shown in Figure 2. Figure 2. Transition from foundation to HA/Mobility for SAP bundle Note: In the solution validated, the EMC Symmetrix VMAX 20K model was used, but other EMC Symmetrix VMAX models can also be used. 13

14 This bundle combines EMC VPLEX Metro and VMware vsphere stretched cluster with the EMC Cloud-enabled Infrastructure for SAP foundation bundle, as well as providing a high level of availability. VPLEX Metro and vsphere HA/DRS in a stretched cluster provide high availability through the effective use of cluster resources at both datacenters, as shown in Figure 3. Figure 3. High availability infrastructure layers Storage high availability In the EMC Cloud-enabled Infrastructure for SAP foundation bundle, all vsphere virtual machines in both the management and resource clusters are stored in VMware datastores based on LUNs in the local EMC Symmetrix VMAX. This bundle uses EMC VPLEX Metro federated solution across the two datacenters to provide the distributed storage to the all the vsphere ESXi hosts across datacenters, using the EMC VPLEX Metro distributed virtual volumes. An EMC VPLEX Metro Distributed Virtual Volume is seen as only one volume (datastore). It is replicated synchronously (read/write) on the EMC Symmetrix VMAX arrays on both datacenters, independent from where updates are being made to them. EMC VPLEX Metro provides active copies of data at both datacenters and across a stretched layer 2 IP network. VMware vmotion provides uninterrupted mobility of application workloads across the datacenters enabled by EMC VPLEX. 14

15 This bundle also uses EMC VPLEX Witness to monitor connectivity between the two EMC VPLEX clusters on each datacenter and ensure continued availability in the event of an inter-cluster link failure or a datacenter failure. EMC VPLEX Witness is deployed on a virtual machine at a third, separate failure domain (Datacenter C). SAN resiliency with EMC PowerPath/VE EMC PowerPath/VE provides the capability to: Automate and optimize data paths in virtual environments to ensure business process availability and performance. Protect virtual environments from physical hardware failures to ensure uninterrupted service and the automatic failover and recovery. Simplify load balancing to help eliminate I/O bottlenecks. The load balancing algorithms of EMC PowerPath/VE automatically adjust the I/O path use from virtual machines to local I/O loads. With EMC VPLEX Cross-Cluster Connect, PowerPath/VE leaves the cross cluster volumes in the standby mode, if it detects a preferred path to the local VPLEX storage volumes. Network high availability In each datacenter, a redundant 10Gb Ethernet fabric provides the core IP Network for datacenter traffic and uplinks for user access. The same IP subnets are available at both datacenters. The layer 2 routers extend the layer 2 broadcast domains across the two datacenters. 15

16 Physical architecture Figure 4 shows the physical architecture of all layers of the solution, including the IP and SAN network components. Figure 4. Physical architecture The physical architecture depicted in Figure 4 provides redundant components and connections in all infrastructure layers. Brocade gear was used for the SAN and IP networks in the validated lab environment, but Cisco gear with the same specifications can be used instead, or a combination of both can be deployed. IP network layer The IP network in each datacenter is built using two Brocade VDX 6720 switches in a VCS configuration. All vsphere ESXi hosts are connected to the network using redundant 10 GbE connections provided by 10 Gb CNA cards. The two switches at each datacenter 16

17 are connected to a router using a Virtual Link Aggregation Group (vlag). The routers extend the Layer 2 network between two datacenters. Note: A vlag is a fabric service that enables a Link Aggregation Group (LAG) to originate from switches. As the standard LAG, a vlag uses the Link Aggregation Control Protocol (LACP) to control the bundling of several physical ports together to form a single logical channel. All network traffic between Datacenter A and Datacenter B are routed using multiple ports configured as a LAG. Figure 5 shows the IP network layer. Figure 5. IP network layer SAN layer The SAN in each datacenter was built with Brocade DCX 8510 Backbones switches as shown in Figure 6. All vsphere ESXi hosts are connected to the SAN using redundant 8 GbE connections provided by the pair of HBAs in each server. The multiple Fibre Channel (FC) connections between the Brocade DCX 8510 Backbone switches are not only used for mirroring EMC VPLEX Metro storage across datacenters, but also provide a HA Cross- Cluster Connect between datacenters. Figure 6. SAN Layer 17

18 Logical architecture Figure 7 shows the logical architecture of all layers of the solution, including the network components. Figure 7. Logical architecture The logical architecture depicted in Figure 7 demonstrates that the management and resource clusters are stretched to span both datacenters. 18

19 Protection layers Table 4 summarizes the high availability layers provided before and after the transition between the foundation and HA bundle. DC stands for datacenter. Table 4. High availability in local and remote datacenters ID Layer Protected components DC DC Protection 01 SAP Application SAP work processes (DIA/UPD/UPD2/SPO) A A & B Multiple SAP Appl. Servers 02 Operating System SAP ASCS, SAP Database, SAP PAS, SAP AAS1, SAP AAS2, SAP AAS3 and SAP shared file systems A A & B Symantec ApplicationHA 03 Virtual machine Management and SAP virtual machines A A & B VMware vsphere HA / DRS 04 Host Server A A, B, & C VMware vsphere ESXi 05 Fabric SAN Fabric paths management A A & B EMC PowerPath/VE 06 LAN IP Network uplinks A A & B Redundant 10GbE IP networks 07 SAN Storage area network A A & B Redundant 8Gb FC networks 08 SAN Storage area network paths A A & B EMC Cross-Cluster Connect 09a SAN Storage array A EMC VPLEX Local 09b SAN Storage array A & B EMC VPLEX Metro 10 Storage Local storage resources (drives, FAs, Das, and so on) A B EMC Symmetrix VMAX 11 Storage Local storage resources (drives, FAs, Das, and so on) A A EMC Symmetrix VMAX Local high availability can also be provided for a single datacenter by using the components numbered 01 to 07 and 11 described in Table 4. For more information, refer to the VMware and EMC documents listed in the References section. EMC VPLEX Local provides a higher level of local high availability with a clustering architecture that provides virtual volumes replicated between two local storage arrays. This allows the servers in the local datacenter to have read/write access to both shared block storage devices. This can be accomplished by using the components described in Table 4 except for component numbered 09b. For more information, refer to the EMC VPLEX documents listed in the References section. EMC VPLEX Metro extends the concept of local high availability to an additional metro distance datacenter. This storage transformation takes high availability to a new level of mission-critical business continuity. The components required for this configuration are all the components described in the Protection column in Table 4, except the component with the number 09a. All scenarios described above require the installation, configuration, and the implementation of the best practices for each component as described in the corresponding sections later in this document and in the documents listed in the References section. Figure 7 shows the high-availability design with VPLEX Witness and Cross-Cluster Connect deployed in this bundle, providing the highest level of resilience. Each of the components shown in Table 4 is explored in more detail in the relevant sections of this white paper. 19

20 Hardware resources Table 5 details the hardware resources for the solution. Table 5. Solution hardware resources Purpose Quantity Configuration Storage (Datacenter A) Storage (Datacenter B) 1 1 EMC Symmetrix VMAX 20K, with FLASH/SAS/NL-SAS Storage Pools under FAST VP control EMC Symmetrix VMAX 20K, with FLASH/SAS/NL-SAS Storage Pools under FAST VP control Distributed storage federation 2 EMC VPLEX Metro cluster, with2 x VS2 engines ESXi hosts for management cluster 4 2 x four-core CPUs, 128 GB RAM ESXi hosts for resource cluster 4 4 x eight-core CPU, 256GB RAM ESXi hosts for VPLEX Witness 2 2 x two-core CPUs, 48 GB RAM Network switching and routing platform 2 Brocade DCX 8510 Backbone, with: Fx8-24 FC extension card 2 x 48-port FC Blades with 16 Gb FC line speed support Brocade MLXe Router 4 Brocade VDX 6720 in VCS mode Software resources Table 6 details the software resources used in the solution. Note: Other operating system and database combinations supported by the EMC Solutions Support Matrix (SSM) can also be used. Table 6. Solution software resources Software Version Purpose EMC Enginuity 5876 Operating environment for Symmetrix VMAX EMC PowerPath/VE 5.8 Multipathing software used to provide continuous active EMC Solutions Enabler 7.5 paths Symmetrix command line interface EMC Unisphere VMAX management software EMC Unisphere for VPLEX 5.2 VPLEX management software EMC VPLEX Witness 5.2 Handles VPLEX failures & inter-cluster communication SAP ERP 6.0 EHP5 loss IDES/Unicode - Reference system SAP NetWeaver 7.02 Unicode x86_64 Oracle Database Used on the SAP systems Microsoft SQL Server 2008 R2 Used by VMware vcenter Chargeback and vcloud Director VMware vsphere 5.1 Hypervisor hosting all virtual machines VMware vcenter Server 5.1 Management and resource clusters VMware vcloud Director 5.1 Orchestration Tool for Provisioning Virtual Infrastructure VMware vcloud Networking and Security 5.1 Security software 20

21 Software Version Purpose SUSE Linux Enterprise for SAP Applications 11 SP2 Operating system for the SAP virtual machines Symantec ApplicationHA Application-aware clustering software for the Guest OS of virtual machines 1 For release availability information, see the Symantec website. 21

22 Key components Overview EMC VPLEX This section introduces the following technology components: EMC VPLEX EMC Symmetrix VMAX EMC PowerPath /VE EMC Virtual Storage Integrator (VSI) VMware vcloud Suite Symantec ApplicationHA EMC VPLEX Metro is the primary technology enabler in the solution. EMC VPLEX Metro is a storage area network-based (SAN) federation solution that delivers both local and distributed storage federation. VPLEX Metro enables the same data to exist in two datacenters in separate geographical locations, and to be accessed and updated at both datacenters at the same time. With EMC VPLEX Witness in the solution, applications continue to be available, with minimal interruption or downtime, in the event of an outage on EMC VPLEX at one of the datacenters. EMC VPLEX is a storage virtualization solution for both EMC and non-emc storage arrays. EMC offers VPLEX in three configurations to address customer needs for high availability and data mobility: EMC VPLEX Local EMC VPLEX Metro EMC VPLEX Geo For more information, refer to EMC VPLEX Metro infrastructure. EMC Symmetrix VMAX EMC PowerPath/VE EMC VSI The EMC Symmetrix VMAX and the Enginuity operating environment is built to support top IT priorities of enterprise customers, including the transformation to the private cloud and big data processing. The EMC Symmetrix VMAX system builds on the revolutionary scale-out Virtual Matrix Architecture that features unprecedented performance, availability, functionality, and cost efficiency. PowerPath/VE is a path management solution for VMware. It provides the highest level of dynamic load balancing, path failover, path restoration, path testing, and automated performance optimization. EMC VSI provides multiple feature sets including Storage Viewer (SV), Path Management, and Unified Storage Management. It facilitates the discovery and identification of all EMC storage devices allocated to vsphere ESXi hosts and virtual machines. Unified Storage Management simplifies the provisioning of Symmetrix VMAX virtual pooled storage for datacenters, vsphere ESXi host, and resource pools. Path Management allows you to control how users access datastores. 22

23 VMware vcloud Suite VMware vcloud Suite provides all components for building and running a private cloud infrastructure, based on VMware vsphere that leverages the software-defined datacenter architecture. This architectural approach delivers virtualized infrastructure services (compute, network, security, and availability) with built-in intelligence to automate the on-demand provisioning, placement, configuration and control of applications based on defined policies. This solution includes the following vcloud Suite components: VMware vsphere Compute virtualization platform with policy-based automation VMware vcenter Site Recovery Manager 2 Automated disaster recovery planning, testing, and execution VMware vcloud networking and security 3 Networking and security with ecosystem integration for a virtualized compute environment. VMware vcenter Operations Management Suite Integrated, proactive performance, capacity, and configuration management for dynamic cloud environments. VMware vcloud Director Virtualized datacenters with multi-tenancy and public cloud extensibility VMware vsphere VMware vsphere virtualizes and aggregates the underlying physical hardware resources across multiple systems and provides pools of virtual resources to the datacenter. As a cloud operating system, VMware vsphere manages large collections of infrastructure (such as CPUs, storage, and networking) as a seamless and dynamic operating environment, managing the complexity of a datacenter. vsphere HA maximize uptime across your virtualized infrastructure, reducing unplanned downtime and eliminating planned downtime for server and storage maintenance. VMware vcloud networking and security VMware vcloud networking and security is the leading software-defined networking and security solution that enhances operational efficiency and unlocks agility, enables extensibility to rapidly respond to business needs, and provides a broad range of services in a single solution, including virtual firewall, VPN, load balancing, and VXLAN extended networks. VMware vcloud Director VMware vcloud Director orchestrates the provisioning of software-defined datacenter services as complete virtual datacenters that are ready for consumption in a matter of minutes. Virtual datacenters provide virtualized computing, networking, storage, and security. Software-defined datacenter services and the virtual datacenters fundamentally simplify infrastructure provisioning, and enable IT to move at the speed of business. 2 This is used in the disaster recovery bundle solution. 3 This is used in the foundation bundle solution. 23

24 Symantec ApplicationHA Symantec ApplicationHA is a software solution integrated on top of vsphere HA and provides application awareness to vsphere HA, so it can monitor and restart them automatically if a hardware or software failure occurs inside the VMware virtual machines. Symantec ApplicationHA adds a layer of application awareness to the core HA functionality offered by VMware vsphere virtualization technology. Symantec ApplicationHA is based on Veritas Cluster Server (VCS) and uses similar concepts such as agents, resources, and service groups. Symantec ApplicationHA has a lightweight server footprint that allows faster installation and configuration. 24

25 EMC VPLEX Metro infrastructure Introduction This section describes the VPLEX Metro infrastructure for the solution, which is composed of the following components: EMC VPLEX Metro cluster at each datacenter (Datacenter A and B) EMC VPLEX Witness in a separate failure domain (Datacenter C) EMC VPLEX EMC VPLEX is a storage virtualization solution for both EMC and non-emc storage arrays. EMC offers VPLEX in different configurations to address customer needs for high availability and data mobility. For detailed descriptions of these VPLEX configurations, refer to the documents listed in the References section. EMC VPLEX Metro VPLEX Metro uses a unique clustering architecture to allow servers at multiple datacenters, geographically dispersed, to have read/write access to shared block storage devices. VPLEX Metro delivers active/active, block-level access to data on two datacenters within synchronous distances with a round-trip time of up to 5 milliseconds. For this bundle, a round-trip time is not to exceed 1 millisecond with the Cross-Cluster Connect configuration. EMC VPLEX Witness VPLEX Witness is an external server that is installed as a virtual machine in a separate failure domain to the VPLEX clusters. VPLEX Witness connects to both VPLEX clusters using a Virtual Private Network (VPN) over the management IP network. It requires a round-trip time (RTT) that does not exceed 1 second. By reconciling its own observations with information reported periodically by the clusters, VPLEX Witness enables the cluster(s) to distinguish between inter-cluster network partition failures and cluster failures and automatically resume I/O at the appropriate datacenter. VPLEX Witness failure handling semantics apply only to distributed volumes within a consistency group and only when the detach rules identify a static preferred cluster for the consistency group (see VPLEX consistency groups for further details). EMC VPLEX management interface You can manage and administer a VPLEX environment with the web-based Unisphere for VPLEX or you can connect directly to a management server and start a VPLEX command line interface (VPLEXcli) session. 25

26 EMC VPLEX HA VPLEX Metro enables application and data mobility. When configured with VPLEX Witness, it provides a high-availability infrastructure for clustered applications. VPLEX Metro enables you to build a stretched cluster like a local cluster, and removes the datacenter as an SPOF. Furthermore, as the data and applications are active at both datacenters (active/active), this configuration provides a simple high availability business continuity solution. VPLEX Witness is part of the VPLEX Metro and enables two optional architectures as follows: VPLEX Metro HA with Cross-Cluster Connect Hosts are connected to the local VPLEX cluster on which they reside, and also have an alternate path to the remote VPLEX cluster. You can achieve the highest degree of availability by using a VPLEX Cross-Cluster Connect configuration. In the unlikely event that an entire VPLEX cluster, storage array, or SAN fails, with Cross-Cluster Connect configuration, hosts have an alternate path to the VPLEX Metro distributed volumes through the surviving VPLEX cluster, eliminating a short downtime associated with restarting the virtual machine on the secondary datacenter. The inter-cluster network latency should not exceed 1 millisecond RTT between VPLEX clusters. VPLEX Metro HA without Cross-Cluster Connect Hosts are only connected to the local VPLEX cluster on which they reside. Because host clusters are connected to the VPLEX distributed volume providing the same block data between two datacenters, when components (hosts, storage, servers and so on) fail, it minimizes the recovery time by automatically restarting the virtual machine on the secondary datacenter. The latency exceeds 1 millisecond but should be less than 5 milliseconds RTT. VPLEX logical storage structures VPLEX encapsulates traditional physical storage array devices and applies layers of logical abstraction to these exported LUNs, as shown in Figure 8. Figure 8. VPLEX logical storage structures VPLEX storage volume A storage volume is a LUN exported from an array and encapsulated by VPLEX. An extent is the mechanism VPLEX uses to divide storage volumes and may use all or part of the 26

27 capacity of the underlying storage volume. A device encapsulates an extent or combines multiple extents or other devices into one large device with a specific RAID type. At the top layer of the VPLEX storage structures are virtual volumes. These are created from a top-level device (a device or distributed device) and always use the full capacity of the top-level device. Virtual volumes are the elements that VPLEX exposes to hosts using its front-end ports. VPLEX presents a virtual volume to a host through a storage view. VPLEX can encapsulate devices across heterogeneous storage arrays, including virtually provisioned thin devices and traditional LUNs. VPLEX consistency groups Consistency groups aggregate virtual volumes so that the same properties (detach rules and others) can be applied to all volumes in the group. There are two types of consistency groups: Synchronous consistency groups These are used in VPLEX Local and VPLEX Metro to apply the same detach rules and other properties to a group of volumes in a configuration. This simplifies configuration and administration in large systems. With write-through caching in synchronous consistency groups, in the separated cluster environment, VPLEX Metro supports up to 5 milliseconds of latency. VPLEX Metro sends writes to the back-end storage volumes, and acknowledges a write to the application only when the back-end storage volumes in both clusters acknowledge the write. Asynchronous consistency groups These are used for distributed volumes in VPLEX Geo, where clusters can be separated by up to 50 milliseconds of latency. Consistency groups are particularly important for databases and their applications. For example: Write-order fidelity Maintains data integrity, vsphere ESXi LUNs forming a datastore cluster should be placed together in a single consistency group. Transactional dependency Multiple databases often have transaction dependencies, such as when an application issues transactions to multiple databases and expects the databases to be consistent with each other. All LUNs that require I/O dependency to be preserved should reside in a single consistency group. Application dependency SAP stores database files within a set of datastores that must be accessible to maintain database availability. The datastore devices of database files should reside in a single consistency group. Detach rules Detach rules are predefined rules that determine I/O processing semantics for a consistency group when connectivity with a remote cluster is lost, in the case of a network partitioning or remote cluster failure, for example. Synchronous consistency groups support the following detach rules to determine cluster behavior during a failure: The static preference rule identifies a preferred cluster. 27

28 The no-automatic-winner rule suspends I/O on both clusters. Setting a detach rule is always invoked when connectivity is lost between clusters. However, VPLEX Witness can be deployed to override the static preference rule and ensure that the non-preferred cluster remains active if the preferred cluster fails. VPLEX Metro solution configuration Storage structures Figure 9 shows the physical and logical storage structure used by VPLEX Metro in this solution. Figure 9. VPLEX physical and logical storage structures for solution A one-to-one mapping between storage volumes, extents, and devices exists at each datacenter. The devices inside of both Datacenter A (cluster-1) and Datacenter B (cluster-2) are virtually provisioned thin devices. All cluster-1 devices are mirrored remotely on cluster-2, in a distributed configuration, to create distributed devices. These distributed devices are encapsulated by virtual volumes, which are then presented to the hosts through storage views. Configuration process For this bundle, we used the configuration wizard provided by Unisphere for VPLEX to configure the VPLEX Metro logical storage structures. Storage volume Figure 10 shows that several storage volumes were created on Datacenter A, as displayed in the VPLEX Management Console. 28

29 Figure 10. EMC VPLEX storage volumes (Datacenter A) Extent VPLEX divides storage volumes in extents. This bundle includes a one-toone mapping between extents and storage volumes, as shown in Figure 11. Extents have the same size as the storage volumes from which they are created. Figure 11. EMC VPLEX Extents Device One-to-one mapping was configured between devices and extents. Figure 12 shows the option used to configure this mapping. Figure 12. EMC VPLEX device creation wizard: mapping extents Distributed device The distributed devices were created by mirroring a device remotely in a distributed RAID 1 configuration, as shown in Figure

30 Figure 13. EMC VPLEX device creation wizard: selecting mirrors Virtual volume All top-level devices are distributed devices. These devices are encapsulated by virtual volumes, which EMC VPLEX presents to the hosts through storage views. The storage views define which hosts access which virtual volumes on which EMC VPLEX ports. Consistency group In this bundle, two synchronous consistency groups were created as shown in Figure 14. Resource cluster consistency group contains the SAP virtual machines. Management cluster consistency group contains the management virtual machines. Figure 14. Consistency groups configuration in Unisphere for VPLEX 30

31 VPLEX Witness configuration This bundle uses EMC VPLEX Witness to monitor connectivity between the two VPLEX clusters and ensure continued availability when an inter-cluster network partition fails or a datacenter fails. This is considered a VPLEX Metro HA configuration as storage availability is ensured at the surviving datacenter. EMC VPLEX Witness was deployed at a third, separate failure domain (Datacenter C) and was connected to the EMC VPLEX clusters at Datacenter A and Datacenter B. Datacenter C is located at a distance of less than 1 second RTT from Datacenters A and B. When a VPLEX Witness has been installed and configured, the VPLEX Management Console displays the status of cluster witness components, as shown in Figure 15. Figure 15. EMC VPLEX Witness components and status For additional details, refer to the Implementation and Planning Best Practices for EMC VPLEX Technical Note and the Using VPLEX Metro with VMware High Availability and Fault Tolerance for Ultimate Availability White Paper listed in the References section. 31

32 VMware virtual datacenter Introduction VMware virtual datacenter is the key enabling technology introduced in the EMC Cloud- Enabled Infrastructure for SAP Foundation Bundle White Paper. This solution reuses the existing architecture in the foundation bundle and configures the HA capability as an add-on. The SAP systems are fully virtualized in vapps using VMware vsphere and vcloud Director. The same networking and security configurations in foundation bundle also apply to the HA and application mobility add-on bundle. This section describes the following components, technologies and options: VMware vcloud Director VMware vsphere VMware vcenter Server VMware vcenter Chargeback Manager VMware vsphere vmotion VMware vsphere HA VMware vsphere DRS EMC PowerPath/VE for VMware vsphere EMC VSI for VMware vsphere VMware vcloud Director VMware vcloud Director applies the principles of pooling, abstraction, and automation to all datacenter services like storage, networking, and security using virtual datacenters. A virtual datacenter is an elastic logical container that provides all infrastructure services necessary to make workloads operational in minutes. Applications provisioned into these containers are automatically placed in the most optimal VMware vcenter server cluster. VMware vsphere VMware vsphere is a virtualization platform that provides infrastructure services transforming IT hardware into a shared computing platform, and application services helping IT organizations deliver high levels of availability, security, and scalability. vsphere is the base component in the EMC Cloud-enabled Infrastructure for SAP on all add-on bundles. It provides an abstraction of the physical server layer that allows virtual machines to be independent of the brand, model and type of x86 server architecture where they are running. VMware vcenter Server VMware vcenter Server is the centralized management platform for vsphere environments, enabling control and visibility at every level of the virtual infrastructure. vcenter Servers manage vsphere HA/ DRS clusters that are created between the two datacenters using VMware vsphere 5 hosts, and are connected to the vsphere hosts at both datacenters. 32

33 VMware vcenter Chargeback Manager VMware vcenter Chargeback Manager improves utilization of your virtual infrastructure with accurate visibility into the true costs of virtualized workloads. It enables line-ofbusiness owners to have full cost transparency and accountability for self-service resource requests. It allows IT organizations to customize rate cards and prices to the processes and policies of different organizations. VMware vsphere vmotion VMware vsphere vmotion is VMware technology that supports live migration of virtual machines across servers without any interruption in the availability of the virtual machine. This allows the live relocation of virtual machines to new datastores. VMware vsphere HA VMware vsphere HA is a vsphere component that provides high availability for any application running in a virtual machine, regardless of its operating system or underlying hardware configuration. vsphere HA provides uniform failover protection against hardware and operating system outages within your virtualized IT environment. vsphere HA specifically reduces unplanned downtime by using multiple VMware vsphere ESXi hosts, configured as a cluster, to provide rapid recovery from outages and cost-effective high-availability for applications running on virtual machines. VMware vsphere DRS VMware vsphere DRS dynamically and automatically balances load distribution and virtual machine placement across multiple ESXi hosts using vsphere vmotion. EMC PowerPath/VE for VMware vsphere EMC PowerPath/VE for VMware vsphere delivers multipathing features that optimize VMware vsphere virtual environments. PowerPath/VE installs as a kernel module on the vsphere ESXi host and works as a multipathing plug-in (MPP) that provides enhanced path management and load-balancing capabilities for vsphere ESXi hosts. EMC VSI for VMware vsphere EMC VSI for VMware vsphere is a vcenter plug-in that provides a single management interface for managing EMC storage. VSI provides a unified and flexible user experience that allows each feature to be updated independently, and allows new features to be introduced rapidly in response changing customer requirements. When PowerPath/VE is installed on a vsphere ESXi host, VSI presents important multipathing details for devices, such as the load-balancing policy, the number of active paths, and the number of dead paths. 33

34 Configuring VMware vcloud Director This section describes how the VMware vcloud Director settings were configured to integrate this HA bundle with the EMC Cloud-enabled Infrastructure for SAP foundation bundle. Figure 16 illustrates the cloud architecture with the HA add-on bundle. Figure 16. Cloud architecture with the HA bundle To enable the HA capability, the following components must be configured: vsphere DRS cluster Storage Provider vdc (PvDC) Organization vdc (OvDC) 34

35 Configuring DRS cluster DRS clusters were configured to fully use HA capability for both management and resource clusters: Management cluster vsphere ESXi hosts from Datacenter B were added to the original management DRS cluster, facilitating the management cluster with cross datacenter HA capability. Resource cluster A new DRS cluster named Premium was created. vsphere ESXi hosts from both datacenters were added to the Premium DRS cluster, facilitating the resource cluster with cross datacenter HA capability. Configuring storage New datastores were created based on the EMC VPLEX Metro Distributed virtual volumes for both resource and management clusters together with storage DRS clusters and storage profiles to enable HA on the storage level. The virtual machines on management cluster were non-disruptively migrated from the local EMC Symmetrix VMAX located on Datacenter A to the EMC VPLEX Metro distributed virtual volumes. New datastores were created for both management and resource clusters, and the storage settings configured in vcloud Director are listed in Table 7. Table 7. Storage configuration in vcloud Director Consistency group Storage DRS cluster Storage profile Datastores Management clusters HA MGMT N/A VPLEX_MGMT_001 (4 TB) VPLEX_MGMT_002 (4 TB) Resource clusters HA Gold Tier HA Gold VPLEX_DS1 (4 TB) VPLEX_DS2 (4 TB) VPLEX_DS3 (4 TB) VPLEX_DS4 (4 TB) VPLEX_DS5 (4 TB) HA Silver Tier HA Silver VPLEX_DS1_Silver (4 TB) Eight 4 TB datastores were created for the management and resource clusters. For the resource cluster, two storage DRS clusters were created to group datastores with different performance levels: HA Gold Tier A storage DRS cluster for placing datastores set to the highest performance. HA Silver Tier A storage DRS cluster for placing datastores set to high performance. Five 4 TB datastores were assigned to the HA Gold Tier storage DRS cluster and one 4 TB datastore to the HA Silver Tier storage DRS cluster. Two new storage profiles, HA Gold and HA Silver were created to characterize the datastores with HA capabilities. 35

36 Afterwards, the storage profiles were assigned to the corresponding datastores in the storage DRS clusters as shown in Table 7. All created distributed volumes were presented to the all four vsphere ESXi hosts on both stretched clusters. Figure 17 shows the configuration details for the datastore cluster HA Gold Tier created for the resource cluster. Figure 17. HA Gold Tier datastore cluster Configuring PvDC Two PvDCs, Advanced and Standard, were defined in EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle White Paper. In this bundle a new PvDC named Premium was created based on the Premium cluster from the DRS cluster. The Premium PvDC is dedicated for SAP systems that require cross datacenter HA capability. Storage profiles HA Gold and HA Silver were added to the PvDC storage profile tab in the PvDC configuration. Figure 18 shows the storage profile configuration needed for PvDC. 36

37 Figure 18. PvDC configuration For external networks in PvDC, the configuration in Advanced and Standard PvDC also applies to the Premium PvDC. Configuring OvDC We create new OvDCs to use cross datacenter HA capability. In this solution, a new OvDC called OrgB_Dedicated_HA for OrgB was built on top of Premium PvDC. The same organization networks configured in the foundation bundle were also created in the new OvDC. For vshield Edge gateways, we enabled high availability in the edge property as shown in Figure 19. Figure 19. vshield Edge gateway configuration 37

38 After this configuration, vcloud Director automatically deployed an additional standby vshield Edge gateway. Once the primary vshield Edge gateway is out of service, all the networking and security functions switch over to the standby gateway. After successful configuration of the new OvDC, new SAP systems can be deployed. For the existing SAP systems that also require HA capability, but originally deployed in the foundation bundle, you can schedule a downtime and migrate them to the new OvDC. Configuring vcenter Chargeback Manager vcenter Chargeback Manager provides the virtualized infrastructure metering functionality. It integrates with vcloud Director seamlessly, provides the service provider with the capability to chargeback the resources consumed by tenants, and generates cost and utilization reports periodically or on demand. The EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle White Paper defines several pricing models to differentiate OvDC with different allocation models and storage profiles with different storage performance levels. There are several approaches to reflect the chargeback of HA capability. You can create and bind a pricing model to a new OvDC with HA capability, with reasonable charges for infrastructure including CPU, memory, and storage. In this way, you can accurately charge the infrastructure and account for the resources consumed in Datacenter B. Another simpler approach is to set a rate factor for the storage profiles HA Gold and HA Silver to reflect a charge for the HA capability. You only need to set it once and all the vapps deployed on the storage profiles HA Gold and HA Silver will be charged accordingly with base rate times rate factor. That simplifies the management efforts needed to create pricing models for each OvDC compared to the previously described approach. Figure 20 shows how to set the rate factor in vcenter Chargeback Manager. Figure 20. Setting the factor rate for DR storage profiles VMware deployments on VPLEX Metro EMC VPLEX Metro delivers concurrent access to the same set of devices at two physically separate locations and thus provides the active/active infrastructure that enables geographically stretched clusters based on VMware vsphere. The use of layer 2 routing technology enables extension of VLANs, and their subnets, across different physical datacenters. 38

39 Deploying VPLEX Metro with the following features and components provides the described functionality: vmotion Provides the ability to non-disruptively migrate virtual machines between datacenters in anticipation of planned events such as hardware maintenance or power outage of the datacenter. vsphere DRS Provides automatic load distribution and virtual machine placement across datacenters through the use of DRS groups and affinity rules. vsphere HA A VPLEX Metro environment with VPLEX Witness configured is considered as a VPLEX Metro HA configuration, as it ensures storage availability at the surviving datacenter in the event of a datacenter-level failure. Combining VPLEX Metro HA with a host failover clustering technology such as vsphere HA and Symantec ApplicationHA provides an end-to-end automatic application-aware restart for any datacenter-level failure or other outages. Figure 21 illustrates this HA architecture. Figure 21. vsphere HA with Cross-Cluster Connect and VPLEX Witness logical view VPLEX Metro HA Cross-Cluster Connect Provides additional protection of the vsphere HA cluster by adding a Cross-Cluster Connect between the local vsphere ESXi hosts and the VPLEX cluster on the remote datacenter. EMC VPLEX distributed volumes are created across both locations. With underlying EMC VPLEX distributed volumes, vsphere HA clusters are stretched across both datacenters. 39

40 The physical vsphere ESXi hosts are connected to the local VPLEX cluster on which they physically reside, and also have an alternate path to the remote EMC VPLEX cluster through the additional cross-connect network that physically breaks the VPLEX internal link connecting both VPLEX clusters, as show in Figure Figure 22 shows the paths for Cross-Cluster Connect as displayed by EMC VSI. EMC PowerPath/VE automatically detects and sets the cross-cluster paths and puts in auto standby mode. This minimizes cross-datacenter traffic under normal operation, and handles all paths down (APD) situations more efficiently when a datacenter fails. Figure 22. EMC VSI PathViewer: VPLEX distributed devices and cross cluster paths Best practices for EMC VPLEX Cross-Cluster Connect configuration VPLEX Witness must be deployed in a third failure domain. The inter-cluster network latency is not to exceed 1 millisecond RTT between VPLEX clusters at the time of writing. According to EMC best practice, all remote VPLEX connections should be zoned to the local host and local host initiators must be registered to the remote VPLEX. The distributed volume is then exposed from both VPLEX clusters to the same host. The host path preference should have a local path preference set, ensuring the remote path will only be used if the primary one fails so that no additional latency is incurred. This bundle uses VPLEX Metro HA with Cross-Cluster Connect to maximize the availability of the VMware vsphere virtual machines. The key benefit of this solution is the ability to minimize any recovery time if components or even an entire VPLEX cluster fails. This is unlikely because there is no SPOF within a VPLEX engine. Since the physical host has an alternate path to the same storage actively served up by the remote VPLEX cluster, it will automatically remain online due to the VPLEX Witness, regardless of the rule set. 4 For detailed information, see EMC VPLEX Metro Witness Technology and High Availability EMC TechBook and Using VPLEX Metro with VMware High Availability and Fault Tolerance for Ultimate Availability White Paper in listed in References. 40

41 VMware stretched cluster configuration VMware and EMC support a stretched cluster configuration that includes vsphere ESXi hosts from multiple datacenters. A VMware vsphere Metro Storage Cluster (vmsc) 5 solution is also referred to as a stretched cluster. In this bundle, both management and resource clusters, described in the EMC Cloud- Enabled Infrastructure for SAP Foundation Bundle White Paper, are stretched between Datacenter A and Datacenter B by using the EMC VPLEX Metro distributed virtual volumes with vsphere HA and vsphere DRS. There are four vsphere ESXi hosts (physical servers) in each cluster, two at each datacenter, two for the management cluster and two for the resource cluster. VMware vcloud Director manages the resource cluster. EMC VPLEX Metro HA Cross-Cluster Connect provides increased resiliency to the configuration as shown in Figure 21. In VMware vcenter, you can view the vsphere Web Client configuration of the stretched cluster and the features enabled for it, as shown in Figure 23. This view also shows the memory, CPU, and storage resources available to the resource cluster. Figure 23. vsphere cluster with HA and DRS enabled The management cluster was configured in the same manner, with vsphere HA and DRS enabled and all virtual machines stored in VPLEX-based datastores. 5 For detailed requirements and scenarios, see the VMware Knowledge Base articles : Using VPLEX Metro with vsphere HA and : Implementing vsphere Metro Storage Cluster (vmsc) using EMC VPLEX listed in References. 41

42 Configuring virtual network Each vsphere ESXi host is configured with two 10 GbE physical adapters to provide network failover and high performance. A vsphere distributed switch (dvswitch) 6 provides a single, common switch across all hosts. The 10 GbE physical adapters (also referred to as uplink adapters) are assigned to the dvswitch. Four distributed port groups are assigned to the dvswitch: dvpg_host For virtual machine datacenter network traffic dvpg_management For management traffic dvpg_corp For uplinks to corporate IT Network dvpg_vmotion For vmotion traffic Figure 24 shows the dvswitch configuration. As both vsphere 5.1 distributed switches and physical switches support Link Layer Discovery Protocol (LLDP), the properties of the associated physical switches can also be easily identified from vsphere vcenter server. Figure 24. dvswitch configuration and LLDP detail For details of the dvswitch configurations, refer to the EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle White Paper. The configuration of the dvswitch does not require any changes when moving to the HA bundle. 6 A dvswitch provides a network configuration that spans all member hosts and allows virtual machines to maintain consistent network configuration as they migrate between hosts. For further information, see the VMware vsphere Networking ESXi 5.1 document listed in References. 42

43 VMware vsphere HA configuration Configuring VMware vsphere HA and VMware vsphere DRS vsphere HA provides high availability for virtual machines by pooling the virtual machines and the vsphere ESXi hosts that they reside on into a cluster. Hosts in the cluster are monitored and in the event of a failure, the virtual machines on a failed host are restarted on alternate hosts. 7 vsphere HA uses multiple vsphere ESXi hosts, configured as a cluster, to provide rapid recovery from outages and cost-effective high availability for applications running in virtual machines. vsphere HA protects application availability in the following ways: It protects against a server failure by restarting the virtual machines on other vsphere ESXi hosts within the cluster. It protects against application failure by continuously monitoring a virtual machine and resetting it in the event of guest OS failure (VM monitoring). In this bundle, both vsphere HA and DRS were enabled, as shown in Figure 25. Figure 25. vsphere HA configuration settings VM monitoring VM monitoring was configured to monitor the operating system and the application running inside the guest OS. VM monitoring was configured to restart individual virtual machines if their heartbeat is not received within the configured 60 seconds threshold. The virtual machine and application monitoring option was selected to allow Symantec 7 For further information on vsphere HA, see the VMware vsphere Availability ESXi 5.1 document. 43

44 ApplicationHA to communicate with vsphere HA to exchange virtual machine heartbeat status from the Guest OS and application running. Datastore heartbeating To meet vsphere HA requirements for datastore heartbeating, additional datastores were created on VPLEX distributed volumes and presented to all the ESXi hosts on both clusters. In a production environment, vcenter automatically selects two or more datastores for this purpose, based on host visibility. The resource cluster was configured with five datastores (VPLEX_DS1, DS2, DS3, DS4, and DS5) as described earlier, and they were used for Datastore heartbeating, as shown in Figure 26. Configuring multiple VPLEX datastores in a vmsc configuration provides higher redundancy for both datacenters. This enables vsphere HA to heartbeat to a datastore even in case of an IP link failure between datacenters, enabling vsphere HA to determine the state of a host in any failure scenario. Figure 26. vsphere HA Cluster Status heartbeat datastores VM restart options The VM Restart Priority option for the SAP virtual machines was set per virtual machine. The ABAP SAP Central Services (ASCS) and the SAP Database servers were set to High and the other SAP virtual machines were set to medium (default setting). This instructs the vsphere ESXi hosts to power on virtual machines first in the event of an outage. Figure 27 shows this setting. The Host Isolation Response setting was configured as Leave Powered On. 44

45 Figure 27. VM Restart Priority Response settings vsphere HA monitoring When you create a vsphere HA cluster, a single vsphere ESXi host is automatically elected as the master host. The master host monitors the state of all protected virtual machines and the slave hosts. When the master host cannot communicate with a slave host over the management network, it uses datastore heartbeating to determine whether the slave host has failed, resulting from a network partition or being isolated. Although vsphere HA is configured by vcenter and exchanges virtual machine state information with vsphere HA, vcenter is not involved when vsphere HA responds to a failure, so while vsphere HA, by design, respond to failures without vcenter, HA relies on vcenter to be available to configure and monitor the cluster. In this bundle, both the management cluster and resource cluster vcenter virtual machines were configured with vsphere HA enabled with datastore heartbeating. Admission Control VMware vcenter Server uses HA Admission Control to reserve resources in the cluster to provide failover protection and ensure virtual machine resource reservations. Admission Control was turned on for both resource and management clusters and the policy Percentage of Cluster Resources Reserved was selected and set to 50%. This configuration reserved cluster resources to guarantee the restart of all virtual machines in case of a simultaneous failure of two physical vsphere ESXi hosts in the same cluster. Best practices for HA clusters 8 EMC considered the following key vsphere HA cluster best practices in the bundle to ensure optimal performance: Configure alarms to monitor cluster changes to expedite incident management 8 For further information, see VMware vsphere Availability ESXi 5.1 and VMware vsphere High Availability Deployment Best Practices. 45

46 Configure alarms in VMware vcops to be triggered when vsphere HA takes action to maintain availability and send alerts, such as s, to administrators for facilitating troubleshooting. For further information about how to configure alarms in VMware vcops, refer to EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle White Paper. Monitor cluster validity to guarantee that reserved capacity for failover is available A valid cluster is a cluster in which the admission control policy was not violated. A cluster enabled for vsphere HA becomes invalid when the number of virtual machines powered on exceeds the failover requirement that is either the current failover capacity is smaller than configured failover capacity or when ESXi hosts fail, reducing the available capacity for reservation. Enable Admission Control and select the Percentage of Cluster Resources reserved policy This policy offers the most flexibility in terms of host and virtual machine sizing. Choose a percentage for CPU and memory that reflects the number of host failures that you want to support. This policy uses the actual reservation per virtual machine instead of taking the worst scenario like the other two policies available in Admission Control. The cluster dynamically adjusts when resources are added. Size all the cluster hosts equally For the Percentage of Cluster Resources Reserved policy, an unbalanced cluster results in excess capacity being reserved to handle failures, since vsphere HA reserves capacity for the largest hosts. Mask datastores in a cluster basis (all vsphere ESXi hosts in the cluster) Maximize the chance of restarting virtual machines after a failure by masking the datastores to all vsphere ESXi hosts that are part of the cluster. The vsphere HA master host will be able to access all the datastores to try to communicate to all vsphere ESXi hosts in the cluster to determine their statusif the management network is not available after the failure. To enable better handling of all-paths-down scenarios, the value of the advanced setting das.maskcleanshutdownenabled was set to true. This setting allows vsphere HA to trigger a restart response for a virtual machine that has been shut down automatically due to a power outage condition. This setting is not enabled by default. Configure vsphere DRS to support vsphere HA in highly utilized clusters Combine vsphere HA and DRS to protect against failures and to provide load balancing across hosts within a cluster. In a failure scenario, if vsphere HA cannot restart some virtual machines, it asks DRS to try to defragment resources to offer HA another opportunity to restart virtual machines. In order to achieve this, DRS needs to be enabled and configured to be fully automated. 46

47 Best practices for networking 9 The best practices for the configuration of host NICs and network topology for vsphere HA include recommendations for vsphere ESXi hosts, cabling, switches, routers, and firewalls as follows. Suspend the Host Monitoring feature when making network changes vsphere HA uses the management network to send/receive heartbeats to/from the clustered ESXi hosts. During network maintenance, heartbeat interruptions may happen and vsphere HA may trigger undesired attempts to failover virtual machines, reducing availability. Reconfigure vsphere HA on all cluster hosts after ESXi network changes vsphere HA will re-inspect the network information, and then re-enable Host Monitoring. Notify the vsphere HA Admin in advance before any network maintenance Networking is a vital component of vsphere HA. If any network maintenance must be performed, the vsphere HA administrator must be informed in advance to prepare the environment to ignore false positives. Use the das.isolationaddress advanced attribute to add additional isolation addresses for additional networks A network isolation address is an IP address that is pinged to determine whether a host is isolated from the management network (VMkernel). This IP address is pinged when a host stops receiving heartbeats from other hosts in the cluster. If the host can ping its isolation address, the host is not network isolated. If the host cannot ping the isolation address, the host has likely became isolated from the network. Implement network redundancy at the NIC level with NIC teaming connecting to separate switches A single management network can result in failovers although only the network has failed. Redundant management networking allows the reliable detection of failures and prevents isolation or partition conditions from occurring, because heartbeats can be sent over multiple networks. Network heartbeating is the primary method to determine the state of a vsphere ESXi host, providing a resilient management network to enable a faster and proper host state determination, without involving datastore heartbeating. Configure the fewest number of segments between clustered vsphere ESXi hosts This recommendation is intended to limit the network SPOF. Routes with too many hops can cause networking packet delays for heartbeats, and increase the possible points of failure. 9 For further information, see VMware vsphere Availability ESXi 5.1 and VMware vsphere High Availability Deployment Best Practices. 47

48 Configuring VMware vsphere DRS vsphere DRS host groups and virtual machine groups DRS host groups and virtual machine groups simplify management of the vsphere ESXi host resources. In this bundle, the host groups and virtual machine groups were created for the resource cluster as shown in Figure 28. Figure 28. Creating host groups and virtual machine groups Table 8 and Table 9 show the DRS groups created for the solution and their assignments. Table 8. Management cluster DRS Group Name Group Type Type Group members Datacenter_A_Servers Host DRS Group vsphere ESXi hosts r710a, r710b Datacenter_B_Servers Host DRS Group vsphere ESXi hosts r710c, r710d Datacenter_A_VMs VM DRS Group DNS virtual machine DNS server Datacenter_B_VMs VM DRS Group DNS2 virtual machine DNS slave server Cross_Datacenter_VMs VM DRS Group vcloud Director and vcenter virtual machines vcloud Director, vcentermc, vcenterrc, vchargeback, SymantecHA, and so on The group members, vcentermc and vcenterrc respectively stand for management cluster vcenter and resource cluster vcenter. The group members r710a, r710b, r710c, and r710d, respectively stand for vsphere ESXi hosts r710a and r710b (both physically located on Datacenter A), and vsphere ESXi hosts r710c and r710d (both physically located on Datacenter B). 48

49 Table 9. Resource cluster DRS Group Name Group Type Type Group members Datacenter_A_Servers Host DRS Group ESXi hosts c460-1 and c460-3 Datacenter_B_Servers Host DRS Group ESXi hosts c460-2 and c460-4 Datacenter_A_VMs Virtual machine DRS Group vshield virtual machine Datacenter_B_VMs Virtual machine DRS Group vshield virtual machine Cross_Datacenter_VMs Virtual machine DRS Group SAP ERP 6.0 virtual machines vse-edge_orgb_ha2-0 vse-edge_orgb_ha2-1 SAPERPDB, SAPERPASCS, SAPERPPAS, SAPERPAAS1, SAPERPAAS2, SAPERPAAS3 AND SAPQAS Note: The DRS groups identify which vsphere ESXi hosts are in which physical datacenters. The group members c460-1, c460-3, c460-2, and c460-4, respectively stand for vsphere ESXi hosts c460-1 and c460-3 (both physically located on Datacenter A), vsphere ESXi hosts c460-2 and c460-4 (both physically located on Datacenter B). The group members SAPERPDB, SAPERPASCS, SAPERPPAS, SAPERPAAS1, SAPERPAAS2, SAPERPAAS3, and SAPQAS, respectively stand for SAP ERP Database server, SAP ERP ABAP SAP Central Services (ASCS), SAP ERP Primary Application Server (PAS), SAP ERP Additional Application Server 1, SAP ERP Additional Application Server 2, SAP ERP Additional Application Server 3, and SAP ERP Quality Assurance system. vsphere DRS affinity rules DRS uses affinity rules to control the placement of virtual machines on hosts within a cluster. DRS provides two types of affinity rules: A VM-Host affinity rule specifies an affinity relationship between a group of virtual machines and a group of hosts. VM-Host affinity rule controls two vshield Edge gateway virtual machines running on different datacenters, as shown in Figure 29. Figure 29. VM-Host affinity rule for resource cluster 49

50 A VM-VM affinity rule specifies whether particular virtual machines should run on the same host or be kept on separate vsphere ESXi hosts. In this solution, both vcenter servers and SQL server virtual machines were configured to run on the same vsphere ESXi hosts in the management cluster, as shown in Figure 30. Figure 30. vsphere DRS affinity rules for management cluster A rule was created to keep the SAP DB and ASCS instances from running on the same host, as shown in Figure 31. This is because both systems are SPOFs and must be protected against ESXi host failures. Additional rules were created to keep pairs of SAP Application Servers running on different ESXi hosts to provide redundancy of SAP work processes. The vshield Edge virtual machines protecting the resource cluster have two redundant virtual machines that should run respectively on Datacenter A and Datacenter B. The rules vshield_edge_datacenter_x, where X can be A or B, were created to instruct vsphere DRS to enforce that requirement. Figure 31. vsphere DRS affinity rules for resource cluster 50

51 Table 10 lists all the vsphere DRS affinity rules consolidated for the management and resource clusters. vcentermc and vcenterrc stands for vcenter for the management cluster and for the resource cluster respectively. Table 10. Consolidated vsphere DRS affinity and anti-affinity rules Cluster DRS rule names DRS Affinity Rules configured Rule type Object Management Datacenter A and B DNS1 and DNS 2 servers Separate Datacenters vcloud vcloud Director and vcenterrc Separate ESXi hosts vcenter SQL, vcentermc and vcenterrc Same ESXi hosts Resource vshield Edge Datacenter A vshield Edge 0 Should run Datacenter A vshield Edge Datacenter B vshield Edge 1 Should run Datacenter B vshield Edge Anti Affinity vshield Edge VMs 0 and 1 Separate ESXi hosts SAP Application Server HA SAP PAS and AAS1 Separate ESXi hosts SAP Application Server HA2 SAP AAS2 and AAS3 Separate ESXi hosts SAP Database ASCS ESXi Failure SAP DB and ASCS Separate ESXi hosts vsphere DRS rules are created to protect virtual machines against vsphere ESXi host failures on both management and resource clusters, as shown in Table 10. The SAP PAS and AAS1 are in the same rule, which instructs vsphere DRS to keep them running in separate vsphere ESXi hosts. However the SAP PAS and SAP AAS2 or AAS3 could run in the same vsphere ESXi host, giving flexibility for vsphere DRS to provide the availability and the protection level required at the same time. EMC VSI and VPLEX EMC VSI provides enhanced visibility into VPLEX directly from the vcenter GUI. The Storage Viewer and Path Management features are accessible through the EMC VSI tab, as shown in Figure 32. In this solution, VPLEX distributed volumes host the VPLEX_MGMT_DS1 and VPLEX_MGMT_DS2 Virtual Machine File System (VMFS) datastore, and Storage Viewer provides details of the datastore s virtual volumes, storage volumes, and paths. LUNs that make up the datastore are 4 TB distributed RAID 1 VPLEX Metro volumes that are accessible via PowerPath, as shown in Figure 32. Figure 32. VSI Storage Viewer datastores 51

52 Symantec ApplicationHA Symantec ApplicationHA adds a layer of application awareness to the core vsphere HA functionality offered by VMware virtualization technology. Key benefits include the following: Out of the box integration with the VMware vsphere vcenter and vsphere HA Full visibility and control over applications with the ability to start, stop, and monitor applications running inside virtual machines Standardized way to manage applications using a single interface that is integrated with VMware vsphere Client Specialized Application Maintenance mode, in which ApplicationHA allows you to intentionally take an application out of its execution for maintenance or troubleshooting Symantec ApplicationHA components Symantec ApplicationHA consists of the following components in a VMware virtualization environment: Symantec ApplicationHA Console The ApplicationHA Console is installed separately in the Symantec ApplicationHA monitoring environment and resides on a separate virtual machine. Symantec ApplicationHA guest components for virtual machines The Symantec ApplicationHA guest components are installed separately on the virtual machines where you want to monitor applications. The guest components include the configuration wizard and the ApplicationHA agents that are used for configuring and monitoring applications. The guest components also include the Veritas Storage Foundation Messaging Service. This service communicates the application monitoring status on the virtual machine and displays it in the ApplicationHA tab. Symantec ApplicationHA agents Agents are application-specific modules that plug into the ApplicationHA framework that manages applications and resources of predefined resource types in a system. The agents are installed when you install Symantec ApplicationHA guest components. These agents start, stop, and monitor the resources configured for the applications and report state changes. If an application or its components fail, ApplicationHA restarts the application and its resources inside the virtual machine. Symantec ApplicationHA agents are classified as follows: Infrastructure agents Agents such as NIC, IP, and Mount are classified as infrastructure agents. Infrastructure agents are automatically installed as part of the ApplicationHA installation on virtual machines. Application agents The ApplicationHA agent pack is released on a quarterly basis. The agent pack includes support for new applications as well as fixes and enhancements to 52

53 existing agents. You can install the agent pack on an existing ApplicationHA guest components installation. Refer to the Symantec Operations Readiness Tools (SORT) website for information on the latest agent pack availability. Refer to the agent-specific configuration guide for more details about the application agents. Working with vsphere vcenter Server Symantec ApplicationHA communicates with vsphere HA. ApplicationHA conveys the application health status in the form of an application heartbeat. This allows vsphere HA to automatically reset or restart a virtual machine if the application heartbeat is not received within a specified interval. Figure 33 displays the sample deployment of Symantec ApplicationHA. Figure 33. Sample deployment of Symantec ApplicationHA ApplicationHA provides a vcenter plug-in for integration with vsphere Client and adds the following interfaces to perform application monitoring tasks. The following interfaces appear in the vsphere Client after you install ApplicationHA Console: ApplicationHA: The ApplicationHA tab as shown in Figure 34 is the primary interface for performing the application monitoring operations on a virtual machine. You configure application monitoring and then monitor and control the configured application on the virtual machines. After configuring application monitoring, the Symantec ApplicationHA tab displays the state of the application and the component dependencies. 53

54 Figure 34. ApplicationHA tab ApplicationHA dashboard: The ApplicationHA dashboard is the primary interface for administering the configured applications on a VMware environment. After configuring application monitoring, the ApplicationHA dashboard displays the state of the application. Figure 35 displays the ApplicationHA dashboard with the SAP and Oracle applications configured for monitoring the SAP systems. System administrators with the correct permissions can coordinate the start and stop of SAP services from the dashboard if required, as shown in Figure 35. Figure 35. ApplicationHA dashboard 54

55 In this solution, Symantec ApplicationHA is transparent to vcloud Director and is an optional component that adds important application-awareness to the vsphere Administrator for operations, as well as providing fast, reliable, and automated restart of SAP and database services running inside the SAP virtual machines during process, virtual machine, and vsphere ESXi hosts tested outages. Symantec ApplicationHA provides support for several OS, DB and application combinations described in the compatibility lists referred in the Symantec Operations Readiness Tools (SORT) website. Symantec ApplicationHA Console server was installed in a VMware virtual machine named SymantecHA, which was deployed in the management cluster and protected by vsphere HA with a high restart priority configured. According to Symantec ApplicationHA architecture, the guest agents and their updates are deployed in the Symantec ApplicationHA Console server and distributed by pushing technology to the guest virtual machines where they are installed or updated. The Symantec ApplicationHA guest components were installed on all SAP virtual machines in the resource cluster. SAP and Oracle agents were configured to manage the automatic restart of the SAP and database services detected in a running state during the initial installation/activation of the agents. Symantec ApplicationHA agents automatically restart the SAP and database services configured inside the Guest OS of the SAP virtual machines after a virtual machine outage or restart. 55

56 SAP system architecture Introduction This section describes the SAP system architecture deployed for the solution in the two datacenters. The SAP application layer uses these SAP components: SAP ERP 6.0 Core IDES Enhancement Package 5 SAP NetWeaver 7.02 SAP ABAP SAP Central Services (ASCS) SAP Database Service (DB) SAP Primary Application Server (PAS) SAP Additional Application Servers (AAS) SAP Global File systems (/sapmnt and /usr/sap/trans) All the SAP systems were installed and configured in a distributed architecture where we had separate virtual machines for the SAP central services, the database, and application servers. All SAP instances were installed on VMware vsphere virtual machines with SAP Applications running inside of them. Note: Other operating systems and database combinations supported by the SAP Product Availability Matrix (PAM) and the EMC Solutions Support Matrix (SSM) can also be used with this solution. Configuring SAP system The solution implemented a recommended distributed SAP system architecture, as shown in Figure 36. Figure 36. SAP system architecture 56

57 The enqueue server and message server are decoupled from the Central Instance and implemented as standalone services within the ASCS instance 10. Four SAP Application Servers (PAS and AAS) (formerly known as dialog instances) were installed to provide redundant work processes such as dialog (DIA), background (BGD), update (UPD), spool (SPO), and gateway, to provide workload balance and protection against vsphere ESXi host or virtual machine failures. Outage impact The SAP ASCS, SAP Database, and the SAP Global File systems (NFS) server are SPOF components in an SAP NetWeaver distributed architecture without the traditional OS/DB specific and complex clustering technologies until recently. This solution addresses these SPOFs as described in the Key design considerations section. SAP Database Server outage The whole SAP system is stopped in case of the SAP Database server component failure. Once the database becomes available again, the SAP work processes in the SAP Application Servers reconnect automatically and the users can resume their work. All transactions currently in progress and not committed are rolled back for consistency. SAP ASCS instance outage (enqueue and message server services) In the case of SAP ASCS instance component failure, either or both the enqueue and the message server services fail. Either way a quick restart is required to restore these services to keep the SAP system running. If the enqueue service fails the SAP application transaction locks stops. Therefore, the SAP system hangs because the applications wait for the lock or are terminated. If that happens, the SAP functional team will have to analyze the business process impact after this service is restated. If the message server stops working, new requests cannot be executed for dialog, update and enqueue server, but the existing connections are not affected. SAP Global File systems (/sapmnt, /usr/sap/trans, and /usr/sap/<sid>) In the case of SAP Global File systems (NFS server) component failure, the NFS shares /sapmnt/<sid>, /usr/sap/<sid>, and the /usr/sap/trans are not available. The /sapmnt directory not being available prevents additional SAP application servers to be started, but the already active ones continue to run. If the global file system /usr/sap/<sid>/sys/global/ is not available, batch and spool processes cannot write their logs there, so batch and print job activities will be canceled. If the /usr/sap/trans transport directories are not available, the change and transport system cannot be used. Key design considerations The SAP system deployed for this solution implements these key design features: The Database and ASCS instances were installed with virtual hostnames to decouple them from the fixed virtual machine hostnames and IPs. 10 The enqueue server manages logical locks, its objective being to minimize the duration of a database lock. Unlike database locks, an SAP lock can exist across several database LUWs. The message server informs all servers (instances) in an SAP system of the existence of the other servers. Other clients (for example, SAPlogon and RFC clients with load balancing) can also contact it for information about load balancing. 57

58 SAP patches, parameters, basis settings, and load balancing settings are all installed and configured according to the SAP installation guide and the SAP Notes listed in the References section. VMware best practices for SAP were adopted in this solution 11. The SAP update processes (UPD/UP2) were configured on the primary and additional application server instances. The SAP ASCS instance profile, start profiles, and dialog instance profiles were configured appropriately. The SAP ASCS instance also holds the SAP Global File systems /sapmnt, /usr/sap/<sid>, and /usr/sap/trans central directories, which are maintained by a central NFS server that restarts automatically in case of a virtual machine outage. The SAP Database server running Oracle was configured according to the SAP notes listed in the References section. SAP shared file systems, including /sapmnt/<sid> (available to all SAP instances) and /usr/sap/<sid> (available to SAP nodes and ASCS instance were stored on the ASCS instance server and mounted automatically as Network File System (NFS) shares on the other SAP virtual machines. All virtual disks in the SAP virtual machines were created in the thick format (eagerzeroed). Multiple virtual disks were configured to separate the I/O pattern in different vmdks and in different paravirtual SCSI (PVSCSI) controllers for example, SCSI Controller 1 Oracle Redo Logs and SCSI Controller 2 Oracle Datafiles and so on. In the SAP Application servers two vmdks were created with its own SCSI controllers, for the OS and the OS Swap file for best performance as recommended by SAP notes listed in the References section. This solution uses the encapsulated and virtualized storage for the entire SAP environment. Each SAP virtual machine was stored in one VPLEX-based datastore as shown in Table 11 on page 59, as required by vcloud Director. Separate datastores were used for high availability purposes. The datastores are based on VPLEX virtual volumes distributed across the both datacenters and presented to the vsphere ESXi hosts running the SAP virtual machines through EMC VPLEX Metro. All the SAP virtual machines in the solution were configured to be protected by Symantec ApplicationHA and VMware vsphere HA with restart priorities set, vsphere DRS with specific restart priorities, vsphere DRS with affinity and antiaffinity rules as described in Table 10. The DRS anti-affinity rules were created to protect the SAP virtual machines against vsphere ESXi host failures that trigger a virtual machine restart, protecting the SAP systems, specially the SPOFs with fast restarts and a timely and automated recovery of the SAP services to the end users. All SAP virtual machines had VMware tools installed and were configured with vmxnet3 network cards for best network performance. 11 For details about SAP Solutions on VMware vsphere, refer to High Availability and Best Practices Guide. 58

59 All SAP system EP1 production-role virtual machines memory reservation was set to the maximum size to avoid virtual machine paging and optimize for best performance, since SAP allocates memory permanently and does not release it again. Table 11 shows the configuration of the SAP virtual machines. DC stands for datacenter. Table 11. SAP virtual machines configuration DC ESXi Virtual machine role vsphere HA restart priority vcpus Memory (GB) Disk (GB) Virtual machine name Datastore name 12 B C460-4 SAP QAS Low SAPQAS VPLEX_DS3 A C460-1 SAP DB High SAPERPDB VPLEX_DS5 B C460-2 SAP ASCS High SAPERPASCS VPLEX_DS1 A C460-1 SAP PAS Medium SAPERPPAS VPLEX_DS2 A C SAPERPAAS1 VPLEX_DS3 B C460-2 SAP AAS Medium SAPERPAAS2 VPLEX_DS4 B C SAPERPAAS3 VPLEX_DS1 12 VMware vcloud Director consumes storage from the configured storage profiles based on capacity requirements. For demonstration purposes, the SAP virtual machines were placed on HA Gold Tier (EFDs based) specific datastores as shown in Table

60 EMC storage infrastructure Introduction Overview This section describes the storage infrastructure for the solution: Symmetrix VMAX arrays are the storage platform at both datacenters. Two storage arrays are deployed with a matching LUN configuration. EMC Symmetrix VMAX EMC Symmetrix VMAX is a high-end storage array based on Intel Xeon processors and optimized for the virtual datacenter. Built on the strategy of simple, intelligent, modular storage, Symmetrix VMAX incorporates a highly scalable Virtual Matrix Architecture that enables it to grow seamlessly and cost-effectively from an entry-level configuration into the very large storage system. EMC VMAX supports flash, SAS, and NL-SAS drives within a single array, and an extensive range of RAID types. EMC FAST VP automates tiered storage strategies. The EMC Enginuity operating environment provides the intelligence that controls all components in a VMAX array. Symmetrix HA Symmetrix systems ensure high availability by combining a simplified suite of redundant hardware components with continuous system monitoring. Symmetrix systems proactively diagnose problems and automatically activate repair, thereby preventing many failures before they happen. Occurred failures are often fixed transparently, without disrupting business operations or incurring downtime. A Symmetrix system provides redundant hardware that allows the system to remain operational when a component fails and also during subsequent component repair. Redundant hardware components include the following: A maximum of eight VMAX Engines Physical memory Multiple direct or switched host connections combined with EMC PowerPath software for failover, load balancing, and recovery Fully independent FC loops, switch-on-loop topology, and dual point-to-point connections to each drive Redundant virtual matrix components and connections Redundant power systems providing support for two power zones (A and B) that connect to two dedicated AC power lines with battery backup to vault cache if AC power fails. 60

61 Configuring Symmetrix VMAX Storage layout For this solution, all storage is pooled into Symmetrix VP Pools. Thin devices (TDEVs) are bound to the Symmetrix pools and presented to the VPLEX array through masking views. Standard TDEV Volume sizes of 2TB and 4TB are presented from the VMAX storage array to the VPLEX storage. Both VMAX at Datacenter A and B have similar configurations. We used Unisphere Storage Templates to create the VMAX LUNs. This enables storage administrators to quickly and easily create standard volumes. Figure 37 shows an example of the Unisphere storage template for creating a 4-TB Meta volume. We configured the details of the Meta volume under Advanced Options, which is a one-time effort that can be reused multiple times. Figure 37. Storage Template Details 61

62 Workload generation SAP standard SD Benchmark An SAP Standard Application Benchmark, the SAP ERP Sales and Distribution (SD) Benchmark, was used to generate a significant sample SAP workload on the SAP ERP 6.0 EhP 5 system EP1 built for the test scenarios. The objective of adding an SAP-specific workload to two test scenarios in Testing and validation is to demonstrate that even under workload, this solution provides consistent uptime without compromising the SAP system s availability. The SAP Standard SD Benchmark toolkit performed predefined standard SD transactions against the SAP EP1 system. The toolkit covers a sell-from-stock business scenario, which includes the creation of a customer sales order with five line items and its corresponding delivery with subsequent goods movement and invoicing. The scenario consists of the following transactions: Creating an order with five line items (VA01) Creating a delivery for the order (VL01N) Displaying the customer order (VA03) Changing the delivery (VL02N) and post a goods issue Listing 40 orders for one sold-to party (VA05) Creating an invoice (VF01) SAP Standard SD Benchmark configuration SAP SD Benchmark Driver was installed on a dedicated virtual machine and ran the workload from a central location to facilitate monitoring. To avoid data-locking situations, each benchmark user had its own master data. Table 12 presents the SD Benchmark configuration, which is a typical user activity for customers. Table 12. SD Benchmark configuration Users per app server Ramp-up time Think time Loops per user High-load phase run time Concurrent users minutes 5 seconds 5 60 minutes 1, Benchmark run Sales orders/min A benchmark run consists of a ramp-up phase where all users log on one after another, a high-load phase where all users run their actions concurrently, and a ramp-down phase where all users log off one after another. Figure 38 shows an example of the Benchmark run. Figure 38. Benchmark run example 62

63 Testing and validation Introduction The EMC validation team initially installed and validated the environment without any high-availability protection scheme. We then transformed the environment to the mission-critical business continuity HA and application mobility solution described earlier in this white paper. We carried out the following tests to validate different scenarios addressed in this solution and demonstrate the level of protection provided to the lab environment built. The testing and validation was divided between planned downtime and unplanned downtime test scenarios and from the less to the most disruptive, in order to demonstrate different benefits of this solution. Planned downtime a. vsphere ESXi host maintenance Unplanned downtime b. VMware virtual machine guest OS process failure with Symantec ApplicationHA c. vsphere ESXi host hardware failure with Symantec ApplicationHA d. vsphere ESXi hosts failure e. Datacenter failure f. EMC VPLEX isolation failure with SAP workload Planned downtime Planned downtime always can be controlled. Reasons for planned downtime include hardware maintenance, hypervisor maintenance, installation of patches, upgrades of the server BIOS, and drivers or the hypervisor itself. All these activities can cause downtime in your environment. However, you can plan when to perform these activities, and choose the most appropriate time. This solution provides a value for Downtime Avoidance that allows SAP or other workloads to be non-disruptively moved across datacenters when a foreseeable downtime results from an external event, such as a power upgrade planned by the utility company, for example. a. Test scenario vsphere ESXi host maintenance This test scenario validates that: No disruption occurs in the SAP application running inside the VMware virtual machines, under a SAP Benchmark sample workload, running on a vsphere ESXi host that requires a planned hardware or software maintenance. The EMC VPLEX Metro, vsphere HA, and vsphere DRS configurations described in earlier in this white paper ensure high protection levels during physical server maintenance. The SAP workload described in Workload generation was executed in this test scenario. 63

64 Objectives Verify that an ESXi host can be set to maintenance mode without interrupting the SAP application running in its virtual machines. Verify that the SAP virtual machines, which run under SAP workload on the ESXi host set on maintenance mode and located on Datacenter A, are non-disruptively moved across datacenters using the VMware vmotion and EMC VPLEX Metro. Verify that the vsphere DRS affinity and anti-affinity rules are enforced by vcenter during the non-disruptive migration of the SAP virtual machines from Datacenter A to Datacenter B. Testing procedure 1. Identify in which ESXi hosts the SAP EP1 system virtual machines were running. 2. Start the SAP Standard SD Benchmark run and wait for the High Load phase. 3. Select the ESXi host c460-1, where the SAP EP1 Database virtual machine was running and enter in maintenance mode 4. Verify the non-disruptive migration of the SAP virtual machines from the ESXi host c460-1 to the other ESXi hosts on Datacenter A and B 5. Verify that the SAP system EP1 was not interrupted. 6. Verify that all vsphere DRS affinity and anti-affinity rules were followed. Results Figure 39 summarizes the test environment and provides reference duration metrics in seconds obtained during the testing. The metric used was the Migration Time 13. Figure 39. Test environment 13 Migration Time is the total time taken for the migration to complete, beginning from the initiation of the migration. For more information refer to VMware vsphere 5.1 vmotion Architecture, Performance and Best Practices Technical White Paper. 64

65 The SAP servers were moved across two datacenters physically apart, but the SAP end users never experienced any service interruption, as verified in the SAP system logs (SM21 transaction) of all SAP Application Servers. Analysis The non-disruptive migration in three virtual machines, two SAP virtual machines (SAP ERP DB and SAP ERP AAS1) and the vshield Edge0, was completed on average in 48 seconds after the c460-1 ESXi host was entered in maintenance mode. The same test was repeated three times in the same conditions. The least favorable result was 50 seconds. This test scenario demonstrates that: A planned maintenance can be performed non-disruptively and on demand without affecting the planned downtime commitments on your current SLAs. vsphere DRS automatically enforced the affinity rules configured to keep the SAP virtual machines ASCS, Database, and AAS1 and PAS Instances in separate physical vsphere ESXi hosts, maintaining the same protection level that existed before the first server was set to enter in maintenance mode. An administrator can proactively avoid the impact of a foreseeable outage, to preserve the availability of the business critical SAP applications, moving them live and within seconds to a remote datacenter, outside the area to be affected by the foreseeable outage. Unplanned downtime Unplanned downtime, which cannot be controlled or anticipated, can result from occurrences such as hardware failure, software failure, datacenter failure, and natural disaster. HA is a term used often connected to unplanned downtime. HA is a complex topic, which covers designing a solution to make not only your software highly available, but the whole IT infrastructure underneath in all layers (networks, routers, power supply, storage and so on) must be designed with high availability in mind. b. Test scenario VMware virtual machine guest OS process failure with Symantec ApplicationHA This test scenario validates that an SAP system Guest OS processes running inside virtual machines can be restarted quickly and automatically by Symantec ApplicationHA alone or in combination with VMware vsphere HA after a process failure. No SAP workload was applied in the SAP system EP1 Application Servers during the test. Objectives Verify that Symantec ApplicationHA detects and restarts the SAP database processes inside a virtual machine after an OS triggered process failure. Verify that the SAP database services (Oracle listener and Oracle database) running inside the SAP virtual machine are restarted automatically by Symantec ApplicationHA agents. Verify that Symantec ApplicationHA agent will take a corrective action coordinated with vsphere HA to try to restore the SAP database services (Oracle listener and Oracle database) when attempts of the service restart are not successful. 65

66 Testing procedure The testing process below was executed three times in the same scenario to validate the results obtained. This process was executed with three variations: single process failure, and persisting process failure with and without graceful shutdown, which are described as follows. 1. Identify the SAP Database Instance specific Oracle processes running and collect the process ID of the oracle_pmon process. 2. Verify the status of the SAP Database instance from the vsphere Client Symantec ApplicationHA tab. 3. Execute an OS command line on the Guest OS of the virtual machine where the SAP ERP DB (SAP system EP1 Database server) is running to kill the oracle_pmon process. 4. Verify that Symantec ApplicationHA agent detects the process outage and take actions to automatically restart the Oracle database processes in a timely manner without any manual intervention. 5. Verify that the SAP EP1 database server processes (Oracle listener and Oracle database) are restarted automatically and that the SAP EP1 system Application servers can reconnect to the SAP EP1 system Database instance. 6. Execute step 3 again and verify the actions taken by Symantec ApplicationHA agents to restore the SAP Database services and try to minimize the downtime 14. Results Single process failure testing variation Figure 40 summarizes the single SAP Database process failure set of tests performed and provides reference duration metrics obtained during the testing. The metrics are shown in minutes and seconds (mm:ss). Figure 40. Single SAP database instance (oracle_pmon) process failure The columns show the duration values obtained during the three testing cycles. Figure 40 depicts the durations of Oracle database restarting and SAP workprocesses reconnecting across the three tests. The values from the column Oracle Restart (mm:ss) were obtained from the Symantec ApplicationHA cluster log file (located at/var/vrtsvcs/log/engine_a.log), which 14 This step is applicable only for the variations Persisting Process Failure Testing with and without Graceful Shutdown, since the Guest OS shutdown is only invoked after the number of process restart attempts cross the value configured in the Symantec ApplicationHA parameter App.RestartAttempts 66

67 contains all the activities executed by the Symantec ApplicationHA cluster after the process failure happened. This log file lists the detailed actions taken and also lists the output of database instance restart. Symantec ApplicationHA automatically executed these processes after a process failure: 1. Detect the process failure 2. Cleanup all other dependent Oracle processes 3. Restart the Oracle EP1 Database instance The total duration of the execution of the above three processes was 50 seconds on average, considering all times obtained in all three tests cycles performed. The SAP EP1 system Application Servers took on average 2 min and 28 seconds, according to the SAP System Log to reconnect their work process to the SAP EP1 system database (***LOG BYY=> work process left reconnect state ) after manually initiating database failure. The Symantec ApplicationHA agent for Oracle was configured to allow two restart attempts (App.RestartAttempts) to allow a single process failure testing without invoking vsphere HA to restart the virtual machine. Persisting process failure testing with/without graceful shutdown Figure 41and Figure 42 provide reference duration metrics obtained during the testing, and summarize the persisting SAP Database process failure with/without graceful shutdown set of tests performed. The metrics are shown in in minutes and seconds (mm:ss). Figure 41. Persisting SAP database (oracle_pmon) process failure with graceful shutdown Figure 42. Persisting SAP database (oracle_pmon) process failure without graceful shutdown Figure 41and Figure 42 describe the durations of persisting process failure with OS restart and SAP workprocess reconnection across three tests. 67

68 Symantec ApplicationHA automatically executed the steps below after the manual trigger of the first process failure: 1. Detect the process failure 2. Cleanup all other dependent Oracle processes 3. Restart the Oracle EP1 Database instance Symantec ApplicationHA executed automatically the steps below after the manual trigger of the second process failure: 4. Detect the process failure 5. Cleanup all other dependent Oracle processes 6. Stop Oracle listener service 7. Trigger a graceful shutdown of the Guest OS (step skipped in the non-graceful shutdown scenario) 8. Stop the virtual machine application heartbeat to notify vsphere HA to trigger a cold restart of the virtual machine To validate the recovery process workflow, both test cycles were performed in the exact same conditions. For the scenario with the graceful shutdown, the duration of 4 min and 25 seconds on average was observed to execute the eight processes described above and the five processes below. For the scenario without the graceful shutdown, the duration of 3 min and 33 seconds on average was observed to execute all seven processes described above and the five processes below. Restart vsphere HA virtual machine Restart Guest OS Start Symantec ApplicationHA cluster and agents Start Oracle listener Start Oracle database The Symantec ApplicationHA agent for Oracle was configured to allow one restart attempt (App.RestartAttempts) to force a second process failure to invoke vsphere HA to restart the virtual machine. The Symantec ApplicationHA parameter (VM.GracefullRebootPolicy) was set to enabled to execute a graceful shutdown of the operating system, before invoking vsphere HA to restart the virtual machine in the graceful OS shutdown variant test scenario. The Symantec ApplicationHA parameter (VM.GracefullRebootPolicy) was set to disabled to avoid the shutdown of the operating system, before invoking vsphere HA to restart the virtual machine in the without-graceful OS shutdown variant test scenario. Note: Symantec ApplicationHA did stop all the referred Oracle processes in the correct sequence, before invoking vsphere HA to restart the virtual machine independent of the value set for the parameter VM.GracefullRebootPolicy, protecting the Oracle services even in an inevitable virtual machine restart without graceful OS shutdown test scenario. 68

69 For more information about the SAP NetWeaver database reconnect mechanism, refer to the SAP Note Database Reconnect: Architecture and function listed in References section. Analysis Figure 43 summarizes the SAP Database instance process failures durations obtained in the different scenarios performed and provides reference duration metrics obtained during the testing for comparison reasons. The metrics are shown in in minutes and seconds (mm:ss). Figure 43. Persisting SAP database (oracle_pmon) process failure without graceful shutdown This test scenario demonstrated the following: Symantec ApplicationHA combined with vsphere HA restarted the failed SAP Database Instance quickly, automatically and without any manual intervention of the SAP administrator or DBA, even after a persisting SAP critical database instance process failure. Using Symantec Application HA agents to recover from a single process failure inside the Guest OS of a virtual machine is faster than executing a Guest OS shutdown and then subsequently restarting the virtual machine, due the wait required for the Guest OS to load and the applications services to restart correctly and resume the SAP services. The SAP WP Reconnect time difference between the graceful and non-graceful Guest OS shutdown durations obtained during the tests after a persistent process failure is only 12%. The difference to restart is minimal; therefore the risk brought by a non-graceful OS shutdown outweighs the time savings to restore the SAP services faster in the event of a process outage inside the Guest OS. Symantec ApplicationHA agents restarted the SAP database services (Oracle listener and Oracle EP1 database) cleanly, quickly, and automatically, recovering in a timely manner from a severe incident, minimizing the total downtime incurred to the SAP EP1 system. This significantly reduces the mission-critical SAP application downtime and potentially reduces the costs associated with the downtime. Symantec ApplicationHA provides application monitoring, adding a new level of resiliency, integrating the virtualization layer with the application layer. Symantec ApplicationHA demonstrates that the application services were restarted in the correct order, avoiding the common problems associated with manual restarts and maintenance of SLAs. Symantec ApplicationHA provides the automation required to start SAP application and database services in case of outages, eliminating the need for the creation and maintenance of custom OS-based scripts to start the SAP services. 69

70 c. Test scenario vsphere ESXi host hardware failure with Symantec ApplicationHA This test scenario validates that the SAP database services running inside the guest OS of a virtual machine can be restarted quickly and correctly in an automated manner using a combination of the vsphere HA, vsphere DRS, and Symantec ApplicationHA technologies after a simulated vsphere ESXi host failure. Objectives Verify that vsphere HA will restart the SAP database virtual machine in surviving ESXi hosts in the resource cluster after an ESXi host simulated failure. Verify that the SAP database services (Oracle listener and Oracle database) running inside the SAP virtual machine is restarted automatically by Symantec ApplicationHA agents after the Guest OS is restarted by vsphere HA. Verify that the vsphere DRS anti-affinity rules are enforced during the restart of the SAP virtual machines. Testing procedure Execute the following steps three times in the exact same conditions to validate the results: 1. Identify vsphere ESXi hosts in which each SAP EP1 database and ASCS virtual machines were running. 2. Select the vsphere ESXi host where the SAP ERP DB virtual machine is running and power it off from the hardware management console. 3. Verify that vsphere HA restarts the SAP virtual machines from the ESXi host 1 over to ESXi host 2 quickly and automatically. 4. Verify that the SAP ERP DB virtual machine is restarted by vsphere HA and the Oracle services are also restarted automatically and correctly by Symantec ApplicationHA agents. 5. Verify that all vsphere DRS anti-affinity rules were enforced. 70

71 Results Figure 44 summarizes the test environment described above and provides reference duration metrics obtained during the testing. The metrics are shown in in minutes and seconds (mm:ss). Figure 44. ESXi failure with Symantec ApplicationHA vsphere HA restarted the SAP database virtual machine (SAP ERP DB) across datacenters, from the ESXi host c460-3 on Datacenter A to the ESXi host c460-4 on Datacenter B, after the simulated failure (physical server power off) in the ESXi host c460-3 on Datacenter A. The average virtual machine restart time observed was on average 43 seconds, as shown in Figure 44. Symantec ApplicationHA agent restarted the SAP database services (Oracle listener and Oracle database) inside the guest OS of the SAP ERP DB virtual machine in 3 min and 12 seconds on average, as shown in Figure 44. vsphere DRS, based on the configured affinity rules described in Table 10, identified that the SAP ERP DB virtual machine was restarted by vsphere HA on the ESXi host c460-4 where the SAP ERP ASCS virtual machine was also running and then, using VMware vmotion, non-disruptively migrated the SAP ERP ASCS virtual machine from the ESXi host c460-4 to the ESXi host c460-2 on Datacenter B in 6 seconds on average. Analysis This test scenario demonstrates the following: vsphere HA combined with Symantec ApplicationHA restarted the SAP Database services quickly, automatically, and across datacenters without any manual intervention after a severe ESXi host failure. vsphere DRS identified that affinity rules were violated having the SAP Database virtual machine and the SAP ERP ASCS virtual machine running on the same ESXi host and enforced its affinity rules triggering a VMware vmotion of the SAP ERP ASCS virtual machine to another ESXi host to restore the previous level of protection provided before the ESXi host failure, even after the failure, keeping the SAP virtual machines in separate ESXi hosts providing resiliency. Symantec ApplicationHA restarted the SAP database services (Oracle listener and Oracle EP1 database) automatically after vsphere HA restarted the operating system, recovering from the incident and minimizing the downtime incurred to the SAP system EP1. 71

72 d. Test scenario vsphere ESXi hosts failure This test scenario validates that, when an unexpected vsphere ESXi hosts failure occurred in the management cluster or the resource cluster, the virtual machines on the ESXi hosts are restarted on the surviving ESXi hosts on each cluster based on the previously described vsphere HA Restart priority and DRS affinity rules. To test this failure scenario, we powered down a server in the management cluster and in the resource cluster to simulate unexpected server hardware failures, as shown in Figure 45. Figure 45. Test scenario -vsphere ESXi hosts failure Objectives Verify that the virtual machines running on the failed ESXi hosts are restarted by vsphere HA on other surviving ESXi hosts in both management and resource clusters. Verify that the configured DRS affinity rules are followed when placing the virtual machines after the failure. Testing procedure 1. Shut down one ESXi host in both resource and management clusters to simulate hardware failure. 72

73 2. Verify that the ESXi hosts failure is detected by vsphere HA. vsphere HA automatically detects that the servers are failed and initiates a power on of the virtual machines, as shown Figure 46. Figure 46. Tasks pane in vcenter web client 3. Verify that the restart event is triggered for virtual machines running on the failed ESXi host. Figure 47 shows an example of the recent tasks pane that vsphere HA initiates restart of the virtual machines on one of the surviving ESXi hosts. Figure 47. Tasks pane in vcenter web client 4. Verify that the vsphere DRS affinity rules are followed when placing virtual machines after the failure, as configured on Table 10 in Configuring VMware vsphere DRS. Results Table 13 shows the observed behaviors of the system and reference metrics in minutes and seconds (mm:ss) when the servers failed. DC stands for datacenter. Table 13. Reference metrics for ESXi failure Before After Cluster DC ESXi host Virtual machines DC ESXi host Restart time (mm:ss) DRS rules Management cluster A r710b vcloud Director B r710c 3:26 Compliant A r710a vchargeback A r710b 3:-06 Compliant A r710a DNS1 A r710b 2:56 Compliant Resource cluster A c460-3 vshield Edge 0 A c :05 Compliant A c460-3 SAP ERP DB A c :25 Compliant A c460-3 SAP ERP AAS1 B c :20 Compliant Analysis vsphere HA automatically reacts to the vsphere ESXi host failures incidents and restarts, following the restart priorities defined, the virtual machines affected on the next available vsphere ESXi host in the same cluster, quickly restoring the services provided by the affected virtual machines. vsphere DRS, with the configured affinity and anti-affinity rules, automatically, and nondisruptively adjusts in a 5-second interval the placement of the restarted virtual machines in the surviving ESXi hosts in the respective cluster, restoring the previous high availability service level provided. 73

74 e. Test scenario datacenter failure This test scenario validates that, in the event of a complete datacenter failure, all virtual machines running on one datacenter in both management cluster and resource cluster are restarted quickly, with the correct sequence on surviving datacenter. To test this failure scenario, we simulated a complete failure of Datacenter A, including VPLEX cluster, ESXi host, and network. The VPLEX Witness remained available on Datacenter C. VPLEX cluster-2 remained on Datacenter B to communicate with the VPLEX Witness on Datacenter B, as shown in Figure 48. Figure 48. Complete failure of Datacenter A Objectives Verify that the standby vshield Edge gateway on Datacenter B can take over and resume the networking services. Verify that all virtual machines on both management and resource clusters are restarted on Datacenter B. Verify that the configured DRS affinity rules are followed and kept in compliance. Testing procedure 1. Shut down all the ESXi hosts, switches, and VPLEX cluster-1 in Datacenter A to simulate complete failure. 74

75 2. Verify that the VPLEX cluster-2 on Datacenter B remains available. 3. Execute a continuous ping command from the SAPQAS system running on Datacenter B to verify that the standby vshield Edge gateway on Datacenter B takes over the network service. 4. Verify that vsphere HA detects the ESXi hosts failure on Datacenter A. 5. On both management and resource clusters, verify that vsphere HA restarts all virtual machines running previously on Datacenter A on Datacenter B after failure of Datacenter A. 6. Verify that the vsphere DRS affinity rules are followed when placing virtual machines after the failure on Datacenter A, as configured in Table 10 in Configuring VMware vsphere DRS. Results Table 14 shows the observed behaviors of the system and reference metrics in minutes and seconds (mm:ss) after the Datacenter A failure. DC stands for datacenter. Table 14. Reference metrics after failure on Datacenter A Before After Cluster DC ESXi host Virtual machines DC ESXi host Restart time (mm:ss) DRS rules Management cluster A r710b vcloud Director B r710c 3:30 Compliant A r710a vchargeback B r710d 3:02 Compliant A r710a DNS1 DNS1 was not restarted on Datacenter B due the affinity rule, but the DNS service resumed over to DNS2 on Datacenter B. Compliant Resource cluster A c460-1 SAP ERP DB B c :35 Compliant A c460-1 vshield Edge 0 The vshield Edge 0 was not restarted on Datacenter B due an Affinity Rule, while gateway service failover to the vshield Edge 1 on Datacenter B took 4 seconds Compliant A c460-3 SAP ERP PAS B c :25 Compliant A c460-3 SAP ERP AAS2 B c :20 Compliant Analysis When Datacenter A fails, VPLEX Witness ensures that the consistency group s detach rule, which defines cluster-1 as the preferred cluster, is overridden and the storage served by VPLEX cluster-2 on Datacenter B remains available, as shown in Figure

76 Figure 49. Status of VPLEX cluster after Datacenter A failure When the vsphere ESXi hosts on Datacenter A failed, vsphere HA detected the datacenter failure and: Restarted vcloud Director and vchargeback virtual machines in the management cluster on Datacenter B. Restarted SAPERP_DB, SAPERP_AAS2, and SAPERP_PAS virtual machines in the resource cluster on Datacenter B. SAPERP_DB was restarted on a vsphere ESXi host that is different than the one with SAPERP_ASCS on Datacenter B, based on the configured DRS affinity rules described in Table 10. The SAP end users of SAPERP_PAS lost their sessions due to the ESXi host failure, but logged on again when SAPERP_DB and SAPPAS restarted on Datacenter B. On each cluster, the virtual machines failed on the Datacenter A were restarted on the surviving Datacenter B by vsphere HA. The DNS service was failed over to DNS2 on Datacenter B without interruption. The vshield Edge gateway 1 failed over to the vshield Edge gateway switch 2 in only 4 seconds, which is considerably faster than the SAP virtual machines restart, as shown in Figure 50. Figure 50. vshield Edge gateway failover f. Test scenario VPLEX cluster isolation failure with SAP workload This test scenario validates that, in the event of isolation of a VPLEX cluster, vcloud Suite, vcenter, the SAP applications, and database continue operation without interruption. The SAP workload described in Workload generation was executed in this test scenario. To test this failure scenario, we simulated isolation of the preferred cluster on Datacenter A, with both the external management IP network and the VPLEX WAN communications network partitioned. The LAG network remains available. VPLEX Witness remains available on Datacenter C. On Datacenter B, VPLEX cluster-2 remains in communication with VPLEX Witness, as shown in Figure

77 Figure 51. Datacenter A VPLEX cluster isolation Objectives Verify that all SAP virtual machines on Datacenter A continue to run non-disruptively through the EMC VPLEX counterpart in Datacenter B, through its Cross-Cluster Connect in the event of isolation of Datacenter A VPLEX cluster. Verify that the SAP workload continues to run without interruption by the EMC VPLEX storage failure. Testing procedure 1. Start the SAP Standard SD Benchmark run and monitor until it reaches the high load phase. 2. Simulate isolation of the preferred cluster on Datacenter A, with both the external management IP network and the VPLEX WAN communications network partitioned. The LAG network remains available. 3. Check PowerPath/VE status. The paths to Datacenter A VPLEX storage should be dead, but the paths to the VPLEX storage on Datacenter B should be alive and be serving the vsphere ESXi hosts on Datacenter A. 4. Verify the VPLEX functionality; check that the storage was not lost. 77

78 5. Verify the status of SAP SD Benchmark and check that the processes were not stopped or interrupted at any point in time during the storage failover to Datacenter B. Results Table 15 shows the expected and observed behaviors of the system when the VPLEX at Datacenter A was isolated. Table 15. Virtual machines on Datacenter A management cluster Virtual machines on Datacenter A resource cluster Expected and observed behaviors System name On r710a: vchargeback vcenter Resource Cluster vcenter Management Cluster ssenet (DNS server) On r710b: vcloud Director SQL server On c460-1: SAPERP_DB On c460-3: SAPERP_AAS2 SAPERP_PAS Status prior to VPLEX isolation Expected behavior Observed behavior Available Available Available Available Available Available VPLEX cluster VPLEX1 Datacenter A cluster-1 VPLEX2 Datacenter B cluster-2 VPLEX witness Available Unavailable Available Available Unavailable Available Available SAP services Database/Enqueue/Message Server Available Available Available Analysis When VPLEX on Datacenter A became isolated, the VPLEX Witness ensured that the consistency group s detach rule, which defines cluster-1 as the preferred cluster, was overridden and that the storage served by VPLEX cluster-2 on Datacenter B remained available, as shown in Figure 52. Figure 52. VPLEX status after Datacenter A VPLEX isolation 78

79 Figure 53 shows the behavior of EMC PowerPath during the isolation event. Prior to the isolation, PowerPath sets the Cross-Cluster paths on hosts at Datacenter A to auto standby proxy. When the isolation event occurs, PowerPath is able to detect the isolation of the VPLEX LUNs at Datacenter A and service I/O on the standby paths. When the cluster is recovered, PowerPath automatically recovers the dead paths. For details of VPLEX Cross-Cluster Connect, refer to VMware deployments on VPLEX Metro. Figure 53. PowerPath status from after Datacenter A VPLEX isolation Since the vsphere ESXi hosts are connected to both VPLEX clusters in each datacenter, vsphere ESXi with EMC PowerPath/VE simply re-routes the I/O to the alternate path, which is available since VPLEX is configured with a VPLEX Witness protected distributed volume. In both management cluster and resource cluster, the vsphere ESXi hosts on Datacenter A remain available and all virtual machines remain active due to the use of VPLEX Metro HA Cross-Cluster Connect. The SAP EP1 system keeps running without interruption, as shown in Figure 54. Figure 54. No interruption of SAP system after VPLEX cluster isolation on Datacenter A This test scenario demonstrates the following: Even after VPLEX isolation, the SAP SD Benchmark workload is not interrupted during the transparent failover of the storage services from the VPLEX from Datacenter A to Datacenter B. EMC PowerPath/VE identifies the failed paths and re-directs the FC traffic through the active paths, which are connected to the VPLEX storage on Datacenter B. This makes the VPLEX failure transparent to the SAP application running inside the SAP virtual machines on Datacenter A, as well as avoiding the virtual machines to be restarted on other ESXi hosts, avoiding a downtime. 79