EMC CLOUD-ENABLED INFRASTRUCTURE FOR SAP BUSINESS CONTINUITY SERIES: HIGH AVAILABILITY AND APPLICATION MOBILITY BUNDLE VNX

Transcription

1 White Paper EMC CLOUD-ENABLED INFRASTRUCTURE FOR SAP BUSINESS CONTINUITY SERIES: HIGH AVAILABILITY AND APPLICATION MOBILITY BUNDLE VNX EMC VPLEX, EMC Next-Generation VNX, VMware vcloud Suite, and VMware vsphere Metro Storage Cluster with vsphere HA and DRS Resilient mission-critical SAP deployment in a private cloud Application mobility across datacenters Active/active datacenters EMC Solutions Abstract This white paper focuses on high availability and application mobility add-on bundle of the on-premises Cloud-enabled Infrastructure for SAP. It explains the transformation of a single datacenter into a mission-critical business continuity solution with active/active datacenters. The solution is enabled by EMC VPLEX Metro, EMC next-generation VNX, VMware vcloud Suite, and VMware vsphere Metro Storage Cluster with vsphere High Availability and vsphere Distributed Resource Scheduler. December 2013

2 Copyright 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. All test results contained in this report were obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a customer can or will achieve similar results. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H

3 Table of contents Executive summary... 5 Business case... 5 Solution overview... 5 Key results... 6 Introduction... 7 Purpose... 7 Objectives... 8 Audience... 8 Terminology... 9 Technology overview Introduction Solution architecture Physical architecture Logical architecture Protection layers Hardware resources Software resources Key components Overview EMC VPLEX EMC next-generation VNX EMC PowerPath/VE EMC VSI VMware vcloud Suite Symantec ApplicationHA EMC VPLEX Metro infrastructure Introduction VPLEX Metro solution configuration VPLEX Witness configuration VMware virtual datacenter Introduction Configuring VMware vcloud Director Configuring vcenter Chargeback Manager VMware deployments on VPLEX Metro VMware stretched cluster configuration

4 VMware vsphere HA configuration Configuring VMware vsphere DRS EMC VSI and VPLEX Symantec ApplicationHA SAP system architecture Introduction Configuring SAP system Outage impact Key design considerations EMC storage infrastructure Introduction EMC VNX HA Configuring VNX Workload generation SAP standard SD Benchmark Testing and validation Introduction Planned downtime Unplanned downtime Conclusion Summary Findings References EMC VMware SAP Symantec

5 Executive summary Business case Application and infrastructure availability is a key consideration in today s modern business continuity strategies for SAP cloud infrastructures stretched across distances. In traditional infrastructure designs for SAP high availability, workloads across physical datacenters are disrupted when any of the infrastructure components fails at the same time. These traditional high availability designs are limited and usually require special combinations of software and hardware that are operating system-specific, databasespecific, or both, making them complex, inflexible, costly, and difficult to implement and maintain. Often these designs require more effort to keep them running than actually providing the required protection and availability promised in the first place. This white paper introduces an EMC solution for mission-critical SAP high availability with the following objectives: High availability, which manages risks in an IT environment and plans for recovery across distances resulting from unplanned outages. Application mobility, which enables non-disruptive movement of running SAP workloads seamlessly across servers or datacenters. The main business challenges addressed by this solution include: Protect against single points of failure (SPOFs) Minimize the impact caused by planned and unplanned downtimes Reduce the infrastructure complexity and operational costs Provide an end-to-end automated resilience model Improve the utilization of resources across datacenters Solution overview This solution is an add-on to an existing EMC Cloud-enabled Infrastructure for SAP foundation bundle, and it adds the vsphere Metro Storage Cluster (vmsc) configuration commonly referred to as a stretched cluster. VMware vmsc is a VMware certified solution, which is a combination of storage array based clustering and synchronous replication provided by EMC VPLEX Metro. The main goal of this VMware vmsc solution is to extend the high availability capability provided by the local clusters within a local datacenter to a geographically separated model with two datacenters in different locations. The architecture extends what is defined as local in terms of network and storage. This enables the network and storage subsystems to span a metropolitan area, presenting a single and common base infrastructure set of resources to the VMware vsphere cluster at both datacenters. This solution provides a new level of availability with an end-to-end, out-of-the-box, integrated, simplified, and cost-effective solution that is also application-aware. 5

6 The white paper demonstrates how the following technologies create this innovative solution: EMC VPLEX Metro Provides the virtual storage layer that enables an active/active Metro datacenter and supports continuous availability, even in the event of disruption at one of the datacenters. VMware vcloud Director Applies the principles of pooling, abstraction, and automation to all datacenter services such as compute, storage, networking, and security. It enables the multi-tenancy business model for IT operation and provides the portal for self-service provisioning. EMC next-generation VNX flash-optimized unified storage platform Delivers innovation and enterprise capabilities for file, block, and object storage in a single, scalable, and easy-to-use solution. VMware vsphere Transforms datacenters into simplified cloud computing infrastructures and enables IT organizations to deliver flexible and reliable IT services. Provides high availability independent of hardware, operating system, and applications. Reduces planned downtime for common maintenance operations and prevents extended downtime with automatic, rapid recovery in case of failures. VMware vsphere Distributed Resource Scheduler (DRS) Aligns compute resources with business priorities and affinity rules by automatically balancing virtual machines across hosts. Symantec ApplicationHA Simplifies and centralizes application administration and management through integration with VMware vsphere and guest applications. Key results The solution builds the HA and application mobility add-on bundle by providing the following functions: Mission-critical application protection with the automatic restart of virtual machines and SAP application-specific processes in the event of a datacenter, server, or process outages Simple and automated HA protection for complete SAP cloud datacenters provided by vsphere HA Additional benefits include the following: Increased use of hardware and software assets: Improved resource utilization across datacenters Automatic load balancing between datacenters Zero downtime on planned infrastructure maintenance Reduced resources required with single clustered systems that eliminate the need for additional standby systems. Reduced management, maintenance costs and risks with full end-to-end application restart automation, minimizing human intervention. 6

7 Introduction Purpose This white paper describes the HA and application mobility add-on bundle for the EMC Cloud-enabled Infrastructure for the SAP foundation bundle. This solution is intended to provide enhanced application availability and application mobility for workloads across physical datacenter boundaries with EMC VPLEX, VMware vsphere HA, vsphere DRS, and Symantec ApplicationHA in a vmsc configuration. Business continuity for SAP Because SAP is a major part of the core business operations in many organizations, it is critical that these organizations have a business continuity management plan to: Safeguard the continuity of their business operations Protect revenue Recover to at least a minimum level of operation if an outage occurs A business continuity management plan: Helps to manage risks in an IT environment Details contingency strategies of business processes following a disruption of operations Business continuity for IT comprises three components: High availability (HA) Data protection (DP) Disaster recovery (DR) One way to provide high availability is to minimize the effect of an unplanned outage by masking the outage from the end users. This requires an availability automation solution to manage application failover within the same server, or between servers, within or across datacenters to ensure that high availability measures are achieved. The IT Infrastructure Library (ITIL) is a public framework of best practices for IT service management. For more information, refer to the ITIL publication ITSC IT Service Continuity Management. The SAP Solution Management Business Continuity Best Practices document refers to ITIL, which was considered in this white paper. Types of failures Three main types of failures should be considered in IT business continuity management: Technical failures Range from crashes of individual hardware components to entire datacenters. They disrupt normal operations and require different solutions based on their severity to resume the business operations. Logical failures Can be caused by faulty or malicious software, or the incorrect use of software such as corrupt data that can disrupt business processes. Logistical failures Are operational or logistical business operations such as unavailable staff or facilities. 7

8 This paper focuses on the high availability component to address technical failures and increase the availability of a cloud-enabled SAP landscape. Recovery The following are two major types of recovery after an outage: System recovery Technical availability of failed systems must be reestablished. Technical availability through high availability is the main focus of this paper. Disaster recovery is another form of system recovery after a disaster event. The EMC Cloud-enabled Infrastructure for SAP disaster recovery add-on bundle addresses this kind of recovery type. Business recovery Logical or data inconsistencies must be corrected. (Logical errors may sometimes be the result of technical failures). The EMC Cloud-enabled Infrastructure for SAP data protection add-on bundle addresses the technical aspect of this recovery type. Simplifying SAP high availability This white paper describes a solution that addresses technical failures and provides high availability protection in a simplified and cost-effective manner compared to traditional HA solutions in the market. This solution was implemented in an EMC lab environment to validate the protection of an SAP system or landscape after minor and major technical failures. Objectives Audience The objectives of the white paper are to: Introduce the key enabling technologies Describe the solution architecture and design Describe how the key components are configured Present the results of the tests and validation performed Identify the key business benefits of the solution This white paper is intended for SAP Basis Administrators, storage administrators, IT architects, and technical managers responsible for designing, creating, and managing mission-critical SAP applications in 24/7 landscapes. Previous technical knowledge of cloud, virtualization, server, networking, and storage solutions is required to fully comprehend the concepts and benefits described in this paper. 8

9 Terminology Table 1 defines the terms and abbreviations used in this white paper. Table 1. Term AAS ABAP ASCS dvswitch EHP ERP Terminology Description SAP Additional Application Server (previously known as Dialog instance) SAP Advanced Business Application Programming, the programming language used by SAP, describes the type of SAP architecture used in this solution. ABAP SAP Central Services (runs the central enqueue and message server services) vsphere distributed switch SAP Enhancement Package Enterprise resource planning, powered by the SAP NetWeaver technology platform, is a fully-integrated enterprise resource planning (ERP) application that fulfills the core business needs of midsize companies and large enterprises across all industries and market sectors. SAP ERP DB SAP Database Server that runs the database instance 9

10 Technology overview Introduction This section introduces the concept of the Cloud-enabled Infrastructure as a whole and how its key components are integrated. This section also describes the hardware and software components part of this solution. The Cloud-enabled Infrastructure for SAP solution is a result of the preferred three-way partnership between EMC, SAP, and VMware. The infrastructure is divided into several different functionalities, which are delivered as separate add-on bundles, as illustrated in Figure 1. The EMC Cloud-enabled Infrastructure for SAP solution is designed to offer flexibility so that customers can choose the required cloud functionalities (hereon referred to as add-on bundles), which they can enable without losing sight of their end goal an efficient and reliable private cloud. Figure 1. Cloud-enabled Infrastructure for SAP A journey to the private cloud starts with the foundation bundle, which is described in EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle VNX White Paper. 10

11 The EMC Cloud-enabled Infrastructure for SAP foundation bundle is a mandatory component and it is the base for all add-on bundles. Table 2 outlines the functions and benefits of the EMC Cloud-enabled Infrastructure foundation bundle. Table 2. On-premises EMC Cloud-enabled Infrastructure for SAP solution: Foundation bundle Function Benefits Technology Virtual datacenter for SAP Autonomy of business units and application operations Service catalogs Service level agreements (SLAs) Management of vcloud tenants Resource pooling VMware vcloud Suite Enterprise Infrastructure chargeback Integrated cloud management and performance analysis Storage tiering Cloud networking and security for SAP Cost measurement, analysis, and reporting of the use of compute, network, storage, and backup resources (in combination with data protection add-on bundle) Manage availability, capacity, performance, and health in the SAP landscape Automatically get the right data to the right place at the right time Cloud-enabled Infrastructure security framework Authorization concepts Compliance and non-compliance tracking VMware vcenter Chargeback VMware vcenter Operations Manager EMC Storage Resource Management EMC Storage Analytics (ESA) EMC Virtual Storage Integrator (VSI) EMC next-generation VNX EMC FAST Suite VMware vcloud networking and security For more information about the foundation bundle, refer to the EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle VNX White Paper. 11

12 Table 3 lists the five add-on bundles that enable more cloud functionalities and data center enhancements. Table 3. EMC Cloud-enabled Infrastructure for SAP solution: Add-on bundles IT strategy Bundle Benefits Technology Business continuity High availability (HA) and application mobility for SAP High availability within one datacenter and across two datacenters with application awareness for the complete SAP landscape Non-disruptive movement of applications from one datacenter to another datacenter Improved resource utilization across datacenters Minimize downtime for infrastructure maintenance EMC VPLEX Metro vsphere HA vsphere DRS Symantec ApplicationHA Disaster recovery (DR) for SAP Provide disaster recovery protection for cloud management applications and SAP systems VMware vcenter Site Recovery Manager EMC RecoverPoint Data protection (DP) for SAP Provide data protection to cover the cloud management applications and SAP systems Backup and recovery at all datacenters with remote replication of backup sets for offsite protection EMC Avamar EMC Data Domain EMC Data Protection Advisor Service support/ provisioning Automation and operations for SAP SAP application virtualization enables any service any time on any server Provision SAP systems on demand with automated endto-end process SAP NetWeaver Landscape Virtualization Management Reduce downtime window during maintenance leveraging mass operations Enhanced security Enhanced security and compliance for SAP Efficient, collaborative enterprise governance, risk and compliance (egrc) program across IT, finance, operations, and legal domains RSA Archer egrc RSA Data Loss Prevention RSA SecurID Data loss prevention Secure user to network authentication 12

13 Solution architecture The following sections describe the additional components required for the EMC Cloudenabled Infrastructure for SAP to transition from the foundation bundle to the HA and application mobility add-on bundle, as shown in Figure 2. Figure 2. Transition from foundation to HA/Mobility for SAP bundle This add-on bundle combines EMC VPLEX Metro and VMware vsphere stretched cluster with the EMC Cloud-enabled Infrastructure for SAP foundation bundle, as well as providing a high level of availability. VPLEX Metro and vsphere HA/DRS in a stretched cluster provide high availability through the effective use of cluster resources at both datacenters, as shown in Figure 3. Figure 3. High availability infrastructure layers 13

14 Storage high availability In the EMC Cloud-enabled Infrastructure for SAP foundation bundle, all vsphere virtual machines in both the management and resource clusters are stored in VMware datastores based on LUNs in the local EMC VNX. This bundle uses EMC VPLEX Metro federated solution across the two datacenters to provide the distributed storage to the all the vsphere ESXi hosts across datacenters, using the EMC VPLEX Metro distributed virtual volumes. An EMC VPLEX Metro Distributed Virtual Volume is seen as only one volume (datastore). It is replicated synchronously (read/write) on the EMC VNX arrays on both datacenters, independent from where updates are being made to them. EMC VPLEX Metro provides active copies of data at both datacenters and across a stretched Layer 2 IP network. VMware vmotion provides uninterrupted mobility of application workloads across the datacenters enabled by EMC VPLEX. This bundle also uses EMC VPLEX Witness to monitor connectivity between the two EMC VPLEX clusters on each datacenter and ensure continued availability in the event of an inter-cluster link failure or a datacenter failure. EMC VPLEX Witness is deployed on a virtual machine at a third, separate failure domain (Datacenter C). SAN resiliency with EMC PowerPath /VE EMC PowerPath/VE provides the capability to: Automate and optimize data paths in virtual environments to ensure business process availability and performance. Protect virtual environments from physical hardware failures to ensure uninterrupted service and the automatic failover and recovery. Simplify load balancing to help eliminate I/O bottlenecks. The load balancing algorithms of EMC PowerPath/VE automatically adjust the I/O path use from virtual machines to local I/O loads. With EMC VPLEX Cross-Cluster Connect, PowerPath/VE leaves the cross cluster volumes in the standby mode, if it detects a preferred path to the local VPLEX storage volumes. Network high availability In each datacenter, a redundant 10 GbE fabric provides the core IP Network for datacenter traffic and uplinks for user access. The same IP subnets are available at both datacenters. The Layer 2 routers extend the Layer 2 broadcast domains across the two datacenters. 14

15 Physical architecture Figure 4 shows the physical architecture of all layers of the solution, including the IP and SAN network components. Figure 4. Physical architecture The physical architecture depicted in Figure 4 provides redundant components and connections in all infrastructure layers. Brocade gear was used for the SAN and IP networks in the validated lab environment, but Cisco gear with the same specifications can be used instead, or a combination of both can be deployed. IP network layer The IP network in each datacenter has one Brocade MLXe router and two Brocade VDX 6720 switches in a Virtual Cluster Switch VCS configuration. All vsphere ESXi hosts are connected to the network with redundant 10 GbE connections provided by 10 Gb converged network adapter (CNA) cards. The two switches at each datacenter are connected to a router with a Virtual Link Aggregation Group (vlag). The routers extend the Layer 2 network between two datacenters. 15

16 Note: A vlag is a fabric service that enables a Link Aggregation Group (LAG) to originate from switches. As the standard LAG, a vlag uses the Link Aggregation Control Protocol (LACP) to control the bundling of several physical ports together to form a single logical channel. All network traffic between Datacenter A and Datacenter B are routed by using multiple ports configured as a LAG. Figure 5 shows the IP network layer. Figure 5. SAN layer IP network layer The SAN in each datacenter was built with Brocade DCX 8510 Backbone switches, as shown in Figure 6. All vsphere ESXi hosts are connected to the SAN by using redundant 8 GbE connections provided by the pair of host bus adapters (HBAs) in each server. The multiple Fibre Channel (FC) connections between the Brocade DCX 8510 Backbone switches are not only used for mirroring EMC VPLEX Metro storage across datacenters, but also provide a HA Cross-Cluster Connect between datacenters. Figure 6. SAN Layer 16

17 Logical architecture Figure 7 shows the logical architecture of all layers of the solution, including the network components. Figure 7. Logical architecture The logical architecture depicted in Figure 7 demonstrates that the management and resource clusters are stretched to span both datacenters. 17

18 Protection layers Table 4 summarizes the high availability layers provided before and after the transition between the foundation and HA bundles. DC represents datacenter in the table. Table 4. High availability in local and remote datacenters ID Layer Protected components DC DC Protection 01 SAP Application SAP work processes (DIA/UPD/UPD2/SPO) A A & B Multiple SAP Appl. Servers 02 SAP Application & Database SAP Database, SAP AAS1, SAP AAS2, SAP ASCS, and SAP shared file systems A A & B Symantec ApplicationHA 03 Operating System Operating system services and file systems A A & B VMware vsphere HA and Symantec ApplicationHA 04 Virtual machine Management and SAP virtual machines A A & B VMware vsphere HA / DRS 05 Host Server A A, B, & C VMware vsphere ESXi 06 Fabric SAN Fabric paths management A A & B EMC PowerPath/VE 07 LAN IP Network uplinks A A & B Redundant 10GbE IP networks 08 SAN Storage area network A A & B Redundant 8Gb FC networks 09 SAN Storage area network paths A A & B EMC Cross-Cluster Connect 10a SAN Storage array A EMC VPLEX Local 10b SAN Storage array A & B EMC VPLEX Metro 11 Storage Local storage resources (drives, FAs, DAs, others) A B EMC VNX 12 Storage Local storage resources (drives, FAs, DAs, others) A A EMC VNX Local high availability can also be provided for a single datacenter by using the components numbered 01 to 08 and 12 described in Table 4. For more information, refer to the VMware and EMC documents listed in the References section. EMC VPLEX Local provides a higher level of local high availability with a clustering architecture that provides virtual volumes replicated between two local storage arrays. This allows the servers in the local datacenter to have read/write access to both shared block storage devices. This can be accomplished by using the components described in Table 4 except for component numbered 10a. For more information, refer to the EMC VPLEX documents listed in the References section. EMC VPLEX Metro extends the concept of local high availability to an additional metro distance datacenter. This storage transformation takes high availability to a new level of mission-critical business continuity. The components required for this configuration are all the components described in the Protection column in Table 4, except the component with the number 10b. All scenarios described above require the installation, configuration, and the implementation of the best practices for each component as described in the corresponding sections later in this document and in the documents listed in the References section. Figure 7 shows the high-availability design with VPLEX Witness and Cross-Cluster Connect deployed in this bundle, providing the highest level of resilience. Each of the components shown in Table 4 is explored in more detail in the relevant sections of this white paper. 18

19 Hardware resources Table 5 details the hardware resources for the solution. Table 5. Solution hardware resources Purpose Quantity Configuration Storage (Datacenter A) 1 EMC VNX8000 Storage (Datacenter B) 1 EMC VNX8000 Distributed storage federation 2 EMC VPLEX Metro cluster, with2 x VS2 engines ESXi hosts for management cluster 4 2 x four-core CPUs, 128 GB RAM ESXi hosts for resource cluster 4 4 x eight-core CPU, 256GB RAM ESXi hosts for VPLEX Witness 2 2 x two-core CPUs, 48 GB RAM Network switching and routing platform 2 Brocade DCX 8510 Backbone, with: Fx8-24 FC extension card 2 x 48-port FC Blades with 16 Gb FC line speed support Brocade MLXe Router 4 Brocade VDX 6720 in VCS mode Software resources Table 6 details the software resources used in the solution. Note: Other operating system and database combinations supported by the EMC Solutions Support Matrix (SSM) can also be used. Table 6. Solution software resources Software Version Purpose EMC Block Operating Environment (OE) Operating environment for VNX EMC PowerPath/VE 5.8 Multipathing software providing continuous active paths EMC Unisphere VNX management software EMC Unisphere for VPLEX 5.2 VPLEX management software EMC VPLEX Witness 5.2 Handles VPLEX failures & inter-cluster communication loss SAP ERP 6.0 EHP5 Unicode standard system SAP NetWeaver 7.02 Unicode x86_64 Microsoft SQL Server 2012 Used on the SAP systems 2008 R2 Used by VMware vcenter Chargeback and vcloud Director VMware vsphere 5.1 Hypervisor hosting all virtual machines VMware vcenter Server 5.1 Management and resource clusters VMware vcloud Director 5.1 Orchestration Tool for Provisioning Virtual Infrastructure VMware vcloud Networking and Security 5.1 Security software Microsoft Windows Enterprise Server 2008 R2 Operating system for the SAP virtual machines Symantec ApplicationHA Application-aware clustering software for the guest OS of the virtual machines 1 For release availability information, see the Symantec website. 19

20 Key components Overview EMC VPLEX This section introduces the following technology components: EMC VPLEX EMC next-generation VNX EMC PowerPath/VE EMC Virtual Storage Integrator (VSI) VMware vcloud Suite Symantec ApplicationHA EMC VPLEX is a storage virtualization solution for both EMC and non-emc storage arrays, which offers many advantages for data migrations between arrays. VPLEX performs nondisruptive copying of the underlying storage volumes between the old and new array, as shown in Figure 8. The VPLEX tunable copy process enables dynamic copying without affecting host performance. All hosts remain online during this process. VPLEX substantially reduces the service expenses and time-to-value related in moving to a new infrastructure. Figure 8. Data migration between arrays with VPLEX EMC VPLEX Metro is the primary technology enabled in the solution. EMC VPLEX Metro is a storage area network-based (SAN) federation solution that delivers both local and distributed storage. VPLEX Metro enables the same data to exist in two datacenters in separate geographical locations that can be accessed and updated at both datacenters at the same time. With EMC VPLEX Witness, applications continue to be available, with minimal interruption or downtime, if an EMC VPLEX outage was to occur at one of the datacenters. EMC offers the following three VPLEX configurations for high availability and data mobility: EMC VPLEX Local EMC VPLEX Metro EMC VPLEX Geo For more information, refer to EMC VPLEX Metro infrastructure. 20

21 EMC nextgeneration VNX The EMC VNX flash-optimized unified storage series delivers innovation and enterprise capabilities for file, block, and object storage in a single, scalable, and easy-to-use solution. Ideal for mixed workloads in physical or virtual environments, VNX combines powerful and flexible hardware with advanced efficiency, management, and protection software to meet the demanding needs of virtualized application environments. VNX includes many features and enhancements designed and built upon the first success of previous generations. The following features and enhancements support the performance and scalability demands of virtual environments: More capacity with optimized multicore optimization (MCx) with Multicore Cache, Multicore RAID, and Multicore FAST Cache Greater efficiency with a flash-optimized hybrid storage system EMC PowerPath/VE EMC VSI VMware vcloud Suite PowerPath/VE is a path management solution for VMware. It provides the highest level of dynamic load balancing, path failover, path restoration, path testing, and automated performance optimization. EMC VSI provides multiple feature sets including Storage Viewer (SV), Path Management, and Unified Storage Management. It facilitates the discovery and identification of all EMC storage devices allocated to vsphere ESXi hosts and virtual machines. Unified Storage Management simplifies the provisioning of VNX pool-based storage for datacenters, vsphere ESXi host, and resource pools. Path Management enables you to control how users access datastores. VMware vcloud Suite provides all components for building and running a private cloud infrastructure, based on VMware vsphere that leverages the software-defined datacenter architecture. This architectural approach delivers virtualized infrastructure services (compute, network, security, and availability) with built-in intelligence to automate the on-demand provisioning, placement, configuration and control of applications based on defined policies. This solution includes the following vcloud Suite components: VMware vsphere Compute virtualization platform with policy-based automation VMware vcenter Site Recovery Manager 2 Automated disaster recovery planning, testing, and execution VMware vcloud networking and security 3 Networking and security with ecosystem integration for a virtualized compute environment. VMware vcenter Operations Management Suite 4 Integrated, proactive performance, capacity, and configuration management for dynamic cloud environments. 2 Used in the disaster recovery bundle solution. 3 Used in the foundation bundle solution. 4 Used in the foundation bundle solution. 21

22 VMware vcloud Director Virtualized datacenters with multi-tenancy and public cloud extensibility VMware vsphere VMware vsphere virtualizes and aggregates the underlying physical hardware resources across multiple systems and provides pools of virtual resources to the datacenter. As a cloud operating system, VMware vsphere manages large collections of infrastructure (such as CPUs, storage, and networking) as a seamless and dynamic operating environment, managing the complexity of a datacenter. vsphere HA maximize uptime across your virtualized infrastructure, reducing unplanned downtime and eliminating planned downtime for server and storage maintenance. VMware vcloud networking and security VMware vcloud networking and security is the leading software-defined networking and security solution that enhances operational efficiency and unlocks agility, enables extensibility to rapidly respond to business needs, and provides a broad range of services in a single solution, including virtual firewall, virtual private network (VPN), load balancing, and VXLAN extended networks. VMware vcloud Director VMware vcloud Director orchestrates the provisioning of software-defined datacenter services as complete virtual datacenters that are ready for consumption in a matter of minutes. Virtual datacenters provide virtualized computing, networking, storage, and security. Software-defined datacenter services and the virtual datacenters fundamentally simplify infrastructure provisioning, and enable IT to move at the speed of business. Symantec ApplicationHA Symantec ApplicationHA is a software solution integrated on top of vsphere HA and provides application awareness to vsphere HA, so it can monitor and restart them automatically if a hardware or software failure occurs inside the VMware virtual machines. Symantec ApplicationHA adds a layer of application awareness to the core HA functionality offered by VMware vsphere virtualization technology. Symantec ApplicationHA is based on Veritas Cluster Server and uses similar concepts such as agents, resources, and service groups. Symantec ApplicationHA has a lightweight server footprint that allows faster installation and configuration. 22

23 EMC VPLEX Metro infrastructure Introduction This section describes the VPLEX Metro infrastructure for the solution, which is composed of the following components: EMC VPLEX Metro cluster at each datacenter (Datacenter A and B) EMC VPLEX Witness in a separate failure domain (Datacenter C) EMC VPLEX EMC VPLEX is a storage virtualization solution for both EMC and non-emc storage arrays. EMC offers VPLEX in different configurations to address customer needs for high availability and data mobility. For detailed descriptions of these VPLEX configurations, refer to the documents listed in the References section. EMC VPLEX Metro VPLEX Metro uses a unique clustering architecture to allow servers at multiple datacenters, geographically dispersed, to have read/write access to shared block storage devices. VPLEX Metro delivers active/active, block-level access to data on two datacenters within synchronous distances with a round-trip time of up to 5 milliseconds. For this bundle, a round-trip time is not to exceed 1 millisecond with the Cross-Cluster Connect configuration. EMC VPLEX Witness VPLEX Witness is an external server that is installed as a virtual machine in a separate failure domain to the VPLEX clusters. VPLEX Witness connects to both VPLEX clusters using a VPN over the management IP network. It requires a round-trip time (RTT) that does not exceed 1 second. By reconciling its own observations with information reported periodically by the clusters, VPLEX Witness enables the cluster(s) to distinguish between inter-cluster network partition failures and cluster failures and automatically resume I/O at the appropriate datacenter. VPLEX Witness failure handling semantics apply only to distributed volumes within a consistency group and only when the detach rules identify a static preferred cluster for the consistency group (see VPLEX consistency groups on page 25 for further details). EMC VPLEX management interface You can manage and administer a VPLEX environment with the web-based Unisphere for VPLEX or you can connect directly to a management server and start a VPLEX command line interface (VPLEXcli) session. 23

24 EMC VPLEX HA VPLEX Metro enables application and data mobility. When configured with VPLEX Witness, it provides a high-availability infrastructure for clustered applications. VPLEX Metro enables you to build a stretched cluster like a local cluster, and removes the datacenter as an SPOF. Furthermore, as the data and applications are active at both datacenters (active/active), this configuration provides a simple high availability business continuity solution. VPLEX Witness is part of the VPLEX Metro and enables two optional architectures as follows: VPLEX Metro HA with Cross-Cluster Connect Hosts are connected to the local VPLEX cluster on which they reside, and also have an alternate path to the remote VPLEX cluster. You can achieve the highest degree of availability by using a VPLEX Cross-Cluster Connect configuration. In the unlikely event that an entire VPLEX cluster, storage array, or SAN fails, with Cross-Cluster Connect configuration, hosts have an alternate path to the VPLEX Metro distributed volumes through the surviving VPLEX cluster, eliminating a short downtime associated with restarting the virtual machine on the secondary datacenter. The inter-cluster network latency should not exceed 1 millisecond RTT between VPLEX clusters. VPLEX Metro HA without Cross-Cluster Connect Hosts are only connected to the local VPLEX cluster on which they reside. Because host clusters are connected to the VPLEX distributed volume providing the same block data between two datacenters, when components (hosts, storage, servers and so on) fail, it minimizes the recovery time by automatically restarting the virtual machine on the secondary datacenter. The latency exceeds 1 millisecond but should be less than 5 milliseconds RTT. VPLEX logical storage structures VPLEX encapsulates traditional physical storage array devices and applies layers of logical abstraction to these exported LUNs, as shown in Figure 9. Figure 9. VPLEX logical storage structures 24

25 VPLEX storage volume A storage volume is a LUN exported from an array and encapsulated by VPLEX. An extent is the mechanism VPLEX uses to divide storage volumes and may use all or part of the capacity of the underlying storage volume. A device encapsulates an extent or combines multiple extents or other devices into one large device with a specific RAID type. At the top layer of the VPLEX storage structures are virtual volumes. These are created from a top-level device (a device or distributed device) and always use the full capacity of the top-level device. Virtual volumes are the elements that VPLEX exposes to hosts using its front-end ports. VPLEX presents a virtual volume to a host through a storage view. VPLEX can encapsulate devices across heterogeneous storage arrays, including virtually provisioned thin devices and traditional LUNs. VPLEX consistency groups Consistency groups aggregate virtual volumes so that the same properties (detach rules and others) can be applied to all volumes in the group. There are two types of consistency groups: Synchronous consistency groups These are used in VPLEX Local and VPLEX Metro to apply the same detach rules and other properties to a group of volumes in a configuration. This simplifies configuration and administration in large systems. With write-through caching in synchronous consistency groups, in the separated cluster environment, VPLEX Metro supports up to 5 milliseconds of latency. VPLEX Metro sends writes to the back-end storage volumes, and acknowledges a write to the application only when the back-end storage volumes in both clusters acknowledge the write. Asynchronous consistency groups These are used for distributed volumes in VPLEX Geo, where clusters can be separated by up to 50 milliseconds of latency. Consistency groups are particularly important for databases and their applications. For example: Write-order fidelity Maintains data integrity, vsphere ESXi LUNs forming a datastore cluster should be placed together in a single consistency group. Transactional dependency Multiple databases often have transaction dependencies, such as when an application issues transactions to multiple databases and expects the databases to be consistent with each other. All LUNs that require I/O dependency to be preserved should reside in a single consistency group. Application dependency SAP stores database files within a set of datastores that must be accessible to maintain database availability. The datastore devices of database files should reside in a single consistency group. Detach rules Detach rules are predefined rules that determine I/O processing semantics for a consistency group when connectivity with a remote cluster is lost, for example in the case of a network partitioning or remote cluster failure. 25

26 Synchronous consistency groups support the following detach rules to determine cluster behavior during a failure: The static preference rule identifies a preferred cluster. The no-automatic-winner rule suspends I/O on both clusters. Setting a detach rule is always invoked when connectivity is lost between clusters. However, VPLEX Witness can be deployed to override the static preference rule and ensure that the non-preferred cluster remains active if the preferred cluster fails. VPLEX Metro solution configuration Storage structure Figure 10 shows the physical and logical storage structure used by VPLEX Metro in this solution. Figure 10. VPLEX physical and logical storage structures for this solution A one-to-one mapping between storage volumes, extents, and devices exists at each datacenter. The devices inside both Datacenter A (cluster-1) and Datacenter B (cluster-2) are virtually provisioned thin devices. All cluster-1 devices are mirrored remotely on cluster-2, in a distributed configuration, to create distributed devices. These distributed devices are encapsulated by virtual volumes, which are then presented to the hosts through storage views. 26

27 Configuration process For this bundle, we used the configuration wizard provided by Unisphere for VPLEX to configure the VPLEX Metro logical storage structure. Storage volumes Figure 11 shows that several storage volumes were created on Datacenter A, as displayed in the VPLEX Management Console. Figure 11. EMC VPLEX storage volumes (Datacenter A) Extents VPLEX divides storage volumes in extents. This bundle includes a one-toone mapping between extents and storage volumes, as shown in Figure 12. Extents have the same size as the storage volumes from which they are created. Figure 12. EMC VPLEX Extents Devices One-to-one mapping was configured between devices and extents. Figure 13 shows the option used to configure this mapping. 27

28 Figure 13. EMC VPLEX device creation wizard: mapping extents Distributed devices The distributed devices were created by mirroring a device remotely in a distributed RAID 1 configuration, as shown in Figure 14. Figure 14. EMC VPLEX device creation wizard: selecting mirrors Virtual volumes All top-level devices are distributed devices. These devices are encapsulated by virtual volumes, which EMC VPLEX presents to the hosts through storage views. The storage views define which hosts access which virtual volumes on which EMC VPLEX ports. Consistency groups In this bundle, the following two synchronous consistency groups were created, as shown in Figure 15. Resource cluster consistency group contains the SAP virtual machines. Management cluster consistency group contains the management virtual machines. Figure 15. Consistency-group configuration in Unisphere for VPLEX 28

29 VPLEX Witness configuration This bundle uses EMC VPLEX Witness to monitor connectivity between the two VPLEX clusters and ensure continued availability when an inter-cluster network partition fails or a datacenter fails. This is considered a VPLEX Metro HA configuration as storage availability is ensured at the surviving datacenter. EMC VPLEX Witness was deployed at a third, separate failure domain (Datacenter C) and was connected to the EMC VPLEX clusters at Datacenter A and Datacenter B. Datacenter C is located at a distance of less than 1 second RTT from Datacenters A and B. When a VPLEX Witness has been installed and configured, the VPLEX Management Console displays the status of cluster witness components, as shown in Figure 16. Figure 16. EMC VPLEX Witness components and status For additional details, refer to the Implementation and Planning Best Practices for EMC VPLEX Technical Note and the Using VPLEX Metro with VMware High Availability and Fault Tolerance for Ultimate Availability White Paper listed in the References section. 29

30 VMware virtual datacenter Introduction VMware virtual datacenter is the key enabling technology introduced in the EMC Cloud- Enabled Infrastructure for SAP Foundation Bundle VNX White Paper. This solution reuses the existing architecture in the foundation bundle and configures the HA capability as an add-on. The SAP systems are fully virtualized in vapps using VMware vsphere and vcloud Director. The same networking and security configurations in foundation bundle also apply to the HA and application mobility add-on bundle. This section describes the following components, technologies and options: VMware vcloud Director VMware vsphere VMware vcenter Server VMware vcenter Chargeback Manager VMware vsphere vmotion VMware vsphere HA VMware vsphere DRS EMC PowerPath/VE for VMware vsphere EMC VSI for VMware vsphere VMware vcloud Director VMware vcloud Director applies the principles of pooling, abstraction, and automation to all datacenter services like storage, networking, and security using virtual datacenters. A virtual datacenter is an elastic logical container that provides all infrastructure services necessary to make workloads operational in minutes. Applications provisioned into these containers are automatically placed in the most optimal VMware vcenter server cluster. VMware vsphere VMware vsphere is a virtualization platform that provides infrastructure services transforming IT hardware into a shared computing platform, and application services helping IT organizations deliver high levels of availability, security, and scalability. vsphere is the base component in the EMC Cloud-enabled Infrastructure for SAP on all add-on bundles. It provides an abstraction of the physical server layer that allows virtual machines to be independent of the brand, model and type of x86 server architecture where they are running. VMware vcenter Server VMware vcenter Server is the centralized management platform for vsphere environments, enabling control and visibility at every level of the virtual infrastructure. vcenter Servers manage vsphere HA/ DRS clusters that are created between the two datacenters using VMware vsphere 5 hosts, and are connected to the vsphere hosts at both datacenters. 30

31 VMware vcenter Chargeback Manager VMware vcenter Chargeback Manager improves utilization of your virtual infrastructure with accurate visibility into the true costs of virtualized workloads. It enables line-ofbusiness owners to have full cost transparency and accountability for self-service resource requests. It allows IT organizations to customize rate cards and prices to the processes and policies of different organizations. VMware vsphere vmotion VMware vsphere vmotion is VMware technology that supports live migration of virtual machines across servers without any interruption in the availability of the virtual machine. This allows the live relocation of virtual machines to new datastores. VMware vsphere HA VMware vsphere HA is a vsphere component that provides high availability for any application running in a virtual machine, regardless of its operating system or underlying hardware configuration. vsphere HA provides uniform failover protection against hardware and operating system outages within your virtualized IT environment. vsphere HA specifically reduces unplanned downtime by using multiple VMware vsphere ESXi hosts, configured as a cluster, to provide rapid recovery from outages and cost-effective high-availability for applications running on virtual machines. VMware vsphere DRS VMware vsphere DRS dynamically and automatically balances load distribution and virtual machine placement across multiple ESXi hosts using vsphere vmotion. EMC PowerPath/VE for VMware vsphere EMC PowerPath/VE for VMware vsphere delivers multipathing features that optimize VMware vsphere virtual environments. PowerPath/VE installs as a kernel module on the vsphere ESXi host and works as a multipathing plug-in (MPP) that provides enhanced path management and load-balancing capabilities for vsphere ESXi hosts. EMC VSI for VMware vsphere EMC VSI for VMware vsphere is a vcenter plug-in that provides a single management interface for managing EMC storage. VSI provides a unified and flexible user experience that allows each feature to be updated independently, and allows new features to be introduced rapidly in response changing customer requirements. When PowerPath/VE is installed on a vsphere ESXi host, VSI presents important multipathing details for devices, such as the load-balancing policy, the number of active paths, and the number of dead paths. 31

32 Configuring VMware vcloud Director This section describes how the VMware vcloud Director settings were configured to integrate this HA bundle with the EMC Cloud-enabled Infrastructure for SAP foundation bundle. Figure 17 illustrates the cloud architecture with the HA add-on bundle. Figure 17. Cloud architecture with the HA bundle To enable the HA capability, the following components must be configured: vsphere DRS cluster Storage Provider vdc (PvDC) Organization vdc (OvDC) 32

33 Configuring DRS cluster DRS clusters were configured to fully use HA capability for both management and resource clusters: Management cluster vsphere ESXi hosts from Datacenter B were added to the original management DRS cluster, facilitating the management cluster with cross datacenter HA capability. Resource cluster A new DRS cluster named Premium was created. vsphere ESXi hosts from both datacenters were added to the Premium DRS cluster, facilitating the resource cluster with cross datacenter HA capability. Configuring storage New datastores were created based on the EMC VPLEX Metro Distributed virtual volumes for both resource and management clusters together with storage DRS clusters and storage profiles to enable HA on the storage level. The virtual machines on management cluster were non-disruptively migrated from the local VNX located on Datacenter A to the EMC VPLEX Metro distributed virtual volumes. New datastores were created for both management and resource clusters, and the storage settings configured in vcloud Director are listed in Table 7. Table 7. Storage configuration in vcloud Director Consistency group Storage DRS cluster Storage profile Datastores Management clusters HA MGMT N/A VPLEX_MGMT_001 (4 TB) VPLEX_MGMT_002 (4 TB) Resource clusters HA Gold Tier HA Gold VPLEX_DS1 (4 TB) VPLEX_DS2 (4 TB) VPLEX_DS3 (4 TB) VPLEX_DS4 (4 TB) VPLEX_DS5 (4 TB) HA Silver Tier HA Silver VPLEX_DS1_Silver (4 TB) Eight 4 TB datastores were created for the management and resource clusters. For the resource cluster, two storage DRS clusters were created to group datastores with different performance levels: HA Gold Tier A storage DRS cluster for placing datastores set to the highest performance. HA Silver Tier A storage DRS cluster for placing datastores set to high performance. Five 4 TB datastores were assigned to the HA Gold Tier storage DRS cluster and one 4 TB datastore to the HA Silver Tier storage DRS cluster. Two new storage profiles, HA Gold and HA Silver were created to characterize the datastores with HA capabilities. Afterwards, the storage profiles were assigned to the corresponding datastores in the storage DRS clusters as shown in Table 7. 33

34 All created distributed volumes were presented to the all four vsphere ESXi hosts on both stretched clusters. Figure 18 shows the configuration details for the datastore cluster HA Gold Tier created for the resource cluster. Figure 18. HA Gold Tier datastore cluster Configuring PvDC Two PvDCs, Advanced and Standard, were defined in EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle VNX White Paper. In this bundle a new PvDC named Premium was created based on the Premium cluster from the DRS cluster. The Premium PvDC is dedicated for SAP systems that require cross datacenter HA capability. Storage profiles HA Gold and HA Silver were added to the PvDC storage profile tab in the PvDC configuration. Figure 19 shows the storage profile configuration needed for PvDC. 34

35 Figure 19. PvDC configuration For external networks in PvDC, the configuration in Advanced and Standard PvDC also applies to the Premium PvDC. Configuring OvDC We create new OvDCs to use cross datacenter HA capability. In this solution, a new OvDC called OrgB_Dedicated_HA for OrgB was built on top of Premium PvDC. The same organization networks configured in the foundation bundle were also created in the new OvDC. For vshield Edge gateways, we enabled high availability in the edge property as shown in Figure 20. Figure 20. vshield Edge gateway configuration 35

36 After this configuration, vcloud Director automatically deployed an additional standby vshield Edge gateway. Once the primary vshield Edge gateway is out of service, all the networking and security functions switch over to the standby gateway. After successful configuration of the new OvDC, new SAP systems can be deployed. For the existing SAP systems that also require HA capability, but originally deployed in the foundation bundle, you can schedule a downtime and migrate them to the new OvDC. Configuring vcenter Chargeback Manager vcenter Chargeback Manager provides the virtualized infrastructure metering functionality. It integrates with vcloud Director seamlessly, provides the service provider with the capability to chargeback the resources consumed by tenants, and generates cost and utilization reports periodically or on demand. The EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle VNX White Paper defines several pricing models to differentiate OvDC with different allocation models and storage profiles with different storage performance levels. There are several approaches to reflect the chargeback of HA capability. You can create and bind a pricing model to a new OvDC with HA capability, with reasonable charges for infrastructure including CPU, memory, and storage. In this way, you can accurately charge the infrastructure and account for the resources consumed in Datacenter B. Another simpler approach is to set a rate factor for the storage profiles HA Gold and HA Silver to reflect a charge for the HA capability. You only need to set it once and all the vapps deployed on the storage profiles HA Gold and HA Silver will be charged accordingly with base rate times rate factor. That simplifies the management efforts needed to create pricing models for each OvDC compared to the previously described approach. Figure 21 shows how to set the rate factor in vcenter Chargeback Manager. Figure 21. Setting the factor rate for DR storage profiles VMware deployments on VPLEX Metro EMC VPLEX Metro delivers concurrent access to the same set of devices at two physically separate locations and thus provides the active/active infrastructure that enables geographically stretched clusters based on VMware vsphere. The use of Layer 2 routing technology enables extension of virtual LANs, and their subnets, across different physical datacenters. 36

37 Deploying VPLEX Metro with the following features and components provides the described functionality: vmotion Provides the ability to non-disruptively migrate virtual machines between datacenters in anticipation of planned events such as hardware maintenance or power outage of the datacenter. vsphere DRS Provides automatic load distribution and virtual machine placement across datacenters through the use of DRS groups and affinity rules. vsphere HA A VPLEX Metro environment with VPLEX Witness configured is considered as a VPLEX Metro HA configuration, as it ensures storage availability at the surviving datacenter in the event of a datacenter-level failure. Combining VPLEX Metro HA with a host failover clustering technology such as vsphere HA and Symantec ApplicationHA provides an end-to-end automatic application-aware restart for any datacenter-level failure or other outages. Figure 22 illustrates this HA architecture. Figure 22. vsphere HA with Cross-Cluster Connect and VPLEX Witness logical view VPLEX Metro HA Cross-Cluster Connect Provides additional protection of the vsphere HA cluster by adding a Cross-Cluster Connect between the local vsphere ESXi hosts and the VPLEX cluster on the remote datacenter. EMC VPLEX distributed volumes are created across both locations. With underlying EMC VPLEX distributed volumes, vsphere HA clusters are stretched across both datacenters. 37

38 The physical vsphere ESXi hosts are connected to the local VPLEX cluster on which they physically reside, and also have an alternate path to the remote EMC VPLEX cluster through the additional cross-connect network that physically breaks the VPLEX internal link connecting both VPLEX clusters, as show in Figure Figure 23 shows the paths for Cross-Cluster Connect as displayed by EMC VSI. EMC PowerPath/VE automatically detects and sets the cross-cluster paths and puts in auto standby mode. This minimizes cross-datacenter traffic under normal operation, and handles all paths down (APD) situations more efficiently when a datacenter fails. Figure 23. EMC VSI PathViewer: VPLEX distributed devices and cross cluster paths Best practices for EMC VPLEX Cross-Cluster Connect configuration VPLEX Witness must be deployed in a third failure domain. The inter-cluster network latency is not to exceed 1 millisecond RTT between VPLEX clusters at the time of writing. According to EMC best practice, all remote VPLEX connections should be zoned to the local host and local host initiators must be registered to the remote VPLEX. The distributed volume is then exposed from both VPLEX clusters to the same host. The host path preference should have a local path preference set, ensuring the remote path will only be used if the primary one fails so that no additional latency is incurred. This bundle uses VPLEX Metro HA with Cross-Cluster Connect to maximize the availability of the VMware vsphere virtual machines. The key benefit of this solution is the ability to minimize any recovery time if components or even an entire VPLEX cluster fails. This is unlikely because there is no SPOF within a VPLEX engine. Since the physical host has an alternate path to the same storage actively served up by the remote VPLEX cluster, it will automatically remain online due to the VPLEX Witness, regardless of the rule set. 5 EMC VPLEX Metro Witness Technology and High Availability EMC TechBook and Using VPLEX Metro with VMware High Availability and Fault Tolerance for Ultimate Availability White Paper in the References section provides more details. 38

39 VMware stretched cluster configuration VMware and EMC support a stretched cluster configuration that includes vsphere ESXi hosts from multiple datacenters. A VMware vsphere Metro Storage Cluster (vmsc) 6 solution is also referred to as a stretched cluster. In this bundle, both management and resource clusters, as described in the EMC Cloud- Enabled Infrastructure for SAP Foundation Bundle VNX White Paper, are stretched between Datacenter A and Datacenter B by using the EMC VPLEX Metro distributed virtual volumes with vsphere HA and vsphere DRS. There are four vsphere ESXi hosts (physical servers) in each cluster, two at each datacenter, two for the management cluster and two for the resource cluster. VMware vcloud Director manages the resource cluster. EMC VPLEX Metro HA Cross-Cluster Connect provides increased resiliency to the configuration as shown in Figure 22. In VMware vcenter, you can view the vsphere Web Client configuration of the stretched cluster and the features enabled for it, as shown in Figure 24. This view also shows the memory, CPU, and storage resources available to the resource cluster. Figure 24. vsphere cluster with HA and DRS enabled The management cluster was configured in the same manner, with vsphere HA and DRS enabled and all virtual machines stored in VPLEX-based datastores. 6 For detailed requirements and scenarios, see the VMware Knowledge Base articles : Using VPLEX Metro with vsphere HA and : Implementing vsphere Metro Storage Cluster (vmsc) using EMC VPLEX document in the References section. 39

40 Configuring virtual network Each vsphere ESXi host is configured with two 10 GbE physical adapters to provide network failover and high performance. A vsphere distributed switch (dvswitch) 7 provides a single, common switch across all hosts. The 10 GbE physical adapters (also referred to as uplink adapters) are assigned to the dvswitch. Four distributed port groups are assigned to the dvswitch: dvpg_host For virtual machine datacenter network traffic dvpg_management For management traffic dvpg_corp For uplinks to corporate IT Network dvpg_vmotion For vmotion traffic Figure 25 shows the dvswitch configuration. As both vsphere 5.1 distributed switches and physical switches support Link Layer Discovery Protocol (LLDP), the properties of the associated physical switches can also be easily identified from vsphere vcenter server. Figure 25. dvswitch configuration and LLDP detail For details of the dvswitch configurations, refer to the EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle VNX White Paper. The configuration of the dvswitch does not require any changes when moving to the HA bundle. 7 A dvswitch provides a network configuration that spans all member hosts and enables virtual machines to maintain consistent network configuration for migration between hosts. For further information, see the VMware vsphere Networking ESXi 5.1 document in the References section. 40

41 VMware vsphere HA configuration Configuring VMware vsphere HA and VMware vsphere DRS vsphere HA provides high availability for virtual machines by pooling the virtual machines and the vsphere ESXi hosts that they reside on into a cluster. Hosts in the cluster are monitored and in the event of a failure, the virtual machines on a failed host are restarted on alternate hosts. 8 vsphere HA uses multiple vsphere ESXi hosts, configured as a cluster, to provide rapid recovery from outages and cost-effective high availability for applications running in virtual machines. vsphere HA protects application availability in the following ways: It protects against a server failure by restarting the virtual machines on other vsphere ESXi hosts within the cluster. It protects against application failure by continuously monitoring a virtual machine and resetting it in the event of guest OS failure (VM monitoring). In this bundle, both vsphere HA and DRS were enabled, as shown in Figure 26. Figure 26. vsphere HA configuration settings 8 For further information on vsphere HA, see the VMware vsphere Availability ESXi 5.1 documentation. 41

42 VM monitoring VM monitoring was configured to monitor the operating system and the application running inside the guest OS. VM monitoring was configured to restart individual virtual machines if their heartbeat is not received within the configured 60 seconds threshold. The virtual machine and application monitoring option was selected to allow Symantec ApplicationHA to communicate with vsphere HA to exchange virtual machine heartbeat status from the guest OS and application running. Datastore heartbeating To meet vsphere HA requirements for datastore heartbeating, additional datastores were created on VPLEX distributed volumes and presented to all the ESXi hosts on both clusters. In a production environment, vcenter automatically selects two or more datastores for this purpose, based on host visibility. The resource cluster was configured with five datastores (VPLEX_DS1, DS2, DS3, DS4, and DS5) as described earlier, and they were used for Datastore heartbeating, as shown in Figure 27. Configuring multiple VPLEX datastores in a vmsc configuration provides higher redundancy for both datacenters. This enables vsphere HA to heartbeat to a datastore even in case of an IP link failure between datacenters, enabling vsphere HA to determine the state of a host in any failure scenario. Figure 27. vsphere HA Cluster Status heartbeat datastores Virtual machine restart options The virtual machine Restart Priority option for the SAP virtual machines was set per virtual machine. The ABAP SAP Central Instance Services (ASCS) and the SAP Database servers were set to High and the other SAP virtual machines were set to medium (default setting). This instructs the vsphere ESXi hosts to power on virtual machines first in the event of an outage. Figure 28 shows this setting. The Host Isolation Response setting was configured as Leave Powered On. 42

43 Figure 28. Virtual machine Restart Priority Response settings vsphere HA monitoring When you create a vsphere HA cluster, a single vsphere ESXi host is automatically elected as the master host. The master host monitors the state of all protected virtual machines and the slave hosts. When the master host cannot communicate with a slave host over the management network, it uses datastore heartbeating to determine whether the slave host has failed, resulting from a network partition or being isolated. Although vsphere HA is configured by vcenter and exchanges virtual machine state information with vsphere HA, vcenter is not involved when vsphere HA responds to a failure, so while vsphere HA, by design, respond to failures without vcenter, HA relies on vcenter to be available to configure and monitor the cluster. In this bundle, both the management cluster and resource cluster vcenter virtual machines were configured with vsphere HA enabled with datastore heartbeating. Admission Control VMware vcenter Server uses HA Admission Control to reserve resources in the cluster to provide failover protection and ensure virtual machine resource reservations. Admission Control was turned on for both resource and management clusters and the policy Percentage of Cluster Resources Reserved was selected and set to 50 percent. This configuration reserved cluster resources to guarantee the restart of all virtual machines in case of a simultaneous failure of two physical vsphere ESXi hosts in the same cluster. 43

44 Best practices for HA clusters EMC considered the following key vsphere HA cluster best practices 9 in the bundle to ensure optimal performance: Configure alarms to monitor cluster changes to expedite incident management Configure alarms in VMware vcops to be triggered when vsphere HA takes action to maintain availability and send alerts, such as s, to administrators for facilitating troubleshooting. For further information about how to configure alarms in VMware vcops, refer to the EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle VNX White Paper. Monitor cluster validity to guarantee that reserved capacity for failover is available A valid cluster is a cluster in which the admission control policy was not violated. A cluster enabled for vsphere HA becomes invalid when the number of virtual machines powered on exceeds the failover requirement that is either the current failover capacity is smaller than configured failover capacity or when ESXi hosts fail, reducing the available capacity for reservation. Enable Admission Control and select the Percentage of Cluster Resources reserved policy This policy offers the most flexibility in terms of host and virtual machine sizing. Choose a percentage for CPU and memory that reflects the number of host failures that you want to support. This policy uses the actual reservation per virtual machine instead of taking the worst scenario like the other two policies available in Admission Control. The cluster dynamically adjusts when resources are added. Size all the cluster hosts equally For the Percentage of Cluster Resources Reserved policy, an unbalanced cluster results in excess capacity being reserved to handle failures, since vsphere HA reserves capacity for the largest hosts. Mask datastores in a cluster basis (all vsphere ESXi hosts in the cluster) Maximize the chance of restarting virtual machines after a failure by masking the datastores to all vsphere ESXi hosts that are part of the cluster. The vsphere HA master host will be able to access all the datastores to try to communicate to all vsphere ESXi hosts in the cluster to determine their status if the management network is not available after the failure. To enable better handling of all-paths-down scenarios, the value of the advanced setting das.maskcleanshutdownenabled was set to true. This setting allows vsphere HA to trigger a restart response for a virtual machine that has been shut down automatically due to a power outage condition. This setting is not enabled by default. Configure vsphere DRS to support vsphere HA in highly utilized clusters 9 For further information, see VMware vsphere Availability ESXi 5.1 and VMware vsphere High Availability Deployment Best Practices. 44

45 Combine vsphere HA and DRS to protect against failures and to provide load balancing across hosts within a cluster. In a failure scenario, if vsphere HA cannot restart some virtual machines, it asks DRS to try to defragment resources to offer HA another opportunity to restart virtual machines. In order to achieve this, DRS needs to be enabled and configured to be fully automated. Best practices for networking The best practices 10 for the configuration of host NICs and network topology for vsphere HA include recommendations for vsphere ESXi hosts, cabling, switches, routers, and firewalls as follows. Suspend the Host Monitoring feature when making network changes vsphere HA uses the management network to send/receive heartbeats to/from the clustered ESXi hosts. During network maintenance, heartbeat interruptions may happen and vsphere HA may trigger undesired attempts to failover virtual machines, reducing availability. Reconfigure vsphere HA on all cluster hosts after ESXi network changes vsphere HA will re-inspect the network information, and then re-enable Host Monitoring. Notify the vsphere HA Admin in advance before any network maintenance Networking is a vital component of vsphere HA. If any network maintenance must be performed, the vsphere HA administrator must be informed in advance to prepare the environment to ignore false positives. Use the das.isolationaddress advanced attribute to add additional isolation addresses for additional networks A network isolation address is an IP address that is pinged to determine whether a host is isolated from the management network (VMkernel). This IP address is pinged when a host stops receiving heartbeats from other hosts in the cluster. If the host can ping its isolation address, the host is not network isolated. If the host cannot ping the isolation address, the host has likely became isolated from the network. Implement network redundancy at the NIC level with NIC teaming connecting to separate switches. A single management network can result in failovers although only the network has failed. Redundant management networking allows the reliable detection of failures and prevents isolation or partition conditions from occurring, because heartbeats can be sent over multiple networks. Network heartbeating is the primary method to determine the state of a vsphere ESXi host, providing a resilient management network to enable a faster and proper host state determination, without involving datastore heartbeating. Configure the fewest number of segments between clustered vsphere ESXi hosts 10 For further information, see VMware vsphere Availability ESXi 5.1 and VMware vsphere High Availability Deployment Best Practices. 45

46 This recommendation is intended to limit the network SPOF. Routes with too many hops can cause networking packet delays for heartbeats, and increase the possible points of failure. Configuring VMware vsphere DRS vsphere DRS host groups and virtual machine groups DRS host groups and virtual machine groups simplify management of the vsphere ESXi host resources. In this bundle, the host groups and virtual machine groups were created for the resource cluster as shown in Figure 29. Figure 29. Creating host groups and virtual machine groups Table 8 and Table 9 show the DRS groups created for the solution and their assignments. Table 8. Management cluster DRS Group Name Group Type Type Group members Datacenter_A_Servers Host DRS Group vsphere ESXi hosts r710a, r710b Datacenter_B_Servers Host DRS Group vsphere ESXi hosts r710c, r710d Datacenter_A_VMs VM DRS Group DNS virtual machine DNS server Datacenter_B_VMs VM DRS Group DNS2 virtual machine DNS slave server Cross_Datacenter_VMs VM DRS Group vcloud Director and vcenter virtual machines vcloud Director, vcentermc, vcenterrc, vchargeback, SymantecHA, and others 46

47 The group members, vcentermc and vcenterrc respectively stand for management cluster vcenter and resource cluster vcenter. The group members r710a, r710b, r710c, and r710d, respectively stand for vsphere ESXi hosts r710a and r710b (both located on Datacenter A), and vsphere ESXi hosts r710c and r710d (both located on Datacenter B). Table 9. Resource cluster DRS Group Name Group Type Type Group members Datacenter_A_Servers Host DRS Group ESXi hosts c460-1 and c460-3 Datacenter_B_Servers Host DRS Group ESXi hosts c460-2 and c460-4 Datacenter_A_VMs Virtual machine DRS Group vshield virtual machine vse-edge_orgb_ha2-0 Datacenter_B_VMs Virtual machine DRS Group vshield virtual machine vse-edge_orgb_ha2-1 Cross_Datacenter_VMs Virtual machine DRS Group SAP ERP 6.0 virtual machines SAPAB1DB, SAPAB1ASCS, SAPAB1AAS1, APAB1AAS2 Note: The DRS groups identify which vsphere ESXi hosts are in which physical datacenters. The group members c460-1, c460-3, c460-2, and c460-4, respectively stand for vsphere ESXi hosts c460-1 and c460-3 (both physically located on Datacenter A), vsphere ESXi hosts c460-2 and c460-4 (both physically located on Datacenter B). The group members SAPAB1DB, SAPAB1ASCS, SAPAB1AAS1, and SAPAB1AAS2 respectively stand for the SAP AB1 ERP Database server, SAP AB1 ABAP SAP Central Services, SAP AB1 ABAP Application Server 1, and SAP AB1 Additional Application Server 2. vsphere DRS affinity rules DRS uses affinity rules to control the placement of virtual machines on hosts within a cluster. DRS provides the following two types of affinity rules: A VM-Host affinity rule specifies an affinity relationship between a group of virtual machines and a group of hosts. VM-Host affinity rule controls two vshield Edge gateway virtual machines running on different datacenters, as shown in Figure 30. Figure 30. VM-Host affinity rule for resource cluster 47

48 A VM-VM affinity rule specifies whether particular virtual machines should run on the same host or be kept on separate vsphere ESXi hosts. In this solution, both vcenter servers and SQL server virtual machines were configured to run on the same vsphere ESXi hosts in the management cluster, as shown in Figure 31. Figure 31. vsphere DRS affinity rules for management cluster A rule was created to keep the SAP Database and ASCS instances from running on the same host, as shown in Figure 32. This is because both systems are SPOFs and must be protected against ESXi host failures. An additional rule was created to keep the SAP Application Servers running on different ESXi hosts to provide redundancy of SAP work processes. The vshield Edge virtual machines protecting the resource cluster have two redundant virtual machines that should run respectively on Datacenter A and Datacenter B. The rules vshield_edge_datacenter_x, where X can be A or B, were created to instruct vsphere DRS to enforce that requirement. Figure 32. vsphere DRS affinity rules for resource cluster 48

49 Table 10 lists all the vsphere DRS affinity rules consolidated for the management and resource clusters. vcentermc and vcenterrc stand for vcenter for the management cluster and for the resource cluster respectively. Table 10. Consolidated vsphere DRS affinity and anti-affinity rules Cluster DRS rule names DRS Affinity Rules configured Rule type Object Management Datacenter A and B DNS1 and DNS 2 servers Separate Datacenters vcloud vcloud Director and vcenterrc Separate ESXi hosts vcenter SQL, vcentermc and vcenterrc Same ESXi hosts Resource vshield Edge Datacenter A vshield Edge 0 Should run Datacenter A vshield Edge Datacenter B vshield Edge 1 Should run Datacenter B vshield Edge Anti Affinity vshield Edge VMs 0 and 1 Separate ESXi hosts SAP AB1_Application_Servers SAP AAS1 and AAS2 Separate ESXi hosts SAP_AB1_Central_Services SAP Database and ASCS Separate ESXi hosts vsphere DRS rules are created to protect virtual machines against vsphere ESXi host failures on both management and resource clusters, as shown in Table 10. The SAP AAS1 and AAS2 are in the same rule, which instructs vsphere DRS to keep them running in separate vsphere ESXi hosts, giving flexibility for vsphere DRS to provide the availability and the protection level required at the same time. EMC VSI and VPLEX EMC VSI provides enhanced visibility into VPLEX directly from the vcenter GUI. The Storage Viewer and Path Management features are accessible through the EMC VSI tab, as shown in Figure 33. In this solution, VPLEX distributed volumes host the VPLEX_ DS1 to VPLEX _DS5 Virtual Machine File System (VMFS) datastore, and Storage Viewer provides details of the datastore s virtual volumes, storage volumes, and paths. LUNs that make up the datastore are 4 TB distributed RAID 1 VPLEX Metro volumes that are accessible via PowerPath, as shown in Figure

50 Figure 33. VSI Storage Viewer datastores Symantec ApplicationHA Symantec ApplicationHA adds a layer of application awareness to the core vsphere HA functionality offered by VMware virtualization technology. Key benefits include the following: Out of the box integration with the VMware vsphere vcenter and vsphere HA Full visibility and control over applications with the ability to start, stop, and monitor applications running inside virtual machines Standardized way to manage applications using a single interface that is integrated with VMware vsphere Client Specialized Application Maintenance mode, in which Symantec ApplicationHA allows you to intentionally take an application out of its execution for maintenance or troubleshooting Symantec ApplicationHA components Symantec ApplicationHA consists of the following components in a VMware virtualization environment: Symantec - High Availability Console The HA Console is installed separately in the Symantec ApplicationHA monitoring environment and resides on a separate virtual machine. Symantec - High Availability guest components for virtual machines The Symantec ApplicationHA guest components are installed separately on the virtual machines where you want to monitor applications. The guest components include the configuration wizard and the High Availability agents that are used for configuring and monitoring applications. The guest components also include the Veritas Storage Foundation Messaging Service. This service communicates the application monitoring status to the Symantec High Availability VM tab and dashboard tab. 50

51 Symantec ApplicationHA agents Agents are application-specific modules that plug into the High Availability framework that manages applications and resources of predefined resource types in a system. The agents are installed when you install Symantec ApplicationHA guest components. These agents start, stop, and monitor the resources configured for the applications and report state changes. If an application or its components fail, Symantec ApplicationHA restarts the application and its resources inside the virtual machine. Symantec ApplicationHA agents are classified as follows: Infrastructure agents Agents such as NIC, IP, and Mount are classified as infrastructure agents. Infrastructure agents are automatically installed as part of the Symantec ApplicationHA installation on virtual machines. Application agents The Symantec ApplicationHA agent pack is released on a quarterly basis. The agent pack includes support for new applications as well as fixes and enhancements to existing agents. You can install the agent pack on an existing HA guest components installation. Refer to the Symantec Operations Readiness Tools (SORT) website for information on the latest agent pack availability. Refer to the agent-specific configuration guide for more details about the application agents. Working with vsphere vcenter Server Symantec ApplicationHA communicates with vsphere HA. High Availability conveys the application health status in the form of an application heartbeat. This allows vsphere HA to automatically reset or restart a virtual machine if the application heartbeat is not received within a specified interval. Figure 34 displays the sample deployment of Symantec ApplicationHA. Figure 34. Sample deployment of Symantec ApplicationHA 51

52 Symantec ApplicationHA provides a vcenter plug-in for integration with the vsphere Client and adds the following interfaces to perform application monitoring tasks. The following interfaces appear in the vsphere Client after you install Symantec High Availability Console: Symantec High Availability VM tab: As shown in Figure 35, this tab is the primary interface for performing the application monitoring operations on a virtual machine. You configure application monitoring and then monitor and control the configured application on the virtual machines. After configuring the application monitoring, the Symantec High Availability VM tab displays the state of the application and the component dependencies. Figure 35. Symantec High Availability VM tab Symantec High Availability dashboard: This dashboard is the primary interface for administering the configured applications on a VMware environment. After configuring application monitoring, the High Availability dashboard displays the state of the application. Figure 36 displays the High Availability dashboard with the SAP and Microsoft SQL Server applications that are configured for monitoring the SAP systems. System administrators with the correct permissions can coordinate the start and stop of SAP services from the dashboard if required, as shown in Figure

53 Figure 36. High Availability dashboard In this solution, Symantec ApplicationHA is transparent to vcloud Director and is an optional component that adds important application-awareness to the vsphere Administrator for operations, as well as providing fast and automated restart of SAP and database services inside the virtual machines during process, virtual machine, and vsphere ESXi hosts tested outages. Symantec ApplicationHA provides support for several OS, DB and application combinations described in the compatibility lists referred in the Symantec Operations Readiness Tools (SORT) website. Symantec High Availability Console server was installed in a VMware virtual machine named SymantecHA, which was deployed in the management cluster and protected by vsphere HA with a high restart priority configured. According to Symantec ApplicationHA architecture, the guest agents and their updates are deployed in the Symantec High Availability Console server and distributed by pushing technology to the guest virtual machines where they are installed or updated. The Symantec ApplicationHA guest components were installed on all SAP virtual machines in the resource cluster. SAP and Microsoft SQL Server agents were configured to manage the automatic restart of the SAP and Microsoft SQL database services detected in a running state during the initial installation and activation of the agents and in case of service, process, guest OS, or virtual machine outages. In Windows environments, the SAP and Database Services can be restarted automatically during start up and service outages. Symantec ApplicationHA does that and also monitors the health of the database and communicates with vsphere HA. Symantec ApplicationHA also provides application health status in the Symantec High Availability VM tabs in the vsphere Client or web browser. 53

54 SAP system architecture Introduction This section describes the SAP system architecture deployed for the solution in the two datacenters. The SAP application layer uses these SAP components: SAP ERP 6.0 Core IDES Enhancement Package 5 SAP NetWeaver 7.02 SAP ABAP SAP Central Services (ASCS) SAP Database Services (DB) SAP ABAP Application Server 1 (AAS1) SAP Additional Application Server (AAS2) SAP Global File systems (\sapmnt, \usr\sap\<sid>, and \usr\sap\trans) All the SAP systems were installed and configured in a distributed architecture where we had separate virtual machines for the SAP central services, the database, and application servers. All SAP instances were installed on VMware vsphere virtual machines with SAP Applications running inside of them. Note: Other operating systems and database combinations supported by the SAP Product Availability Matrix (PAM) and the EMC Solutions Support Matrix (SSM) can also be used with this solution. Configuring SAP system The solution implemented a recommended distributed SAP system architecture, as shown in Figure 37. Figure 37. SAP system architecture 54

55 The enqueue server and message server were decoupled from the Central Instance and implemented as standalone services within the ASCS instance 11. Two SAP Application Servers (AAS1 and AAS2) (formerly known as dialog instances) were installed to provide redundant work processes such as dialog (DIA), background (BGD), update (UPD), update V2 (UP2), spool (SPO), and gateway, to provide workload balance and protection against vsphere ESXi host, virtual machine or guest OS failures. Additional SAP Application Servers can be deployed to provide additional redundancy of SAP work processes. Outage impact The SAP ASCS, SAP Database, and the SAP Global File systems server are single points of failure (SPOF) components in SAP NetWeaver distributed architecture without the traditional OS/DB specific and complex clustering technologies until recently. This solution addresses these SPOFs as described in the Key design considerations section. SAP Database Server outage The whole SAP system is stopped in case the SAP Database server component fails. Once the database becomes available again, the SAP work processes in the SAP Application Servers reconnect automatically and the users can resume their work. All transactions currently in progress and not committed are rolled back for consistency. SAP ASCS instance outage (enqueue and message server services) In the case of SAP ASCS instance component failure, either or both the enqueue and the message server services fail. Either way a quick restart is required to restore these services to resume the SAP system operations. If the enqueue service fails, the lock table stored in RAM is lost, the SAP application transaction locks stop, and the application is terminated. All open SAP transactions will be rolled back and the database stays consistent. When the enqueue server fails, the SAP functional team has to analyze the business process impact after this service is restarted. If the message server stops working, new requests cannot be executed for dialog, update and enqueue server, but the existing connections are not affected. SAP Global File systems (\sapmnt, \usr\sap\trans, and \usr\sap\<sid>) In the case of SAP Global File systems (Windows file share) component failure, the file shares \sapmnt\<sid>, \usr\sap\<sid>, and the \usr\sap\trans are not available. The \sapmnt directory not being available prevents additional SAP application servers to be started, but the already active ones continue to run. If the global file system \usr\sap\<sid>\sys\global\ is not available, batch and spool processes cannot write their logs there, so batch and print job activities will be canceled. If the \usr\sap\trans transport directories are not available, the change and transport system cannot be used. 11 The enqueue server manages logical locks, also known as the lock server. The objective of this server is to minimize the duration of a database lock. Unlike database locks, an SAP lock can exist across several database logical units of work (LUWs). The message server informs all servers (instances) in an SAP system of the existence of the other servers. Other clients (for example, SAPlogon and remote function call clients with load balancing) can also contact the enqueue server for information about load balancing. 55

56 Key design considerations The SAP system deployed for this solution implements the following key design features: The Database and ASCS instances were installed with virtual hostnames to decouple them from the fixed virtual machine hostnames and IPs. SAP patches, parameters, basis settings, and load balancing settings are all installed and configured according to the SAP installation guide and the SAP Notes listed in the References section. VMware best practices for SAP were adopted in this solution 12. The SAP update processes (UPD/UP2) were configured on the primary and additional application server instances. The SAP ASCS instance profile, start profiles, and dialog instance profiles were configured appropriately. The SAP ASCS instance also holds the SAP Global File systems \sapmnt, \usr\sap\<sid>, and \usr\sap\trans central directories, which are maintained by a central NFS server that restarts automatically in case of a virtual machine outage. The SAP Database server running Microsoft SQL Server was configured according to the SAP notes listed in the References section. SAP shared file systems, \sapmnt\<sid> and \usr\sap\<sid> were stored on the SAP ASCS server and shared to the all SAP AB1 virtual machines. All virtual disks in the SAP virtual machines were created in the thick format (eagerzeroed). Multiple virtual disks were configured to separate the I/O pattern in different virtual disks (VMDKs) and in different paravirtual SCSI (PVSCSI) controllers for example, SCSI Controller 1 Transaction Logs and SCSI Controller 2 Database Datafiles and so on. This solution uses the encapsulated and virtualized storage for the entire SAP environment. Each SAP virtual machine was stored in one VPLEX-based datastore as shown in Table 11, as required by vcloud Director. Separate datastores were used for high availability purposes. The datastores are based on VPLEX virtual volumes distributed across the both datacenters and presented to the vsphere ESXi hosts running the SAP virtual machines through EMC VPLEX Metro. All the SAP virtual machines in the solution were configured to be protected by Symantec ApplicationHA and VMware vsphere HA with restart priorities set, vsphere DRS with specific restart priorities, vsphere DRS with affinity and antiaffinity rules as described in Table 10. The DRS anti-affinity rules were created to protect the SAP virtual machines against vsphere ESXi host failures that trigger a virtual machine restart, protecting the SAP systems, specially the SPOFs with fast restarts and a timely and automated recovery of the SAP services to the end users. All SAP virtual machines had VMware tools installed and were configured with vmxnet3 network cards for best network performance. 12 For details about SAP Solutions on VMware vsphere, refer to High Availability and Best Practices Guide. 56

57 Because SAP allocates memory permanently and does not release it again, the memory reservation for all SAP system AB1 production-role virtual machines was set to the maximum size to avoid virtual machine paging and to optimize for best performance. Table 11 shows the configuration of the SAP virtual machines. DC represents datacenter in the table. Table 11. SAP virtual machines configuration DC vsphere ESXi host Virtual machine role vsphere HA restart priority Virtual CPUs Memory (GB) Disk (GB) Virtual machine name Datastore name 13 A C460-1 SAP DB Medium SAPAB1DB VPLEX_DS2 B C460-2 SAP ASCS High SAPAB1ASCS VPLEX_DS1 A C460-3 SAP Application SAPAB1AAS1 VPLEX_DS3 Low B C460-4 Servers SAPAB1AAS2 VPLEX_DS4 13 VMware vcloud Director consumes storage from the configured storage profiles based on capacity requirements. For demonstration purposes, the SAP virtual machines were placed on HA Gold Tier (EFDs based) specific datastores, as shown in Table

58 EMC storage infrastructure Introduction EMC VNX HA Overview This section describes the storage infrastructure for the solution: Next-generation VNX is the storage system at both datacenters. Two VNX storage systems are deployed with a matching LUN configuration. EMC VNX series provides five-nines availability in mission-critical business environments. VNX availability and redundancy features include: Mirrored write cache, where each storage processor contains both primary cached data for its LUNs and a secondary copy of the cache for its peer storage processor Battery backup to allow for an orderly shutdown and cache de-staging to vault disks to ensure data protection for possible power failures RAID protection levels 0,1,1/0,3,5, and 6 all of which can co-exist in the same array simultaneously to match different protection requirements Proactive hot sparing enhances system robustness and delivers maximum reliability and availability Redundant data paths, power supplies, drive connections, and storage processors all with non-disruptive field-replacement capabilities Continuous system monitoring, call home notification, and advanced remote diagnostics EMC VPLEX extends the continuous availability within and across datacenters. Configuring VNX EMC VNX Provisioning provides pool-based provisioning by implementing pool LUNs that can be either thin or thick. Thin LUNs provide on-demand storage that maximizes the utilization of storage by allocating storage as needed. In this solution, we created storage pools with different drive types and RAID configurations to take full advantage of FAST VP. We also selected Scheduled Auto-Tiering to enable FAST VP, as shown in Figure

59 Figure 38. Creating a storage pool in Unisphere We also created standard LUNs sizes of 4 TB that are presented from the VNX array to the VPLEX. Both VNX at datacenter A and datacenter B have similar configurations. Use Unisphere to create standard LUNs quickly and easily, as shown in Figure 39. Figure 39. Creating LUNs in Unisphere 59

60 Workload generation SAP standard SD Benchmark An SAP Standard Application Benchmark, the SAP ERP Sales and Distribution (SD) Benchmark, was used to generate a sample SAP workload on the installed SAP ERP 6.0 EhP 5 AB1system, which was built for two of the test scenarios. The SAP-specific workload used in the Testing and validation scenarios demonstrates that even under a heavy workload, this solution provides consistent uptime without compromising the availability of the SAP system. The SAP Standard SD Benchmark toolkit performed predefined standard SD transactions against the SAP AB1 system. The toolkit covers a sell-from-stock business scenario, which includes the creation of a customer sales order with five line items and its corresponding delivery with subsequent goods movement and invoicing. The scenario consists of the following transactions: (VA01)Create a sales order with five line items (VL01N) Create a delivery for the order (VA03) Display the customer order (VL02N) Change the delivery and post a goods issue (VA05) List 40 orders for one sold-to party (VF01) Create an invoice for the order SAP Standard SD Benchmark configuration SAP SD Benchmark Driver was installed on a virtual machine and ran the workload from a central location. To avoid data-locking situations, each benchmark user had its own master data. Table 12 presents the SD Benchmark configuration and workload, which is a typical customer activity. Table 12. SD Benchmark configuration Users per appl. server Ramp-up time minutes Benchmark run Think time 5 seconds Loops per user High-load phase run time Concurrent users Sales orders/min 2 10 minutes A benchmark run consists of a ramp-up phase where all users log on one by one, a highload phase where all users run their actions concurrently, and a ramp-down phase when all users log off. Figure 40 shows an example of the Benchmark run. Figure 40. Benchmark run example 60

61 Testing and validation Introduction The EMC validation team initially installed and validated the environment without any high-availability protection scheme. We then transformed the environment to the mission-critical business continuity HA and application mobility solution described earlier in this white paper. We carried out the following tests to validate different scenarios addressed in this solution and demonstrate the level of protection provided to the lab environment built. The testing and validation was divided between planned downtime and unplanned downtime test scenarios and from the less to the most disruptive, in order to demonstrate different benefits of this solution. Planned downtime a. vsphere ESXi host maintenance Unplanned downtime b. VMware virtual machine guest OS process failure with Symantec ApplicationHA c. vsphere ESXi host hardware failure with Symantec ApplicationHA d. EMC VPLEX isolation failure with SAP workload e. Datacenter failure Planned downtime Planned downtime can always be controlled. Reasons for planned downtime include hardware maintenance, hypervisor maintenance, installation of patches, upgrades of the server BIOS, and drivers or the hypervisor itself. These activities can cause downtime in your environment. However, you can plan the best time to perform these activities. This solution provides the value of Downtime Avoidance in which SAP or other workloads can be moved across datacenters without causing interruption or foreseeable downtime from an external event, such as a power upgrade planned by the utility company. a. Test scenario vsphere ESXi host maintenance The following test scenario validates that: No disruption occurs in the SAP application running inside the VMware virtual machines, under a SAP Benchmark sample workload, running on a vsphere ESXi host that requires a planned hardware or software maintenance. The EMC VPLEX Metro, vsphere HA, and vsphere DRS configurations described in earlier in this white paper ensure high protection levels during physical server maintenance. The SAP workload described in Workload generation was executed in this test scenario. This test scenario demonstrates the SAP application mobility across datacenters during the planned downtime, as shown in Figure

62 Figure 41. SAP application mobility across datacenters Objectives Verify that a vsphere ESXi host can be set to maintenance mode without interrupting the SAP application running in its virtual machines. Verify that the SAP virtual machines running under SAP workload on the vsphere ESXi host on maintenance mode located on Datacenter A, are moved across datacenters without interruption by using VMware vmotion and EMC VPLEX Metro. Verify that the vsphere DRS affinity and anti-affinity rules are enforced by vcenter during the non-disruptive migration of the SAP virtual machines from Datacenter A to Datacenter B. Testing procedure 1. Identify which vsphere ESXi hosts were running the SAP AB1 system virtual machines. 2. Start the SAP Standard SD Benchmark run and wait for the High Load phase. 3. Select the vsphere ESXi host c460-1, where the SAP AB1 Database (SAPAB1DB) and SAP Application Server 1 (SAPAB1AAS1) virtual machines were running and switched to maintenance mode. 4. Verify the non-disruptive migration of the SAP virtual machines from the vsphere ESXi host c460-1 to other vsphere ESXi hosts on Datacenter A and B. 5. Verify that the SAP system AB1system was not interrupted. 6. Verify that all vsphere DRS affinity and anti-affinity rules were followed. 62

63 Results Figure 42 summarizes the test environment and provides reference duration metrics in seconds obtained during the testing. The metric used was the Migration Time 14. Figure 42. Test environment The SAP servers were moved across two datacenters that were physically separate, but the SAP end users never experienced any service interruption, as verified in the SAP system logs (SM21 transaction) of all SAP Application Servers. Analysis The non-disruptive migration between two SAP virtual machines (SAPAB1 DB and SAPAB1 AAS1) was completed on average in 46 seconds after the c460-1 ESXi host switched to maintenance mode. The same test was repeated three times with the same conditions. The least favorable result was 59 seconds. This test scenario demonstrates that: A planned maintenance can be performed non-disruptively and on demand without affecting the planned downtime commitments on your current SLAs. vsphere DRS automatically enforced the affinity rules configured to keep the SAP virtual machines SAPAB1ASCS, SAPAB1DB, and SAPAB1AAS1 and AAS2 instances in separate physical vsphere ESXi hosts, maintaining the same protection level that existed before the first server was set to enter in maintenance mode. 14 Migration Time is the total time taken for the migration to complete, beginning from the initiation of the migration. For more information refer to VMware vsphere 5.1 vmotion Architecture, Performance and Best Practices Technical White Paper. 63

64 An administrator can proactively avoid the impact of a foreseeable outage, to preserve the availability of the business critical SAP applications, moving them live and within seconds to a remote datacenter, outside the area to be affected by the foreseeable outage. Unplanned downtime Unplanned downtime cannot be controlled or anticipated, and can result from occurrences such as hardware failure, software failure, datacenter failure, and natural disasters. HA is a term used often connected to unplanned downtime. HA is a complex topic, which covers designing a solution to make not only your software highly available, but the whole IT infrastructure underneath in all layers (networks, routers, power supply, storage and so on) must be designed with high availability in mind. b. Test scenario VMware virtual machine guest OS process failure with Symantec ApplicationHA This test scenario validates that an SAP system guest OS processes running inside virtual machines can be restarted quickly and automatically by Symantec ApplicationHA alone or in combination with VMware vsphere HA after a process failure. No SAP workload was applied in the SAP system AB1 Application Servers during the test. This test scenario demonstrates a guest OS process failure during the unplanned downtime, as shown in Figure 43. Figure 43. Guest OS process failure with Symantec ApplicationHA Objectives Verify that Symantec ApplicationHA agent detects and restarts the SAP database processes inside a virtual machine after an OS-triggered process failure. Verify that the SAP database services (Microsoft SQL Server agent and Microsoft SQL Server database) running inside the SAP virtual machine are restarted automatically by Symantec ApplicationHA agents. 64

65 Verify that Symantec ApplicationHA agent takes corrective action to restart the SAP database services (Microsoft SQL server Agent and Microsoft SQL server database) when a persistent failure affects the restart of database services. Testing procedure The testing process below was executed three times in the same scenario to validate the results obtained. This process was executed with two variations, a single process failure and a persisting process failure with graceful shutdown, which are described as follows. 1. Identify the SAP Database instance that is specific to the running Microsoft SQL Server process and collect that process ID. 2. Verify the status of the SAP Database instance from the vsphere Client Symantec High Availability VM tab. 3. Open the Windows Task Manager on the guest OS of the virtual machine where the SAPAB1DB (SAP system AB1 Database server) is running and trigger an End Process Tree to the Microsoft SQL Server database process. 4. Verify that Symantec ApplicationHA agent detects the process outage and take actions to automatically restart the Microsoft SQL Server database processes in a timely manner without any manual intervention. 5. Verify that the SAP AB1 database server processes are restarted automatically and that the SAP AB1 system Application servers can reconnect to the SAP AB1 system Database instance. 6. Repeat step 3 to verify that Symantec ApplicationHA agents restart the SAP Database services and try to minimize the downtime 15. Results Single process failure testing variation Figure 44 summarizes the single SAP Database process failure set of tests performed and provides the duration metrics obtained during the three testing cycles. The metrics are shown in minutes and seconds (mm:ss). Figure 44. Single SAP database instance (Microsoft SQL Server) process failure Figure 44 depicts the durations of Microsoft SQL Server database restarting and SAP work processes reconnecting across the three testing cycles. 15 This step is applicable only for the variation Persisting Process Failure Testing, since the guest OS restart is only invoked after the number of process restart attempts cross the value configured in the Symantec ApplicationHA parameter App.RestartAttempts 65

66 The values from the column Microsoft SQL Server Restart (mm:ss) were obtained from the Symantec ApplicationHA cluster log file (c:\program Files\Veritas\cluster server\log\engine_a.log), which contains all of the activities executed by the Symantec ApplicationHA single cluster after the process failure occurrs. This log file lists the detailed actions taken and the output of database instance restart. Symantec ApplicationHA agent automatically executed the following processes after a process failure: 1. Detects the process failure. 2. Cleans up all other dependent Microsoft SQL Server processes. 3. Restarts the Microsoft SQL Server agent and database. The total duration of the execution of the above three processes was 24 seconds on average, considering all times obtained in all three tests cycles performed. This duration may vary based on the recovery process of the Microsoft SQL Server database at restart. The SAP AB1 system Application Servers took on average 34 seconds, according to the SAP System Log to reconnect their work process to the SAP AB1 system database (***LOG BYY=> work process left reconnect state ) after manually initiating database failure. The Symantec ApplicationHA agent for Microsoft SQL Server was configured to allow two restart attempts (App.RestartAttempts) to allow a single process failure testing without invoking the guest OS or vsphere HA to restart the OS or the virtual machine. Persisting process failure testing Figure 45 provides reference duration metrics obtained during the testing performed for the persisting SAP Database process failure with graceful shutdown. The metrics are shown in minutes and seconds (mm:ss). Figure 45. Persisting SAP database (Microsoft SQL server) process failure Figure 45 lists the durations of persisting process failures with the OS restart and the SAP work processes reconnection across three testing cycles. Symantec ApplicationHA agent automatically executed the steps below after the manual trigger of the first process failure: 1. Detects the process failure. 2. Cleans up all other dependent Microsoft SQL Server processes. 3. Restarts the Microsoft SQL Server processes. 66

67 Symantec ApplicationHA automatically executed the following steps below after the manual trigger of the second process failure: 4. Detects the process failure. 5. Cleans up all other dependent Microsoft SQL Server processes. 6. Stops Microsoft SQL Server agent service. 7. Triggers a graceful shutdown of the guest OS. 8. Stops the virtual machine application heartbeat to notify vsphere HA and triggers a guest OS shutdown and restart. The scenario durations of 3 minutes and 6 seconds on average were observed for the previous eight processes and the following four processes. Restarts the guest OS. Starts Symantec ApplicationHA cluster and agents. Starts Microsoft SQL Server database. Starts Microsoft SQL Server agent. The Symantec ApplicationHA agent for Microsoft SQL Server was configured for one restart attempt (App.RestartAttempts), which forced a second process failure to invoke the restart of the guest OS. The Symantec ApplicationHA parameter (VM.GracefullRebootPolicy) was set to enabled to execute a graceful shutdown of the operating system, before invoking a restart of the virtual machine. Note: Symantec ApplicationHA parameter VM.GracefullRebootPolicy is used to enable or disable Symantec HA-initiated virtual machine restart policy. This option determines whether or not Symantec HA restarts the virtual machine in response to application and component failures. When a configured component application fails, Symantec HA attempts to restart the failed components. If the component fails to start, Symantec HA then takes the next corrective action. If this policy is disabled, and a component fails, then Symantec HA stops sending the heartbeat to VMware HA. VMware HA then restarts the virtual machine. If this policy is enabled, Symantec ApplicationHA invokes a native operating system command to restart the virtual machine. For more information about the SAP NetWeaver database reconnect mechanism, refer to the SAP Note Database Reconnect: Architecture and function listed in the References section. 67

68 Analysis Figure 46 summarizes the SAP Database instance process failures durations obtained in this test scenario performed and provides reference duration metrics obtained during the testing for comparison reasons. The metrics are shown in minutes and seconds (mm:ss). Figure 46. Test-result summary of SAP Database instance process failures This test scenario demonstrated the following: Symantec ApplicationHA restarted the failed SAP Database Instance quickly, automatically and without any manual intervention of the SAP administrator or DBA, even after a persisting SAP critical database instance process failure. Using Symantec ApplicationHA agents to recover from a single process failure inside the guest OS of a virtual machine is faster than executing a guest OS shutdown and then subsequently restarting it, due the wait required to load the guest OS and the applications services to restart correctly and resume the SAP services. After a persistent process failure, the SAP WP reconnect time with a graceful shutdown durations obtained during the tests, indicated that the risk of a guest OS corruption is minimized when the parameter VM.GracefullRebootPolicy was set to enabled. This parameter enables a graceful shutdown and automatic restart of the guest OS and the Microsoft SQL Server services after a persistent process failure inside the guest OS. Symantec ApplicationHA agents restarted the SAP database services (Microsoft SQL server AB1 database and Microsoft SQL server agent) to cleanly, quickly, and automatically recover from a severe incident, minimizing the total downtime of the SAP AB1 system. This significantly reduced the mission-critical SAP application downtime and the potential costs associated with the downtime. Symantec ApplicationHA provides application monitoring, adding a new level of resiliency, integrating the virtualization layer with the application layer. Symantec ApplicationHA demonstrates that the application services were restarted in the correct order, avoiding the common problems associated with manual restarts and maintenance of SLAs. Symantec ApplicationHA provides the automation required to start SAP application and database services in case of outages. c. Test scenario vsphere ESXi host hardware failure with vsphere HA and Symantec ApplicationHA This test scenario validates that the management services and the SAP database services running inside the guest OS of a virtual machine can be restarted quickly and correctly in an automated manner using a combination of the vsphere HA, vsphere DRS, 68

69 and Symantec ApplicationHA technologies after a simulated vsphere ESXi host failure, as shown in Figure 47. Figure 47. VMware vsphere ESXi host failure with vsphere HA and Symantec ApplicationHA Objectives Verify that vsphere HA will restart the virtual machines of the surviving vsphere ESXi hosts in the management and resource clusters after a vsphere ESXi host simulated failure. Verify that the SAP database services (Microsoft SQL server database and Microsoft SQL server agent) running inside the SAPAB1DB virtual machine are restarted automatically by Symantec ApplicationHA agents after the virtual machine is restarted by vsphere HA. Verify that the vsphere DRS anti-affinity rules are enforced during the restart of the virtual machines for both management and resource clusters. Testing procedure Execute the following steps three times in the exact same conditions to validate the results: 1. Identify the vsphere ESXi hosts in which each management and SAP virtual machines were running. 2. Identify the vsphere ESXi hosts in which the SAP system AB1 database and ASCS instances virtual machines were running. 3. Select the vsphere ESXi host where the vcloud Director virtual machine of the management cluster is running and power it off from the hardware management console. 69

70 4. Select the vsphere ESXi host where the SAPAB1DB virtual machine of the resource cluster is running and power it off from the hardware management console. 5. Verify that vsphere HA quickly and automatically restarts the vcloud Director virtual machine from one vsphere ESXi host 1 over to ESXi host Verify that the SAPAB1DB virtual machine restarts by using vsphere HA and the Microsoft SQL Server database services also automatically and correctly restart by using Symantec ApplicationHA agents. 7. Verify that all vsphere DRS anti-affinity rules were enforced on both the management and resource clusters. Symantec ApplicationHA agents were only installed and configured on the SAP virtual machines for this solution. Results Figure 48 summarizes the test environment described above and provides reference duration metrics obtained during the testing. The metrics are shown in minutes and seconds (mm:ss). Figure 48. ESXi failure of resource and management clusters As shown in Figure 48, vsphere HA restarted the vcloud Director virtual machine in the same datacenter from the vsphere ESXi host r710b to the vsphere ESXi host r710a after the failure (physical server powers off). vsphere HA restarted the SAP database virtual machine (SAPAB1DB) that is running on the vsphere ESXi host c460-1 on Datacenter A, across datacenters, on the vsphere ESXi 70

71 host c460-2 on Datacenter B, after the failure (physical server power off) of the vsphere ESXi host c460-1, as shown in Figure 49. The virtual machine restart time observed was an average of 50 seconds, as shown in Figure 48. Figure 49. Events pane in vcenter web client Symantec ApplicationHA agent restarted the SAP database services (Microsoft SQL Server agent and database) inside the guest OS of the SAP AB1DB virtual machine in an average of 3 minutes and 36 seconds, as shown in Figure 48. vsphere DRS, based on the configured affinity rules described in Table 10 and identified that the SAPAB1DB virtual machine was restarted by vsphere HA on the vsphere ESXi host c460-2 where the SAPAB1ASCS virtual machine was also running. Then VMware vmotion non-disruptively migrated the SAPAB1ASCS virtual machine from the vsphere ESXi host c460-2 to the vsphere ESXi host c460-3 on Datacenter B, as shown in Figure 50. Figure 50. Events pane in vcenter web client Analysis This test scenario demonstrates the following: vsphere HA restarted the management services quickly, automatically, without any manual intervention after a vsphere ESXi host failure. vsphere HA combined with Symantec ApplicationHA restarted the SAP Database services quickly, automatically, and across datacenters without any manual intervention after a severe vsphere ESXi host failure. vsphere DRS affinity rules were enforced after the failure with the VMware vshield Edge gateway0 virtual machine being restarted on Datacenter A, as configured in the vsphere DRS rules to keep each vshield Edge gateways on their own datacenter for possible failures. vsphere DRS identified that affinity rules were violated having the SAP Database virtual machine and the SAPAB1ASCS virtual machine running on the same vsphere ESXi host. Affinity rules were then enforced, triggering a VMware vmotion migration of the SAPAB1ASCS virtual machine to another vsphere ESXi host, which restored the previous level of protection provided before the vsphere ESXi host failure. Even after the failure, keeping the SAP virtual machines in separate vsphere ESXi hosts provided resiliency. Symantec ApplicationHA automatically restarted the SAP database services (Microsoft SQL Server agent and database) after vsphere HA restarted the operating system, recovering from the incident and minimizing the downtime incurred to the SAP AB1 system. 71

72 d. Test scenario VPLEX cluster isolation failure with SAP workload This test scenario validates that, when an isolation failure of a VPLEX cluster occurs, vcloud Suite, vcenter, the SAP applications, and the database continue operation without interruption. The SAP workload described in Workload generation was executed in this test scenario. Note: EMC VPLEX Witness, EMC Cross-Cluster Connect, and an inter-site link with 1 millisecond round trip time was used in this test scenario. To test this failure scenario, we simulated the isolation of a specific cluster on Datacenter A, with both the external management IP network and the VPLEX WAN communications network partitioned. The LAG network remained available during the test. VPLEX Witness on Datacenter C and VPLEX cluster-2 on Datacenter B also remained available and in communication, as shown in Figure 51. Figure 51. Datacenter A VPLEX cluster isolation failure Objectives Verify that all SAP virtual machines on Datacenter A continue to run non-disruptively through the EMC VPLEX counterpart in Datacenter B, through its Cross-Cluster Connect when an isolation failure of a VPLEX cluster occurs on Datacenter A. Verify that the SAP workload continues to run without interruption by the EMC VPLEX storage failure. 72

73 Testing procedure 1. Start the SAP Standard SD Benchmark run and monitor until it reaches the high load phase. 2. Simulate isolation of the preferred cluster on Datacenter A, with both the external management IP network and the VPLEX WAN communications network partitioned. The LAG network remains available. 3. Check PowerPath/VE status. The paths to Datacenter A VPLEX storage should be dead, but the paths to the VPLEX storage on Datacenter B should be alive and be serving the vsphere ESXi hosts on Datacenter A. 4. Verify the VPLEX functionality; check that the storage access was not lost. 5. Verify the status of SAP SD Benchmark and check that the processes were not stopped or interrupted at any point in time during the storage failover to Datacenter B. Results Table 13 shows the expected and observed behaviors of the system when the VPLEX at Datacenter A was isolated. Table 13. Expected and observed behaviors Virtual machines on Datacenter A management cluster Virtual machines on Datacenter A resource cluster System name On r710a: vchargeback ssenet (DNS server) On r710b: vcloud Director On c460-1: SAPAB1DB On c460-3: SAPAB1ASCS vshield Edge Status prior to VPLEX isolation Expected behavior Observed behavior Available Available Available Available Available Available VPLEX cluster VPLEX1 Datacenter A cluster-1 VPLEX2 Datacenter B cluster-2 VPLEX witness Available Unavailable Available Available Unavailable Available Available SAP services Database, Enqueue, and Message Servers Available Available Available Analysis When VPLEX on Datacenter A became isolated, VPLEX Witness ensured that the consistency group detach rule, which defines cluster-1 as the preferred cluster, was overridden and the storage served by VPLEX cluster-2 on Datacenter B remained available, as shown in Figure

74 Figure 52. VPLEX status after VPLEX isolation on Datacenter A Figure 53 shows the behavior of EMC PowerPath during the isolation event. Prior to the isolation, PowerPath sets the Cross-Cluster paths for the hosts on Datacenter A to automatic standby proxy. When the isolation event occurs, PowerPath is able to detect the isolation of the VPLEX LUNs on Datacenter A and service I/O on the standby paths. When the cluster is recovered, PowerPath automatically recovers the dead paths. For details of VPLEX Cross-Cluster Connect, refer to VMware deployments on VPLEX Metro. Figure 53. PowerPath status from after VPLEX isolation on Datacenter A Because the vsphere ESXi hosts are connected to both VPLEX clusters in each datacenter, vsphere ESXi with EMC PowerPath/VE simply routes the I/O to the alternate path, which is available since VPLEX is configured with a distributed volume protected by VPLEX Witness. In both the management and resource clusters, the vsphere ESXi hosts on Datacenter A remain available and all virtual machines remain active because of VPLEX Metro HA Cross-Cluster Connect. 74

75 The SAP AB1 system keeps running without interruption, as shown in Figure 54. Figure 54. SAP AB1system running during after VPLEX cluster isolation on Datacenter A This test scenario demonstrates the following: Even after VPLEX isolation, the SAP SD Benchmark workload is not interrupted during the transparent failover of the storage services from the VPLEX from Datacenter A to Datacenter B. EMC PowerPath/VE identifies the failed paths and re-directs the FC traffic through the active paths, which are connected to the VPLEX storage on Datacenter B. This makes the VPLEX failure transparent to the SAP application running inside the SAP virtual machines on Datacenter A, as well as avoiding the virtual machines to be restarted on other ESXi hosts, avoiding a downtime. 75

76 e. Test scenario datacenter failure This test scenario validates that, in the event of a complete datacenter failure, all virtual machines running on one datacenter in both management cluster and resource cluster are restarted quickly, with the correct sequence on surviving datacenter. To test this failure scenario, we simulated a complete failure of Datacenter A, including VPLEX cluster, ESXi host, and network. The VPLEX Witness remained available on Datacenter C. VPLEX cluster-2 remained on Datacenter B to communicate with the VPLEX Witness on Datacenter B, as shown in Figure 55. Figure 55. Complete failure of Datacenter A Objectives Verify that the standby vshield Edge gateway on Datacenter B can take over and resume the networking services. Verify that all virtual machines on both management and resource clusters are restarted on Datacenter B. Verify that the configured DRS affinity rules are followed and kept in compliance. Testing procedure 1. Shut down all the ESXi hosts, switches, and VPLEX cluster-1 in Datacenter A to simulate a complete failure. 2. Verify that the VPLEX cluster-2 on Datacenter B remains available. 3. Execute a continuous ping command from the SAPAB1AAS2 system running on Datacenter B to verify that the standby vshield Edge gateway on Datacenter B takes over the network service. 4. Verify that vsphere HA detects the ESXi hosts failure on Datacenter A. 76