IBM Flex System Reference Architecture for Microsoft SQL Server 2012 High Availability using AlwaysOn Availability Groups

Transcription

1 IBM Flex System Reference Architecture for Microsoft SQL Server 0 High Availability using AlwaysOn Availability Groups July 0 Author(s): Michael Lawson

2 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Table of contents Introduction... Business problem and business value... Requirements... Primary Functional requirements... Normal operation... High Availability... Disaster Recovery... 6 Variations of the design... 7 Secondary functional requirements... 7 Read-able Secondaries... 8 Rolling Upgrades... 8 Storage Migration... 8 Multi-subnet Support... 8 Licensing... 8 Automatic Page Repair... 8 Hardware Overview... 8 IBM PureFlex System and IBM Flex System... 8 IBM Flex System x0 Compute Node... 9 IBM Flex System V7000 Storage... 0 IBM Storwize V IBM Storwize V IBM Flex System EN09 Switches... IBM Flex System FC7 Switches... Architectural overview... Component model... Operational model... Physical Layer... 6 Virtual Layer... 0 SQL Server Layer... Deployment considerations... Systems management... Server / Compute Nodes... Networking... Storage integration... Performance Considerations... 6 Best practices and limitations... 7 Other considerations... 7 About the author... 8 Acknowledgements... 8 Appendix : Bill of Material... 8 SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

3 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Resources... Trademarks and special notices... Document history Revision history Date of this revision: July 0 Date of next revision (date) Revision Revision Number Date (.0) ( June 0) Summary of Changes Initial draft Changes marked (N) SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

4 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Introduction This document describes the IBM Flex Reference Architecture for Microsoft SQL Server 0 High Availability using AlwaysOn Availability Groups. IBM reference architecture offerings create virtually turnkey solutions built around the latest IBM x86 servers, storage and networking, literally taking the complexity out of the solution. This IBM reference architecture combines Microsoft software, consolidated guidance and validated configurations for compute, network, and storage. The intended audience of this document is IT professionals, technical architects, sales engineers, and consultants to assist in planning, designing and implementing the IBM Flex Reference Architecture for Microsoft SQL Server 0 High Availability using AlwaysOn Availability Groups. Business problem and business value These following sections outline the value proposition of this solution. Business problem For customers who have selected the Microsoft SQL Server 0 relational database management system, this reference architecture provides a solution which makes SQL Server highly available within a main data center and provides disaster recovery in a remote data center, all using standard hardware and software. Business value This solution leverages the new AlwaysOn Availability Groups feature of SQL Server 0, the IBM Flex System x0 compute nodes, Flex System storage node, Flex System networking components and the IBM Storwize V7000 and V700 storage systems. The AlwaysOn availability group feature is a high availability solution and disaster recovery solution for SQL Servers that offer an alternative to database mirroring. This solution features SQL Server synchronous replication with automatic failover between two compute nodes in the main data center, with no data loss. In addition, it features SQL Server asynchronous replication with manual failover between two compute nodes, one in the main data center and one in a remote data center, with possible data loss, in case of disaster at the main data center. Not only are the compute nodes redundant, but so is the storage, for higher uptime and lower risk of data loss. In the past, this functionality required specialized hardware and software. This solution uses standard hardware and software. This paper provides the configuration steps and best practices to implement the solution. Requirements The following section describes the primary purpose of this solution, as well as some additional benefits. Primary Functional requirements This section describes the high availability and disaster recovery capabilities of this solution. This solution leverages the new AlwaysOn Availability Groups feature of SQL Server 0. This solution is flexible and SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

5 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA can be modified in a variety of ways to suit the customer s environment. In Figure below, three servers are shown, each with their own dedicated storage. Two servers are co-located in the Main Data Center (MDC) and one server is located in a remote Disaster Recovery (DR) site, connected by Ethernet networking. Main Data Center(MDC) Disaster Recovery (DR) Server Server Server Synchronous primary secondary secondary Asynchronous Figure : Normal operation, AlwaysOn Availability Group spanning the Main Data Center and DR Site Normal operation Figure shows the normal operating mode. Server has the updatable primary replica of the database. Users connect to this server to update the database. Changes are replicated synchronously to Server and asynchronously to Server using SQL Server replication. Server waits to receive confirmation that the change has been reliably received on Server. Since the servers are physically close, the additional latency for synchronous replication is low. Server does not wait for confirmation from Server, so even though there may be a longer latency (due to the remote distance to the DR Site), which does not affect the latency experienced by the users performing updates on Server. This arrangement creates three copies of the database, two of which are synchronized and one of which is nearly synchronized. High Availability If Server or its storage were to fail, Server and its storage would take over the role of updatable primary replica (shown in Error! Reference source not found.). This would be automatic. With the proper configuration (prior to the failure), users connected to Server would be automatically connected to Server the next time the application they are using attempts to connect to the database. Configured correctly, this can make the outage of Server, transparent to the users. No data would be lost, since the updates from Server are synchronously transferred to Server. After the failure of Server, the configuration is vulnerable to data loss because the replication to the DR site is asynchronous. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

6 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Main Data Center(MDC) Disaster Recovery (DR) Server Server Server primary secondary Asynchronous Figure : Automatic failover to Server in Main Data Center Disaster Recovery If the Main Data Center has a catastrophic failure and all components fail, Server can be manually configured to take over the role of updatable primary replica (shown in Figure ). These would be preplanned steps, executed manually, after the failure of the Main Data Center has been confirmed. There is some possibility of data loss, because the updates from Server (or Server ) are asynchronously transferred to Server. Not all the updates made on Server may have been transferred to Server before the Main Data Center failed. With the proper configuration, users connected to Server would be automatically connected to Server the next time the application they are using attempts to connect to the database. Users might experience two things in this scenario. First, if they entered data that was lost, they would need to re-enter the lost data. Second, depending on how long the manual failover takes, there may be a short, but noticeable period when their application is not available. After the failure of both servers in the MDC the configuration is vulnerable to data loss and extended outage because only one copy of the data and only one server have survived. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 6

7 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Main Data Center(MDC) Disaster Recovery (DR) Server Server Server primary Figure : Manual failover to DR site Variations of the design This basic design can be modified in a variety of ways. Below are some of the possible variations on the example design presented in this paper. Additional replicas The AlwaysOn Availability Groups feature allows for up to four secondary replicas; two synchronous and two asynchronous. Combine with other HA features Other SQL Server high availability features, such as, Failover Clustered Instances, can be combined with AlwaysOn Availability Groups. This means that four, five, or more servers could be utilized to create higher availability. Multiple databases The concept of a replica collects together one or more databases, so, in the event of a failover, multiple databases failover together. Virtualization variations The design of this paper uses Hyper-V Virtual Machines (VM) and SQL Server is installed within each VM. However, it is also possible to install SQL Server directly on the physical servers. Pass-through disks In this design, the storage has also been provisioned using VHDX formatted virtual disks. However, the storage could also be implemented on the physical disks using pass-through disks. Secondary functional requirements In addition to the main purpose of the design, which is to provide redundancy in the event of various failures, there are other benefits. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 7

8 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Read-able Secondaries The databases connected to Server and Server can be used for read-only tasks, including reporting, database snapshots, database integrity checks and backups. Using this capability can reduce the performance load on the primary replica. Rolling Upgrades When upgrading the software or hardware on the servers, this can be done on the secondary servers first, and then after failing over the primary, on the (old) primary. This can reduce the users downtime during an upgrade. Storage Migration Like rolling upgrades, migrating the database to new storage can be done in a way that shortens downtime for the users, by first making the new copy a secondary replica, and then switching over to it when ready. Multi-subnet Support This solution paper demonstrates how the server in the DR site can be on a different subnet, which supports a remotely located disaster recovery site. Licensing You are allowed one Passive Secondary server that you do not need to license. Automatic Page Repair If SQL Server detects that a database page is corrupted, the replication topology will transfer a good copy of the page to the server with the corrupted copy and repair it. Hardware Overview This section describes the IBM hardware used for the reference architecture. IBM PureFlex System and IBM Flex System IBM PureFlex System is a comprehensive infrastructure system that provides an expert integrated computing system. It combines servers, enterprise storage, networking, virtualization, and management into a single structure. Its built-in expertise enables organizations to manage and deploy integrated patterns of virtual and hardware resources through unified management. These systems are ideally suited for customers who want a system that delivers the simplicity of an integrated solution while still able to tune middleware and the runtime environment. PureFlex System uses workload placement based on virtual machine compatibility and resource availability. Using built-in virtualization across servers, storage, and networking, the infrastructure system enables automated scaling of resources and true workload mobility. PureFlex System has undergone significant testing and validation so that it can mitigate IT complexity without compromising the flexibility to tune systems to the tasks businesses demand. By providing both flexibility and simplicity, PureFlex System can provide extraordinary levels of IT control, efficiency, and operating agility. This combination enables businesses to rapidly deploy IT services at a reduced cost. Moreover, the system is built on decades of expertise. This expertise enables deep integration and central management of the comprehensive, open-choice infrastructure system. It also dramatically cuts down on the skills and training required for managing and deploying the system. The streamlined management SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 8

9 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA console makes it easy to use and provides a single point of control to manage your physical and virtual resources (with KVM now. Hyper-V support is planned) for a vastly simplified management experience. Figure : Front and rear view of the IBM Flex System Enterprise Chassis The hardware used in this paper is IBM Flex System. IBM Flex System takes the components of the PureFlex System and offers them à la carte allowing customers to custom-build their own infrastructure. This gives customers to ability to purchase exactly what they need, with the option to move to the PureFlex System at a later date. IBM Flex System x0 Compute Node IBM Flex System x0 Compute Node, an element of the Flex System, provides outstanding performance for your mission-critical applications. Its energy-efficient design supports up to 6 processor cores and 768 GB of memory capacity in a package that is easy to service and manage. With outstanding computing power per watt and the latest Intel Xeon processors, you can reduce costs while maintaining speed and availability. Highlights Optimized for virtualization, performance and highly scalable networking Embedded IBM Virtual Fabric allows IO flexibility Designed for simplified deployment and management To meet today s complex and ever-changing business demands, the x0 compute node is optimized for virtualization, performance and highly scalable I/O designed to run a wide variety of workloads. The IBM Flex System x0 is available on either your PureFlex System or IBM Flex System solution. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 9

10 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Figure : IBM Flex System x0 Compute Node More information about the IBM Flex System x0 compute node can be found in Resources at the end of the document. IBM Flex System V7000 Storage IBM Flex System V7000 Storage combines best-of-breed storage development with leading /0 Gb iscsi, FCoE, or FC host interfaces and SAS/SSD drive technology. With its simple, efficient and flexible approach to storage, the Flex V7000 Storage is a cost-effective, complement to IBM Flex System. By offering substantial features at a price that fits most budgets, the Flex V7000 delivers superior price/performance ratios, functionality, scalability and ease of use for the mid-range storage user. The Flex V7000 storage offers: Automate and speed deployment with integrated storage for the IBM PureFlex System or IBM Flex System Simplify management with an integrated, intuitive user interface for faster system accessibility Reduce network complexity with FCoE and iscsi connectivity Store up to five times more active data in the same disk space using IBM Real-time Compression Virtualize third-party storage for investment protection of the current storage infrastructure Optimize costs for mixed workloads, with up to 00 percent better performance with solid-state drives (SSDs) using IBM System Storage Easy Tier Improve application availability and resource utilization for organizations of all sizes Support growing business needs while controlling costs with clustered systems Get innovative technology, open standards, excellent performance, and a broad portfolio of proven storage software, hardware and solutions offerings from IBM IBM System Flex V7000 Storage (Figure ) is well-suited for Microsoft virtualized cloud environments. The Flex V7000 Storage complements the IBM Flex System Enterprise Chassis, Flex CN09 Converged Network switches, and x0 compute nodes in an end-to-end Microsoft Hyper-V private cloud solution by delivering proven disk storage in flexible, scalable configurations. Connecting optional EXP00 enclosures to your Flex V7000 Storage can scale up to 0 SAS and SSD disks and up to 960 per clustered system. The Flex V7000 Storage has 8GB cache per controller and 6GB for the whole system. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 0

11 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA The IBM System Flex V7000 Storage comes with advanced features such as System Storage Easy Tier, IBM Flashcopy, internal virtualization and thin provisioning, data migration, system clustering. Optional features include Remote Mirroring, Real-time Compression, and external virtualization. Figure 6: IBM System Flex V7000 Storage IBM Storwize V7000 The IBM Storwize V7000 disk system is a multi-faceted solution that consists of both hardware and software components. The modular hardware enclosures include integrated drives of varying form factors, including both hard disk drives (HDD) and solid-state drives (SSD). The solution also provides external storage virtualization, making it possible to integrate with and manage heterogeneous storage along with the Storwize V7000 storage as a single resource. The Storwize V7000 system is designed to allow quick and efficient storage deployment, thanks to an easy to use Graphical User Interface (GUI), integrated drives, and interoperability with nearly any backend SAN attached storage. The web-based GUI runs on the Storwize V7000 system so there is no longer a separate console server or management software installation required. Highlights A single user interface to manage and virtualize internal and third-party storage that can improve storage utilization Built-in tiering and advanced replication functions are designed to improve performance and availability without constant administration Single user interface simplifies storage administration to allow your experts to focus on innovation Figure 7: IBM Storwize V7000 Control Enclosure SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

12 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA V7000 system details V7000 enclosures support up to twenty-four.-inch drives or up to twelve.-inch drives. Control enclosures contain drives, redundant dual-active intelligent controllers, and dual power supplies, batteries and cooling components. Expansion enclosures contain drives, switches, power supplies and cooling components. You can attach up to nine expansion enclosures to a control enclosure supporting up to 0 drives. The system also supports intermixing.-inch and.-inch type controller and expansion enclosures. Key system characteristics are: Internal storage capacity: up to 6 TB of physical storage per enclosure Drives: SAS HDDs, near-line SAS HDDs and solid-state drives can be mixed in an enclosure to give you extraordinary flexibility Cache memory: 6 GB cache memory (8 GB per controller) as a base feature designed to improve performance and availability More information about the Storwize V7000 can be found in Resources at the end of the document. IBM Storwize V700 IBM Storwize V700 is an entry-level addition to the IBM Storwize family of disk systems, and delivers efficient configurations specifically designed to meet the needs of small and midsize businesses. The system shares the same integrated easy-to use web interface as the XIV and other Storwize systems. The internal disk storage virtualization enables rapid, flexible provisioning and simple configuration changes. IBM Storwize V700 offers advanced hardware and software capabilities usually found in more expensive systems, including: Redundant, battery backed dual controllers Up to 6GB cache, 8 per controller (GB per controller standard) Redundant, hot-swappable power supplies and fans Dual-port, hot-swappable 6 Gb SAS disk drives Support for RAID 0,,,6 and 0 and up to 80 TB of capacity Gb iscsi, optional 0 Gb iscsi/fibre Channel over Ethernet (FCoE) or 8 Gb Fibre Channel Disk storage scales up to 0.-inch disk drives or 60.-inch disk drives with four expansion units. Each unit is a packaged in a compact U 9-inch rack-mount enclosure. The modular design allows a business to start small and hot add additional enclosures as needed and without any downtime. The system also supports SSD drives and IBM Easy Tier automated hot data migration as an optional upgrade feature. As with earlier Storwize systems, advanced data protection and migration features include: Non-disruptive data migration Internal virtualization and thin provisioning Remote mirroring Integrated IBM FlashCopy snapshot technology Reduce power consumption with energy-saving features Advanced upgrades like FlashCopy (more targets) and Easy Tier SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

13 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Figure 8: IBM Storwize V700 Control Enclosure IBM Flex System EN09 Switches The IBM Flex System Fabric EN09 and EN09R 0Gb Scalable Switches provide unmatched scalability and performance, while also delivering innovations to help address a number of networking concerns today and providing capabilities that will help you prepare for the future. These switches are capable of supporting up to sixty-four 0 Gb Ethernet connections while offering Layer / switching. They are designed to install within the I/O module bays of the IBM Flex System Enterprise Chassis. These switches can help clients migrate to a 0 Gb or 0 Gb Ethernet infrastructure and offer virtualization features like Virtual Fabric and VMready, plus the ability to work with IBM Distributed Virtual Switch 000V. Figure 9: IBM Flex EN09 Switch IBM Flex System FC7 Switches The IBM Flex System FC7 8Gb SAN Switch is a full-fabric Fibre Channel component with expanded functionality. The SAN switch supports high speed traffic processing for IBM Flex System configurations and offers scalability in external SAN size and complexity, and enhanced systems management capabilities. The IBM Flex System FC7 8 Gb Pass-thru supports a fully interoperable solution for seamless integration of the Fibre Channel initiators to an existing fabric. The pass-thru module uses industry-standard N_Port ID virtualization (NPIV) technology to provide a cost-effective connectivity solution for the IBM Flex System chassis. 8Gb FULL FABRIC Mgmt 0 Figure 0: IBM Flex FC7 Switch Architectural overview The architectural diagram (Figure ) shows a high-level view of the complete high availability solution. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

14 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA V7000 SAN Array VMDX WAN Users FLEX (MDC) FLEX V7000 SAN Array VMDX V700 SAN Array VMDX FLEX (DR) Hyper-V Hyper-V VM VM SQL SQL Hyper-V VM SQL Server AlwaysOn Availability Group SQL Windows Failover Cluster Figure : Complete high availability solution architecture The solution is composed of two IBM Flex Systems (each with an external storage subsystem), which are located in two separate data centers; the Main Data Center (MDC) and the Disaster Recovery (DR) site. The two data centers have connectivity across a Wide Area Network (WAN). The Flex System in the MDC has two physical servers (x0 compute nodes) and one integrated Flex System V7000 Storage unit. One x0 is connected to the Flex System V7000 Storage unit. The other x0 is connected to the external IBM Storwize V7000 Storage unit. The Flex System in the DR site has one physical server (x0 compute node). That x0 is connected to the external IBM Storwize V700 Storage unit. Each x0 compute node is running Windows Server 0 with the Hyper-V role enabled. A single virtual machine (VM) has been created on each server. Windows Server 0 and a standalone SQL Server 0 has been installed in each VM. The virtual machines have been configured in a Windows Server Failover Cluster with no shared storage. This is a scalable solution and more SQL Servers and VMs can be created depending on resources available. More x0 compute nodes can be added to the configuration also. The disk space for the files used in the SQL Server databases begin as physical arrays, and then physical volumes, created on each of the dedicated storage units. These volumes are then presented to the Windows Hyper-V operating system and formatted as VHDX files. The VHDX files are attached to the VMs, which see them as volumes presented to the Windows operating system inside the VM. It is on those volumes that the SQL Server databases are created. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

15 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA A single database, or multiple databases, can be included in a SQL Server AlwaysOn Availability Group. In this paper, we use a single database called N_, as an example. The Availability Group is initialized so that all three servers have an identical copy of the N_ database. One SQL Server is identified as having the primary replica. Users are automatically routed to the primary replica for updates. Updates made to the primary replica are sent to the other two SQL Servers, keeping them synchronized. In the event of a failure on the primary replica, the AlwaysOn Availability Group feature, along with Windows Server Failover Clustering feature, provide high availability by changing the role of one of the servers to become the new primary replica and automatically routing users to the server with the new primary replica. Component model The component model describes the functional view of the solution, showing the relationship of the components. In Figure below, three servers are shown, each with their own dedicated storage. Two servers are colocated in the Main Data Center (MDC) and one server is located in a remote Disaster Recovery (DR) site, connected by Ethernet networking. Main Data Center(MDC) Disaster Recovery (DR) Server Server Server Synchronous Asynchronous Figure : AlwaysOn Availability Group spanning the Main Data Center and DR Site Operational model This section describes the "operational" aspect of the solution architecture in a technology and product dependent manner. It describes the required operational characteristics and capabilities of the solution SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

16 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA architecture and represents, at an architectural level, the network of computer systems and their associated peripherals, together with the systems software, middleware, and application software that they run in order to support the users of the system. It is useful to view this solution in three layers, each built upon the previous layer: Physical, Virtual and SQL Server. Physical Layer this includes the physical hardware; Flex chassis, servers, switches, storage units, the networking configuration between the hardware components, as well as the operating systems installed on the hardware and the storage volumes created. Virtual Layer this includes the virtual machines (VMs), the virtual disks, the networks created between them, and the Windows Server Failover Cluster created between the VMs. SQL Server Layer this includes the SQL Server instances and the AlwaysOn Availability Group configuration, the replication topology created between them, and the user databases. Physical Layer We begin with a discussion of the placement of the hardware in the racks. There will be one rack in each data center the Main Data Center (MDC) and the Disaster Recovery (DR) site. Please refer to Figure Physical layer rack view. One Flex chassis and one external storage unit is placed in each rack. In the MDC, an integrated Flex V7000 storage unit, two x0 compute nodes, two 0 Gb Ethernet switches and two 8 Gb Fibre Channel switches are placed in the Flex chassis. In the DR site, one x0 compute node, two 0 Gb Ethernet switches and two 8 Gb Fibre Channel switches are placed in the Flex chassis. The physical location of the components in the chassis determines which internal connections are made with the switches. For example, the adapter in IO Module in the x0 connects to switch bays and (which are in positions and viewed from the back, from left to right). In this solution, the Ethernet adapter is installed in IO Module and the Ethernet switches in bays and, so that the Ethernet components are connected internally by the Flex infrastructure. Likewise, the Fibre Channel adapter is installed in IO Module in the x0 and the Fibre Channel switches are installed in switch bays and (which are in positions and viewed from the back, from left to right), so that the Fibre Channel components are connected internally in the same way as the Ethernet components. In addition, the placement of the x0s and integrated Flex V7000 (V7000A) in the Flex chassis determines which internal switch ports are used. For example, the x0 placed in node, called BT0, will utilize internal switch port on each of the switches. We will make use of Windows Ethernet adapter teaming on the x0, which will require an ISL (interswitch link) configuration between the two Ethernet switches within each Flex chassis. A Virtual Aggregation Group (VLAG), to support the team, should be configured on the switches [refer to page 9 Grant Privilege and Enter Configuration mode of Hyper-V Fast Track Reference Architecture for IBM Flex System]. The ISL requires two physical Ethernet cables between the switches. In addition, two physical Ethernet cables are required to connect the Flex chassis to the WAN (wide area network). The external storage unit (V7000B in the MDC and the V700C in the DR site) requires four external fibre cable connections to the fibre channel switches. The integrated Flex V7000 (V7000A) does not require external fibre cables as those connections are made using the internal infrastructure of the Flex chassis. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 6

17 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA The storage on the external Storwize V7000 (V7000B) will be zoned and dedicated to the x0 named BT0. The storage on the integrated Flex V7000 (V7000A) will be zoned and dedicated to the x0 named BT0. The storage on the external Storwize V700 (V7000C) will be zoned and dedicated to the x0 named BT0. Dedicating each storage unit to one x0 provides high availability and performance. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 7

18 System Storage 9 7 System Storage x0 x0 x0 Control Module - Flex System V7000 Control Module Flex System Enterprise Flex System Enterprise Power Supply Power Supply Power Supply Power Supply Power Supply Power Supply 0 Power Supply 0 Power Supply I/O I/O Gb Gb Gb Gb Gb Gb 9 Mgmt 0 Gb Gb 9 Mgmt Disconnect all supply power for complete isolation CAUTION Disconnect all supply power for complete isolation CAUTION CAUTION CAUTION 6 6 I/O 8Gb FULL FABRIC I/O 8Gb FULL FABRIC Mgmt Mgmt Power Supply s s 6 Power Supply s s Power Supply Power Supply Power Supply Power Supply I/O I/O Gb Gb Gb Gb Gb Gb 9 Mgmt 0 Gb Gb 9 Mgmt CAUTION CAUTION CAUTION CAUTION Disconnect all supply power for complete isolation Disconnect all supply power for complete isolation I/O 8Gb FULL FABRIC I/O 8Gb FULL FABRIC Mgmt Mgmt CMM CMM / LINK CMM CMM CMM / LINK CMM IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Physical deployment rack view Main Data Center (MDC) Ethernet Fibre V7000 storage controller V7000B Disconnect all supply power for complete isolation Disconnect all supply power for complete isolation Flex V7000 storage node V7000A Flex EN09 0Gb Ethernet Switch Flex x0 compute node BT0 Flex FC7 8Gb SAN Switch Flex x0 compute node BT0 Flex EN09 0Gb Ethernet Switch Flex FC7 8Gb SAN Switch Front view Back view Disaster Recovery (DR) WAN V700 storage controller V700C Disconnect all supply power for complete isolation Disconnect all supply power for complete isolation Flex EN09 0Gb Ethernet Switch Flex FC7 8Gb SAN Switch Flex x0 compute node BT0 Flex EN09 0Gb Ethernet Switch Flex FC7 8Gb SAN Switch Front view Back view Figure : Physical layer rack view SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 8

19 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Next, we discuss the configuration of the physical hardware, such as, installing the operating system and creating the SAN volumes. Please refer to Figure. We install Windows Server 0 on each x0 compute node (on a RAID mirror between the two local spinning disks) and enable the Hyper-V role. We name these servers BT0, BT0 and BT0. The first two are in the MDC and the last one is in the DR site. We are not going to configure Windows Server Failover Clustering (WSFC) on the Hyper-V servers. (However, we will be using WSFC on the virtual machines, as described below.) We use the new NIC Teaming feature in Windows Server 0, which allows two or more network adapters to behave as a single, virtual device. This improves the reliability of the networking subsystem if one NIC dies, the other continues to function and allows the bandwidth available to each to be pooled for greater total network throughput for SQL Server data. The physical volume configuration is highly dependent on an organization s requirements for space and performance. The client may wish to increase the number of volumes, for example, if there are more databases involved. We describe the sample configuration created for this paper. Each storage unit is configured identically. Three physical volumes were created on each storage unit. (Note that Figure shows a single SAN array physical volume per storage unit for simplicity.) The three volumes are mounted under the local C: drive (in C_mnt, K_mnt and L_mnt directories) of the Hyper-V server. They will be provisioned for a virtual machine operating system disk (C drive), a database data disk (K drive) and a database log disk (L drive), respectively. In this solution, the operating system disk (C_mnt) is on a volume of 6 spindles in a RAID0. The database (K_mnt) and log (L_mnt) disks are on dedicated volumes of 8 spindles each, in a RAID0. It is a SQL Server best practice to place the database and log files on separate dedicated volumes. This improves both availability and performance. Two spindles are assigned as hot spares. Each set of three physical volumes is mapped to and mounted on the corresponding Hyper-V server (meaning the one zoned on the fibre channel switch). SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 9

20 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Physical deployment logical view V7000 SAN Array WAN Users FLEX (MDC) FLEX V7000 SAN Array V700 SAN Array Hyper-V FLEX (DR) Hyper-V Hyper-V Figure : Physical layer logical view Virtual Layer Once the physical layer has been setup, we move on to creating and configuring the virtual layer. We will create virtual switches, virtual disks, virtual machines, and a cluster between the virtual machines. This will lay the foundation for SQL Server AlwaysOn Availability Groups, the final layer. Please refer to Figure. On each Hyper-V server, we create a Hyper-V virtual switch (vswitch) based on the physical teamed network adapter created in the physical layer. vswitch will provide one interface back to the Hyper-V host, and the switch can support N interfaces on the Virtual Machines (VMs). In this case, two virtual ports were configured in the VM. The management port will be used to manage the Hyper-V server (for example, using RDP). The two virtual machine ports will be used in the cluster between the virtual machines to create the public and private (heartbeat) networks. On each Hyper-V server, we create virtual disks (VHDX format) using the volumes mounted in the C_mnt, K_mnt and L_mnt directories created in the physical layer. One virtual machine was created on each Hyper-V server and assign it the desired number of processors and memory, adding the virtual disks as SCSI drives and the two VM network ports created on the vswitch. We install Windows Server 0 in each VM and enable the Failover Clustering Role. We SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 0

21 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA mount the C_mnt volume to the C: drive and the K_mnt and L_mnt volumes to the K: and L: drives, respectively, inside the VM. We assign the static IP addresses to the VM network interfaces, one for the Public network and one for the Private network. Note this solution supports multiple subnets. We name the VMs VM0 (on BT0), VM0 (on BT0) and VM0 (on BT0). We join the VMs to the domain. We create the Windows Server Failover Cluster between the three VMs. Please not that this cluster has no shared storage. It should have multiple static IP addresses, if the VMs are on different subnets. For details on establishing the cluster quorum model, including using a file share, and voting, please see: Building a High Availability and Disaster Recovery Solution using AlwaysOn Availability Groups Virtual deployment V7000 SAN Array VMDX WAN Users FLEX (MDC) FLEX V7000 SAN Array VMDX V700 SAN Array VMDX Hyper-V FLEX (DR) Hyper-V VM VM Hyper-V VM Windows Failover Cluster Figure : Virtual layer SQL Server Layer After the virtual layer has been configured, we create the final layer based on SQL Server and the AlwaysOn Availability Groups feature. In this step, we will install SQL Server, configure the AlwaysOn Availability Groups feature and assign a sample database, N_, to an availability group for high availability protection. Please refer to Figure 6. We install a default standalone SQL Server 0 Enterprise Edition Instance in each VM. Please note that this is not a clustered SQL Server Instance. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

22 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA We place our user database, N_, on the K: and L: drives in VM0, which will be our primary replica. Then we backup the database and restore it (using the NORECOVERY option) on the other two VMs, VM0 and VM0, which will be our secondary replicas. We create the Availability Group for the N_ database, making VM0 the primary replica, making VM0 the synchronous secondary with automatic failover and making VM0 (in the DR site) the asynchronous secondary with manual failover. We create an Availability Group Listener, called N_AG_Listener, giving two static IP addresses (one for each subnet), which provides a DNS name which user applications can connect to and be automatically routed to the primary replica, regardless of which subnet the primary replica is running on. SQL Server AlwaysOn Availability Group deployment V7000 SAN Array VMDX WAN Users FLEX (MDC) FLEX V7000 SAN Array VMDX V700 SAN Array VMDX Hyper-V FLEX (DR) Hyper-V VM VM SQL SQL Hyper-V VM SQL Server AlwaysOn Availability Group SQL Windows Failover Cluster Figure 6: SQL Server AlwaysOn Availability Group layer Deployment considerations This section describes noteworthy deployment considerations. In particular, it describes how features of the Flex System are used in the solution deployment across data centers for high availability. This section also includes a high-level overview of the requirements the customer s IT environment must address for deploying this reference architecture. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

23 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Systems management Systems management for this solution uses the native Windows Server and SQL Server management tools. Server / Compute Nodes The compute nodes are the Flex System x0 compute nodes. Each x0 has processors, 96 GB of memory, two. spinning disks in a RAID pair, a dual port 0Gb Ethernet adapter in IO Module and a dual port 8 Gb fibre channel adapter in IO Module. Processor speed and number of cores, and amount of memory should be selected based on the customer s requirements. Networking This section describes the Ethernet network topology. Figure 7 Ethernet network connections shows the connections between the servers, switches, and the wide area network (WAN). Each x0 compute node has two Ethernet connections from the EN -port 0Gb Ethernet adapter in IO Module, one to each Ethernet switch for redundancy and increased performance. These connections are made via the Flex infrastructure, not by using external cables. Two external cables connect the two Ethernet switches within a single Flex chassis to create a bonded (LACP) Inter Switch (ISL). The ISL link allows the two Ethernet ports in each x0 to be teamed efficiently. The teaming is done using LACP for active/active utilizations of the links. Additional Ethernet cables uplink the switches into the WAN infrastructure. The WAN, which is unique to each customer, provides connectivity between the Flex chassis over a geographic distance. The WAN also connects to the file share (needed for the quorum in Windows Server Failover Cluster), to the customer s Active Directory servers (required for the Windows Server Failover Cluster) and to the users for accessing the application servers using the SQL Server databases. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

24 Gb Gb 0 Gb Gb 9 Mgmt Gb Gb 0 Gb Gb 9 Mgmt Gb Gb 0 Gb Gb 9 Mgmt Gb Gb 0 Gb Gb 9 Mgmt IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Main Data Center(MDC) Flex Chassis Flex EN09 0Gb Ethernet Switches ISL cables BT0 I/O Mod I/O Mod Internal ports Internal ports Internal connection BT0 I/O Mod I/O Mod External cables AD Server WAN Users File Share Disaster Recovery (DR) Flex Chassis Flex EN09 0Gb Ethernet Switches ISL cables Internal ports Internal ports BT0 I/O Mod I/O Mod Figure 7: Solution Ethernet network connections SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

25 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Storage integration This section describes the storage fibre channel network topology. Figure 8 shows the connections between the servers, switches, and the storage units. Each x0 compute node has two fibre channel connections from the F7 -port 8Gb FC adapter in IO Module, one to each fibre channel switch for redundancy. These connections are made via the Flex infrastructure, not by using external fibre cables. The integrated Flex V7000 (V7000A) is also connected to the fibre channel switches via the Flex infrastructure. The external storage units (V7000B and V700C) are connected to their respective Flex chassis via external fibre cables to the fibre channel switches as shown. The effect of this topology is for each server to be connected to its own dedicated storage, with redundancy, including redundant switches. SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0

26 8Gb FULL FABRIC Mgmt 8Gb FULL FABRIC Mgmt 8Gb FULL FABRIC Mgmt 8Gb FULL FABRIC Mgmt IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA Main Data Center(MDC) V7000B Node Node Flex FC7 8Gb Switches Flex Chassis Node Node V7000A BT0 I/O Mod I/O Mod Internal ports Internal connection BT0 I/O Mod I/O Mod Internal ports External cables Disaster Recovery (DR) V700C Node Node Flex FC7 8Gb Switches Flex Chassis Internal ports BT0 I/O Mod I/O Mod Internal ports Figure 8: Solution storage fibre channel network connections Performance Considerations AlwaysOn Availability Groups are sensitive to Ethernet network bandwidth, because potentially high volume updates on the primary replica can be sent over Ethernet to the secondary replicas. This solution provides very high network bandwidth using the 0 Gb/s aggregated teamed network adapters. Updating SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 6

27 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA workload performance on the primary replica can be improved by offloading read-only workloads (such as backup and reporting) to the secondary replicas. Create fixed sized VHDX disks for best storage performance. Chose 6K as the NTFS cluster (Allocation Unit) size for the volumes used by SQL Server. Another improvement in performance could be realized by doing away with the virtual machines and installing SQL Server on the physical servers. Both of these options trade off performance with the manageability benefits of virtualization. Best practices and limitations In this solution, the unit of failover is the availability group (a group of user databases). SQL Server Agent jobs, logins, linked servers, and other objects that are stored outside of the availability databases do not fail over with the availability group. Consider the use of contained databases for containing logins that fail over across the availability replicas. For other objects outside of the user database such as SQL Server Agent jobs, linked servers, and SQL Server Integration Services packages, you will need to take additional synchronization steps across the SQL Server instances. When conducting a planned failover to the DR site, or when failing back to the MDC, put the secondary replica on VM0 in synchronous mode, temporarily. Then the failover can proceed. Once the availability group is operational, set up monitoring to provide alerts when the health of the availability group declines. Set up database and log backup preferences, so that backups are taken regardless of the state of the availability group. Other considerations When deploying the solution, here are a couple of alternatives. Consider installing SQL Server from the command line, rather than via the GUI. After the first VM, VM0, has been created, sysprep it and copy the VM image to the other two servers. This works even if SQL Server is installed (a new feature of SQL Server 0). SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 7

28 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA About the author Michael Lawson works at the IBM Center for Microsoft Technologies in Kirkland, Washington (just miles from the Microsoft main campus). He also has an office on the Microsoft main campus in Redmond, Washington to facilitate close collaboration with Microsoft. Michael has been an IBM employee since 999 and has specialized in SQL Server including data warehousing, high availability, performance testing, and virtualization. mikelaw@us.ibm.com Acknowledgements The author would like to thank the following people for supporting the work in this paper: Vinay Kulkarni, David Ye, Hoai Nguyen, and David West. Appendix : Bill of Material MAIN DATA CENTER (MDC) PN Description Quantity Flex System chassis_mdc 87HC IBM Flex System Enterprise Chassis Base Model A0TB IBM Flex System Fabric EN09 0Gb Scalable Switch 9 0GbE 80 nm Fiber SFP+ Transceiver (SR) for IBM BladeCenter 700 m LC-LC Fiber Cable (networking) 9 0GbE 80 nm Fiber SFP+ Transceiver (SR) for IBM BladeCenter 70 m LC-LC Fiber Cable (networking) AEL IBM Flex System Fabric EN09 0Gb Scalable Switch (Upgrade ) A0UE IBM Flex System Chassis Management Module 79 m Yellow Cate Cable A0UC IBM Flex System Enterprise Chassis 00W Power Module Standard A0UC IBM Flex System Enterprise Chassis 00W Power Module Standard 6.m, 6A/00-0V, C9 to IEC 0-C0 Rack Power Cable A0TW System Documentation and Software - US English A0TA IBM Flex System Enterprise Chassis A0UA IBM Flex System Enterprise Chassis 80mm Module ANF IBM Flex System Console Breakout Cable AEV RFID Tag, AG/AP: 90-98Mhz A0UD IBM Flex System Enterprise Chassis 00W Power Module A0UD IBM Flex System Enterprise Chassis 00W Power Module 69 m, 6A/00-0V, C9 to IEC 0-C0 Rack Power Cable A0TD IBM Flex System FC7 8Gb SAN Switch 07 IBM 8Gb SFP + SW Optical Transceiver SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 8

29 IBM Flex System Reference Architecture for Microsoft SQL Server 0 HA 70 m LC-LC Fiber Cable 00 BladeCenter Chassis Configuration 06 Rack Installation >U Component AZT IBM Fabric Manager Manufacturing Instruction Node_Flex_V X9 IBM Flex System V7000 Control Enclosure 970 Storage Subsystem ID 0 AD 600 GB 0,000 RPM 6Gbps.-inch SAS HDD A 8Gb FC Port Daughter Card ADD Agency label - IBM Logo A Bezel - SRC with IBM Logo Node_x0_BT0 877AC Flex System node x0 Base Model Custom RAID Configuration 99 IBM 00GB 0K 6Gbps SAS." SFF Slim-HS HDD ABL IBM Flex System Compute Node." SAS.0 Backplane AC System Documentation and Software-US English ABD IBM Flex System x0 Compute Node ABM IBM Flex System FC7 -port 8Gb FC Adapter AQY IBM Flex System EN -port 0Gb Ethernet Adapter 89 8GB (x8gb, Rx,.V) PCL-0600 CL9 ECC DDR MHz LP RDIMM AER Intel Xeon Processor E-690 8C.9GHz 0MB Cache 600MHz W AES Addl Intel Xeon Processor E-690 8C.9GHz 0MB Cache 600MHz W ABF IBM Flex System x0 Compute Node Front Bezel A8 IBM Flex System x0 Compute Node Air Baffle Node_x0_BT0 877AC Flex System node x0 Base Model Custom RAID Configuration 99 IBM 00GB 0K 6Gbps SAS." SFF Slim-HS HDD ABL IBM Flex System Compute Node." SAS.0 Backplane AC System Documentation and Software-US English ABD IBM Flex System x0 Compute Node ABM IBM Flex System FC7 -port 8Gb FC Adapter AQY IBM Flex System EN -port 0Gb Ethernet Adapter 89 8GB (x8gb, Rx,.V) PCL-0600 CL9 ECC DDR MHz LP RDIMM AER Intel Xeon Processor E-690 8C.9GHz 0MB Cache 600MHz W AES Addl Intel Xeon Processor E-690 8C.9GHz 0MB Cache 600MHz W ABF IBM Flex System x0 Compute Node Front Bezel A8 IBM Flex System x0 Compute Node Air Baffle SQL Server AlwaysOn Availability Groups Reference architecture Copyright IBM Corporation 0 9