Solution Guide: Flex System Solution for Oracle RAC

Bryan Reese Ajay Dholakia Thomas Parker Solution Guide: Flex System Solution for Oracle RAC Solution Reference Number: BDARACFX043 Contents Solution Guide: Flex System Solution for Oracle RAC.............................. 1 Introduction.............................................................. 2 Business value............................................................ 3 Solution overview.......................................................... 3 Solution architecture....................................................... 5 Hardware components...................................................... 7 Usage scenarios.......................................................... 9 Integration.............................................................. 11 Supported platforms....................................................... 11 Partial Bill of Materials..................................................... 12 Related information....................................................... 12 Copyright Lenovo 2015. All rights reserved. ibm.com/redbooks 1

Introduction The Flex System Solution for Oracle RAC provides a robust database foundation for any cloud infrastructure or enterprise environment. The combination of reliable System x servers, the highly scalable Flex System, and the high availability and scalability of Oracle Real Application Clusters (RAC) provides a comprehensive solution for any database application. The solution (see Figure 1) deploys Oracle RAC database instances on Flex System X6 family compute nodes with Intel Xeon E7 v2 processors. The IBM FlashSystem 840 is used for shared storage, and the solution uses the Flex System EN4023 10Gb Scalable Switch and Flex System FC5022 16Gb SAN Scalable Switch for connectivity to storage and network communication. Oracle Real Application Cluster Users Application Servers Node 1 Node 2 Shared Storage Figure 1 Overview of Flex System Solution for Oracle RAC This paper provides the planning requirements, design considerations, and leading practices for implementing a Flex System-based Oracle RAC solution. It is intended for IT professionals, technical architects, sales engineers, and consultants who participate in implementing an Oracle RAC solution on Flex System. The information that is provided here assumes that the reader is familiar with the Linux operating system, Oracle databases, Oracle RAC technology, and the various hardware products that are involved. Detailed documentation and training about RAC exists outside of this solution guide, which covers only what was tested for this configuration. 2 Solution Guide: Flex System Solution for Oracle RAC

Business value The Flex System Solution for Oracle RAC answers a common need among organizations by providing a comprehensive, high-availability database solution for any application. The resiliency of an IT organization's access to its critical information is one of the most important considerations when planning to buy or upgrade equipment. All aspects of the database, from the hardware it runs on to the software platform that provides and manages the information, should be reliable, dependable, and include features that ensure auditability and accountability. Beyond the need for consistent access to data, there are other important considerations, including how quickly the data can be retrieved and the ease with which the database can be maintained. To meet these needs, Lenovo is combining Oracle RAC technology and Flex System elements, including the x480 X6 compute node. Oracle RAC provides a resilient software solution for database access on multiple servers. By using this high-performance, high-availability software with the database, queries can be sent seamlessly to any number of back-end servers without any involvement from the client. Also, if there is a server failure, all database instances in the Oracle RAC system automatically recover by failover to another server, so the system can send the needed response to the client from a server that is functioning as expected. The Flex System x480 compute node has built-in hardware redundancy. This feature allows multiple system components to function together or as backups if there is a power loss or with most configuration or hardware failures. This redundancy helps companies achieve extreme performance levels that allow all connected users and applications to perform optimally. Flex System is designed specifically to provide easy server management and consolidation, which allows for more systems in a single rack, and multiple hardware options provide critical flexibility. These features allow clients to select appropriate options to optimize their system for the applications that are being deployed. For example, you can deploy 16 Gb Fibre Channel and 10 Gb Ethernet to operate together to form a single, cohesive, and optimized solution. The Flex System Solution for Oracle RAC uses the Intel Xeon E7-4800 v2 family of processors in the Flex System x480 compute nodes, which enable application performance that is up to 100% faster than a previous generation, four-socket system. This speed reduces the capital costs of initial deployment by requiring less space and fewer nodes to complete the same tasks. The Flex System Solution for Oracle RAC reduces administrative costs and provides software-optimized performance from Oracle to provide a comprehensive, high-performance database solution. Solution overview The Flex System Solution for Oracle RAC is a scalable, resilient, and high-performance database solution. Load balancing, scaling, and optimized performance are supported by using Oracle RAC features, including Clusterware, Automatic Storage Management (ASM), and Single Client Access Name (SCAN). Solution Guide: Flex System Solution for Oracle RAC 3

The solution uses Oracle technologies that allow for redundancy and ease of management. The Oracle Grid Infrastructure allows for traditional management of the Oracle RAC system to determine which nodes run certain database tasks, and enables policy-based control and automated migration of database resources for optimal qualities of service under heavy loads. Starting with Oracle release 12c, a new multitenant feature also is supported. This feature creates a container database with multiple databases within it that are separated from other containers on the same Oracle RAC infrastructure while still providing for a seamless client experience. Other key software differentiators in Oracle s 12c release include the ability to have heterogeneous servers with different processor or memory technology, which enables fine-grained control in scaling the servers that host the database. Another differentiator is Oracle Database Application Continuity, which allows for replays of database information after recoverable errors. If there is an error, this feature enables automatic corrections to a server or client where Oracle software is running on both sides of the transaction. Other key components of the solution's hardware architecture include Brocade Fibre Channel and Ethernet fabrics, which enable centralized management and simple deployment of more chassis. This simplicity allows seamless deployment of extra nodes with minimal configuration needed for each chassis. When combined with the management that is provided through Oracle s Grid Infrastructure, centralized chassis management allows an administrator to automate and optimize workflows, which results in lower administrative overhead and reduced total cost of ownership (TCO) for the solution as a whole. To provide this scalable, high-performance solution, the tested implementation of the solution uses 10 Gb Ethernet, 16 Gb Fibre Channel, and a variable number of X6 servers that are housed within a Flex System chassis. For storage, it uses the IBM FlashSystem 840, an industry-leading storage enclosure with high bandwidth and high input/output operations per second (IOPS). The client can be any server or desktop that must access the data. The core feature of the Flex System Solution for Oracle RAC is the flexibility of the system elements that are involved. With storage that can total 2-48 TBs in a single unit, and memory that can scale 3-12 TBs per node, it becomes relatively easy to meet the needs of each specific deployment, whether the consideration is cost, performance, or available storage. This flexibility makes initial deployments cheaper and future expansion simpler, with no downtime in the data center. The FlashSystem 840 can be expanded with an extra flash storage module that can be sized at 1 TB, 2 TB, or 4 TB, or by adding a FlashSystem 840 that can be used to expand an existing data store or create one. 4 Solution Guide: Flex System Solution for Oracle RAC

Solution architecture Figure 2 shows the high-level architecture of the Flex System Solution for Oracle RAC. Database Server 1 Database request is sent to correct server Database Server 2 Fibre Channel high-throughput interconnect used for data store Ethernet network Fibre Channel Switch Database request Client Shared Storage Figure 2 High-level architecture of the Flex System Solution for Oracle RAC Flexibility and scalability in design are the primary advantages of the Flex System X6 Compute Node product family. Each compute node is composed of two processors. Depending on the processor type that is installed, the X6 compute nodes can scale to four processors or even eight processors. In our testing, we used the x480 X6 compute node; with its E7-4800 v2 series processors, two x480 X6 can be connected to form a single four-socket server. For maximum scalability, the four x880 X6 compute nodes can be connected to form a single eight-socket server, as shown in Figure 3 on page 6. Solution Guide: Flex System Solution for Oracle RAC 5

Figure 3 Four x880 Compute Nodes in an eight-socket complex The deployment units for the Flex System Solution for Oracle RAC can be broken down into small compute units (at least two Flex System x6 compute nodes) or chassis (only one is required). As business needs change, the deployment can be expanded with minimal overhead thanks to the rapid deployment abilities of the X6 compute family and the Brocade fabric-based networks. With this solution, it is easy to plug in a new chassis or a new compute node after which configuration can be done by using Oracle response files and operating system deployment can be done with a tool, such as xcat or Tivoli Provisioning Manager for Operating System Deployment. If more compute power must be added to a compute node that is not yet scaled to its limit, no operating system setup is required. The network scales with minimal user intervention. When a node is added, changes to the network can include zoning and the configuration of a VLAG for the new nodes. Adding a chassis requires changing the EN4023 rbridge ID. After the solution is scaled beyond one chassis, we suggest expanding the Fibre Channel network with Flex System FC5022 16Gb SAN Scalable Switches that are designated as the core switches with other switches connecting to them in Brocade's Access Gateway mode. This configuration keeps the points of administration to a minimum. Full-fabric mode is supported throughout the whole deployment, if wanted. The optimal configuration sometimes depends on whether there is any shared infrastructure and the administrator's preference, as there are no performance or scaling considerations with this decision. For the Ethernet network, a core can be formed for the EN4023 Ethernet fabric switch, but at smaller scales, switches can be connected in a full mesh to ensure optimal bandwidth usage, with connections leaving the switches as a single VLAG. Multiple chassis can share FlashSystem 840 storage systems. Each FlashSystem 840 can contain 2-48 TBs of storage with fully hot swappable flash storage modules, which enables the addition and configuration of more storage without bringing the system down. This flexibility means that more logical unit numbers (LUNs) can be made available to the operating system, the SCSI bus can be rescanned, and the Oracle ASM volume manager can be used to create and configure more disk groups without any interruption of service. 6 Solution Guide: Flex System Solution for Oracle RAC

Because of redundancies and hot swap capabilities, more capacity can be added to the database without requiring a full system maintenance window. This capability also applies to storage, network, and compute capacity. Ideally, the largest possible single units are purchased in all cases to enable this scaling. These units are the 4 TB flash storage modules for the FlashSystem 840 and the x880 or x480 compute nodes. Hardware components In this section, we briefly describe the key features of the compute, network, and storage hardware components that are used in the solution. Flex System Enterprise Chassis The Flex Enterprise Chassis was developed to allow node-to-node traffic, which bypasses the need for a Top of Rack Switch (TOR) and reduces up-front capital costs. Because Flex System compute nodes support more memory per blade, and in turn, more VMs, you can save on software licensing costs. You also get increased agility because Flex System was designed for technology in development today and going forward. You can upgrade in place, that is, without ripping out your entire infrastructure. You can go from 1 Gb to 10 Gb to 40 Gb of internal bandwidth within the same chassis, or mix solid-state drive (SSD) and hard disk drive (HDD) technology, as needed. You also can scale from two sockets to four sockets to eight sockets with the Flex x880 X6, which is the only eight-socket blade solution that is available today. Flex System X6 Compute Nodes The Flex System X6 Compute Node family, which is composed of the x880 X6, x480 X6, and x280 X6, are high-performance, scalable compute nodes that are designed to take on the most demanding workloads. The performance, flexibility, and resiliency features enable the X6 to run mission-critical workloads, such as key business applications, database, analytics, and large virtual machine deployments. Each compute node contains two Intel Xeon E7 processors. The x880 X6 uses E7-8800 v2 processors and supports joining two or four x880 compute nodes to form a single eight-socket server with 192 DIMM slots. The x480 (see Figure 4 on page 8) that was used in testing the solution uses E7-4800 v2 processors and supports joining two x480 compute nodes to form a single four-socket server with 96 DIMM slots. The x280 has two E7-2800 v2 processors and 48 DIMM slots but does not offer scaling. Solution Guide: Flex System Solution for Oracle RAC 7

Figure 4 Flex System X6 Compute Node Flex System EN4023 10 Gb Scalable Switch The EN4023 scalable switch (see Figure 5) offers full Layer 3 support, including OSPF, BGP, and VRF-lite in an embedded switch. It also supports all industry-standard layer 2 features, which allows for easy deployment in any environment. The switch s primary cost-saving attributes are the VCS Fabric features that enable scaling and ease of management for any configuration, large or small. Figure 5 Flex System EN4023 Brocade scalable Ethernet switch IBM Flex System FC5022 16Gb SAN Scalable Switch The Flex System FC5022 16 Gb SAN Scalable Switch is an embedded Fibre Channel switch with 20 external ports (see Figure 6) that enable designs without a TOR switch. The large number of ports also creates a unique opportunity for an embedded environment with self-contained configurations that were created only with TOR switches and fixed configurations. The FC5022 is Brocade Fabric Vision-capable and fully tested and supported in upstream configurations. Figure 6 Flex System FC5022 16Gb SAN Scalable switch 8 Solution Guide: Flex System Solution for Oracle RAC

IBM FlashSystem 840 The IBM FlashSystem 840 (see Figure 7) delivers high performance, efficiency, and reliability in a compact form factor. It scales 2-48 TBs of addressable storage and is usable in any deployment, large or small. With a simple user interface that is similar to IBM s Storwize family of products, the FlashSystem 840 speeds time-to-value and future administrative costs are minimized. Figure 7 IBM FlashSystem 840 Usage scenarios In addition to infrastructure redundancy, Oracle RAC technology provides redundant software mechanisms. If there is a full node failure or misconfiguration, a virtual IP address moves from one node to another, which allows the database to continue operating seamlessly from the perspective of the user. This transition occurs when a heartbeat between the servers on their dedicated networks ceases to be received as expected after which the failover logic is implemented. For optimal, normal operation of hardware resources, all of the servers in the cluster receive requests for their services that they manage. In a typical usage scenario, an application server sends its request over the Ethernet network (the Flex System x480 servers that run the database are connected to an EN4023 high-speed 10 Gigabit Ethernet switch that is Brocade VCS Fabric-capable). When the request arrives, it is processed in local memory, if possible. If the data is in storage instead of the local memory that contains the buffer cache, the FC5022 Fibre Channel switch carries the traffic to the FlashSystem 840 storage array. Figure 8 on page 10 shows how these services are distributed. Solution Guide: Flex System Solution for Oracle RAC 9

HR Database Access Requestor Sales Database Access Requestor Internal Operations Requestor s Oracle Net Services Client Access HR Database Instance Sales Database Instance Internal Operations Database Instance Figure 8 Normal operation of the database Figure 9 shows what occurs if there is a failure. HR Database Access Requestor Sales Database Access Requestor Internal Operations Requestor s Oracle Net Services Client Access HR Database Instance Sales Database Instance Internal Operations Database Instance * Physical Server 1 *Now included in Physical Server 1 without client knowledge Figure 9 Fail over operation of the database 10 Solution Guide: Flex System Solution for Oracle RAC

Integration This section describes some special deployment considerations for the Flex System Solution for Oracle RAC and includes recommendations for integrating systems management into an existing IT environment. Networking This solution provides for a redundant and robust network for operation of the Oracle RAC database. There are individual, separate networks for storage and logically paired Ethernet switches, which allows for the bonding of interfaces through VLAGs and operating system-controlled interfaces. The separate networks are fully redundant, so misconfigurations or hardware-related failures can occur without a full loss of system functionality. When the solution is implemented, it is suggested that you isolate traffic by using VLANs and deploy single HBA zoning on all switches. It is further suggested that you connect the EN4023 Ethernet switches to each other to form a single VCS fabric unit and logical chassis, but leave the FC5022 Fibre Channel switches unconnected as separate fabrics. For more information about Fibre Channel zoning best practices, see the Brocade white paper, Secure SAN Zoning Best Practices, which is available at this website: http://www.brocade.com/downloads/documents/white_papers/zoning_best_practices_wp-0.pdf Systems management It is recommended (but not required) that the system management node is kept separate from the solution to manage the Integrated Management Module (IMM) and Chassis Management Module (CMM) and to provision the operating system for the nodes. If the solution is being installed without any other IT infrastructure, you can manage the nodes as single units for the operating system and manage the hardware through the CMM web interface that can control each individual node s IMM. Supported platforms Red Hat Enterprise Linux 6 update 5 was used in testing the solution. Other operating systems can be used and should function as expected, but they were not validated. Round Robin DNS should be configured and the network interface cards should all be bonded and set to a static IP address in the operating system. Solution Guide: Flex System Solution for Oracle RAC 11

Partial Bill of Materials Table 1 is a partial sample Bill of Materials that lists the primary components of the Flex System Solution for Oracle RAC. Table 1 Partial sample Bill of Materials, Flex System Solution for Oracle RAC Part number Description Quantity Flex System Chassis (Lenovo) 8721HC1 IBM Flex System Enterprise Chassis Base Model 1 ESWD IBM Flex System EN4023 10Gb Scalable Switch 2 A2RQ IBM Flex System FC5022 24-port 16Gb ESB SAN Scalable Switch 2 x480 Compute Node (Lenovo) 7903AC2 Flex System Compute Node x480 X6 Base Model 4 A5QU Flex System x480 X6 Compute Node 4 A45S IBM Flex System FC5054 4-port 16Gb FC Adapter 4 A4K3 IBM Flex System CN4022 2-port 10Gb Converged Adapter 4 A3QM A4EA A4DQ 16GB (1x16GB, 2Rx4, 1.35V) PC3L-12800 CL11 ECC DDR3 1600MHz LP RDIMM Intel Xeon Processor E7-4880 v2 15C 2.5GHz 37.5MB Cache 1600MHz 130W Extra Intel Xeon Processor E7-4880 v2 15C 2.5GHz 37.5MB Cache 1600MHz 130W 72 4 4 FlashSystem 840 (IBM) 9843-AE1 IBM FlashSystem 840 1 AF11 4TB emlc Flash Module 10 AF15 FC/FCoE Host Interface Card 2 AF19 16Gb FC 4 Port Host Optics 2 Related information For more information about the Flex System Solution for Oracle RAC and its components, see the following resources: Channel Bonding interfaces on Red Hat Enterprise Linux 6: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/d eployment_guide/s2-networkscripts-interfaces-chan.html Multipath configuration on Red Hat Enterprise Linux 6: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html-s ingle/dm_multipath/ Oracle ASM configuration guide: http://docs.oracle.com/database/121/ostmg/toc.htm 12 Solution Guide: Flex System Solution for Oracle RAC

Oracle Net Services configuration guide: http://docs.oracle.com/database/121/netag/toc.htm Full documentation for Oracle Database and Grid Computing: http://docs.oracle.com/database/121/nav/portal_11.htm Fibre Channel zoning best practices: http://www.brocade.com/downloads/documents/white_papers/zoning_best_practices_w P-00.pdf Flex System X6 Compute Nodes Product Guide http://www.redbooks.ibm.com/abstracts/tips1160.html?open Oracle RAC sizing guidance: http://www.oracleracsig.org/pls/apex/rac_sig.download_my_file?p_file=1001042 Oracle Database Performance Tuning Guide: http://docs.oracle.com/database/121/tgdba/toc.htm Oracle SCAN: http://www.oracle.com/technetwork/products/clustering/overview/scan-129069.pdf Solution Guide: Flex System Solution for Oracle RAC 13

14 Solution Guide: Flex System Solution for Oracle RAC

Notices Lenovo may not offer the products, services, or features discussed in this document in all countries. Consulty our local Lenovo representative for information on the products and services currently available in yourarea. Any reference to a Lenovo product, program, or service is not intended to state or imply that only that Lenovo product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any Lenovo intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any other product, program, or service. Lenovo may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: Lenovo (United States), Inc. 1009 Think Place - Building One Morrisville, NC 27560 U.S.A. Attention: Lenovo Director of Licensing LENOVO PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. Lenovo may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. The products described in this document are not intended for use in implantation or other life support applications where malfunction may result in injury or death to persons. The information contained in this document does not affect or change Lenovo product specifications or warranties. Nothing in this document shall operate as an express or implied license or indemnity under the intellectual property rights of Lenovo or third parties. All information contained in this document was obtained in specific environments and is presented as an illustration. The result obtained in other operating environments may vary. Lenovo may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Any references in this publication to non-lenovo Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this Lenovo product, and use of those Web sites is at your own risk. Any performance data contained herein was determined in a controlled environment. Therefore, the result obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Copyright Lenovo 2015. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by Global Services Administration (GSA) ADP Schedule Contract 15

This document REDP-5149-00 was created or updated on January 14, 2015. Send us your comments in one of the following ways: Use the online Contact us review Redbooks form found at: ibm.com/redbooks Send your comments in an email to: redbooks@us.ibm.com Trademarks Lenovo, the Lenovo logo, and For Those Who Do are trademarks or registered trademarks of Lenovo in the United States, other countries, or both. These and other Lenovo trademarked terms are marked on their first occurrence in this information with the appropriate symbol ( or ), indicating US registered or common law trademarks owned by Lenovo at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of Lenovo trademarks is available on the web at http://www.lenovo.com/legal/copytrade.html. The following terms are trademarks of Lenovo in the United States, other countries, or both: Lenovo Lenovo(logo) System x TruDDR4 The following terms are trademarks of other companies: Intel, Intel Xeon, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Microsoft, Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. 16 Solution Guide: Flex System Solution for Oracle RAC