Oracle Database on Cisco UCS C-Series Servers with Fusion-io iomemory

Transcription

1 White Paper Oracle Database on Cisco UCS C-Series Servers with Fusion-io iomemory August Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 1 of 32

2 Contents What You Will Learn... 3 Objectives... 3 Audience... 3 Purpose of This Document... 4 Solution Overview... 4 Cisco Unified Computing System... 6 Cisco Nexus 5548UP Switch... 8 Fusion-io... 9 Hardware and Software Components... 9 Infrastructure Setup Network Design Fusion-io iodrive2 Configuration Oracle Database 11g Configuration Performance and Failover Analysis Workload Description Test Scenarios Conclusion For More Information Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 2 of 32

3 What You Will Learn The Cisco Unified Computing System (Cisco UCS ) is a next-generation data center platform that unites computing, networking, storage access, and virtualization resources in a cohesive system designed to reduce total cost of ownership (TCO) and increase business agility. Cisco UCS C-Series Rack Servers extend unified computing innovations to a rack-mount form factor, including unified management through a single wire, a standards-based unified network fabric, and radical simplification through Cisco fabric extender technology. Based on industry-standard, intelligent Intel Xeon processors, Cisco UCS C-Series Rack Servers can be configured with PCI Express (PCIe) based Fusion-io iodrive2 cards, which provides ultra-low latency with storage capacity of up to 3 terabytes (TB) per Fusion-io iodrive2 card. PCIe-connected flash-memory technology is widely accepted in data centers because it allows existing applications to be used transparently while offering a significant performance improvement with a smaller board footprint. PCIe flash-memory-based storage is directly connected to the PCIe bus, providing a direct connection between the PCIe SAS or SATA solid-state drive (SSD) and the CPU, as well as system memory. This tightly coupled connection provides a storage environment that has lower latency than a SAS or SATA SSD. Fusion-io iodrive2, a PCIe flash-memory-based storage solution, provides low latency, high capacity, and high performance. Oracle Database provides the foundation that allows IT to successfully deliver more information with higher quality of service (QoS), reduce the risk of change, and make more efficient use of the IT budget. All across small and midsized businesses, high performance, database failover, data backup, and data restoration techniques are the major challenges in adoption of a single-instance Oracle Database deployment. This document discusses how customers can achieve high I/O throughput with very low disk latency when using Cisco UCS Rack Servers with Fusion-io cards for their enterprise-class Oracle Database deployments. Cisco UCS Rack Server direct-connect technology allows the use of Cisco UCS Service Profiles on Cisco UCS Managed Rack Servers, resulting in little downtime during migration after a server failure. Objectives This document provides a reference architecture that illustrates the benefits of using Cisco UCS and Fusion-io, with Oracle Database to provide a robust, resilient, and efficient infrastructure solution that can meet the demanding needs of businesses today. This document assumes that the user is familiar with Cisco UCS, Cisco Nexus switches, Fusion-io, and Oracle Database product technologies. Audience This document is intended for solution architects, sales engineers, field engineers, and design consultants involved in planning, designing, and deploying an Oracle Database Solution on Cisco UCS and Fusion-io infrastructure. It assumes that the reader has an architectural understanding of the base configuration and implementation of Oracle Database, Cisco UCS, and Fusion-io Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 3 of 32

4 Purpose of This Document The testing reported in this document had two purposes: To evaluate the maximum performance of a single instance of Oracle Database on Cisco UCS C240 M3 Rack Servers with Fusion-io To illustrate how a single-instance Oracle Database with Fusion-io can be used as an enterprise solution when combined with robust backup and recovery techniques leading to little downtime and high data reliability Enterprises can use the comparisons, failover and backup techniques reported here to help make value-based decisions in choosing Cisco UCS and Fusion-io technologies to implement reliable and cost-effective Oracle Database solutions. Solution Overview The solution discussed in this document demonstrates the deployment of a single instance of Oracle Database 11g on a Cisco UCS C240 M3 server equipped with two 3-TB Fusion-io iodrive2 cards. Cisco UCS C240 M3 servers are directly connected to the Fabric Interconnect using 10-Gbps unified network fabric. Cisco UCS Manager 2.2 supports an option to connect Cisco UCS C-Series Rack Servers directly to the Cisco UCS Fabric Interconnects. This option enables Cisco UCS Manager to manage the Cisco UCS C-Series servers using a single cable for both management traffic and data traffic. The Cisco UCS 6248UP 48-Port Fabric Interconnect (Figure 2) is a 1RU 10 Gigabit Ethernet, FCoE, and Fibre Channel switch offering up to 960-Gbps throughput and up to 48 unified ports. The switch has 32 1/10-Gbps fixed Ethernet, FCoE, and Fibre Channel ports on the base with the option of one expansion slot with 16 unified ports. Cisco Fabric Interconnects creates a unified network fabric and provides uniform access to both network and storage resources. Fabric Interconnects are deployed in pairs for high availability. The Fabric Interconnects are connected to two upstream Cisco Nexus 5548UP Switches. Oracle Database (single instance) on Cisco UCS Rack Servers with Fusion-io cards can be configured with multiple options. The configuration depends on the performance and high-availability requirements of the deployed solution. Some possible configurations are listed here: Oracle Database (single instance) on a Cisco UCS Rack Server with a single Fusion-io card: This option provides high performance but lacks in high-availability features. If the Fusion-io card fails, the customer has to install another Fusion-io card on the server and restore the database through backup and archive logs. Oracle Database (single instance) on a Cisco UCS Rack Server with two Fusion-io cards: This option provides protection from failure of a Fusion-io card. Oracle Automatic Storage Management (ASM) with normal redundancy can be configured to mirror data across two Fusion-io cards. Hence, failure of one Fusion-io card is tolerated. A maximum of four 3-TB Fusion-io cards can be installed in a Cisco UCS C240 M3 server. Oracle Database (single instance) with Oracle Data Guard on a Cisco UCS Rack Server and Fusionio card: This option provides high performance and high availability for Oracle Database during Fusion-io card failure. Oracle Data Guard is a high-availability and disaster-recovery solution that provides very fast automatic failover (referred to as fast-start failover) after database failures, node failures, data corruption, and media failures Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 4 of 32

5 In the solution discussed here, two Fusion-io iodrive2 cards are installed on a single Cisco UCS C240 M3 and configured as a normal-redundancy disk group. This configuration provides storage capacity equivalent to one Fusion-io card and protection during failure of one Fusion-io card. Additionally, the Cisco UCS C240 M3 was provisioned with external network-attached storage (NAS), which allows storage of Oracle database backup and archive logs. Oracle Recovery Manager (RMAN) was used to regularly back up Oracle Database. This setup allows a failed Oracle Database instance to be reprovisioned within a certain downtime. The solution described here presents the following aspects of deployment: Configuration of Oracle Database Enterprise Edition (single instance) on Cisco UCS C240 M3 server (single-wire Cisco UCS Management) with two Fusion-io iodrive2 cards and Red Hat Enterprise Linux 6.4 Configuration of Oracle ASM using ASMlib with normal redundancy for the Oracle ASM disk group Performance validation using an online transaction processing (OLTP) workload Performance impact during failure of one of the Fusion-io cards Restoration of the database through archive and backup logs Figure 1 shows the physical layout of the test environment. Figure 1. Physical Layout of Managed Cisco UCS C240 M3 Rack Server 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 5 of 32

6 Cisco Unified Computing System Cisco UCS is a next-generation data center platform that unites computing, network, and storage access. The platform, optimized for virtual environments, is designed using open industry-standard technologies. It aims to reduce the total cost of ownership (TCO) and increase business agility. The system integrates a low-latency; lossless 10 Gigabit Ethernet unified network fabric with enterprise-class, x86-architecture servers. It is an integrated, scalable, multichassis platform in which all resources participate in a unified management domain. The main components of the Cisco UCS are: Computing: The system is based on an entirely new class of computing system that incorporates blade and rack servers based on the Intel Xeon processor 5500, 5600, and E series. Network: The system is integrated onto a low-latency, lossless, 10-Gbps unified network fabric. This network foundation consolidates LANs, SANs, and high-performance computing networks, which are separate networks today. The unified fabric lowers costs by reducing the number of network adapters, switches, and cables needed and by decreasing the power and cooling requirements. Virtualization: The system unleashes the full potential of virtualization by enhancing the scalability, performance, and operation control of virtual environments. Cisco security, policy enforcement, and diagnostic features are now extended to virtualized environments to support changing business and IT requirements. Storage access: The system provides consolidated access to both the SAN and NAS over the unified fabric. By unifying storage access, Cisco UCS enables access to storage over Ethernet, Fibre Channel, FCoE, and Small Computer System Interface over IP (iscsi). This capability provides customers with a choice of storage access plus investment protection. In addition, server administrators can preassign storage access policies for system connectivity to storage resources. The result is simplified storage connectivity and increased productivity. Management: The system uniquely integrates all the components within the solution. This single entity can be effectively managed using Cisco UCS Manager. Cisco UCS Manager has an intuitive GUI, a commandline interface (CLI), and a robust API to manage all system configuration and operations. Cisco UCS is designed to deliver: Reduced TCO and increased business agility Increased IT staff productivity through just-in-time provisioning and mobility support A cohesive, integrated system that unifies the technology at the data center; the system is managed, serviced, and tested as a whole Scalability through a design for hundreds of discrete servers and thousands of virtual machines and the capability to scale I/O bandwidth to match demand Industry standards supported by a partner ecosystem of industry leaders 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 6 of 32

7 Cisco UCS C240 M3 Rack Server Building on the success of the Cisco UCS C-Series M2 Rack Servers, the enterprise-class Cisco UCS C240 M3 Rack Server (Figure 2) enhances the capabilities of the Cisco UCS portfolio in a 2RU form factor. With the addition of the Intel Xeon processor E product family, it delivers significant performance and efficiency gains. Figure 2. Cisco UCS C240 M3 Rack Server The Cisco UCS C240 M3 also offers up to 768 GB of RAM, 24 hard disk drives or SSD drives, and four Gigabit Ethernet LAN interfaces built into the motherboard. The result is a server with outstanding levels of density and performance in a compact package. The Cisco UCS C240 M3 balances simplicity, performance, and density for production-level virtualization and other mainstream data-center workloads. The server is a 2-socket server with substantial throughput and scalability. The Cisco UCS C240 M3 extends the capabilities of Cisco UCS. It uses the latest Intel Xeon processor E series multicore CPUs to deliver enhanced performance and efficiency. These processors adjust the server performance according to application needs and use DDR3 memory technology with memory scalable up to 768 GB for demanding virtualization and large-data-set applications. Alternatively, these servers can have a more costeffective memory footprint for less demanding workloads. Cisco UCS C240 M3 Rack Servers offer these main benefits: Outstanding server performance and expandability: The Cisco UCS C240 M3 offers up to two Intel Xeon processor E or E v2 CPUs, 24 DIMM slots, 24 disk drives, and four 1 Gigabit Ethernet LAN-on-motherboard (LOM) ports to provide outstanding levels of internal memory and storage expandability and exceptional performance. Flexible deployment and easy management: The Cisco UCS C240 M3 can operate in standalone environments or as part of Cisco UCS for greater flexibility and computing density in a small footprint. In a multiple-rack environment, because of the ease of data migration between racks, the complexity of rack management is greatly reduced. Workload scalability: Cisco UCS C240 M3 servers along with four Fusion-io iodrive2 cards, each with 3- TB capacity for x8 PCIe slots, allows organizations to scale to meet growing workload demands, handle greater spikes in traffic, and consolidate infrastructure Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 7 of 32

8 Cisco UCS 1225 VIC A Cisco innovation, the Cisco UCS VIC 1225 (Figure 3) is a dual-port Enhanced Small Form-Factor Pluggable (SFP+) 10 Gigabit Ethernet and FCoE-capable PCIe card designed exclusively for Cisco UCS C-Series Rack Servers. With its half-height design, the card preserves full-height slots in servers for third-party adapters certified by Cisco. It incorporates next-generation converged network adapter (CNA) technology from Cisco, providing investment protection for future feature releases. The card enables a policy-based, stateless, agile server infrastructure that can present up to 256 PCIe standards-compliant interfaces to the host that can be dynamically configured as either network interface cards (NICs) or host bus adapters (HBAs). In addition, the Cisco UCS VIC 1225 supports Cisco Data Center Virtual Machine Fabric Extender (VM-FEX) technology, which extends the Cisco UCS fabric interconnect ports to virtual machines, simplifying server virtualization deployment. Figure 3. Cisco UCS VIC 1225 Architecture Cisco Nexus 5548UP Switch The Cisco Nexus 5548UP (Figure 4) is a 1RU and 10 Gigabit Ethernet switch that offers up to 960 Gbps of throughput and scales up to 48 ports. It offers 32 1 and 10 Gigabit Ethernet fixed SFP+ Ethernet and FCoE or 1/2/4/8-Gbps native Fibre Channel unified ports and three expansion slots. These slots have a combination of Ethernet and FCoE and native Fibre Channel ports. Figure 4. Cisco Nexus 5548UP Switch 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 8 of 32

9 Fusion-io The Fusion iomemory platform (Fusion-io iodrive2) combines the Fusion-io Virtual Storage Layer (VSL) and iomemory modules into a Fusion-io iomemory platform that provides enhanced capabilities for enterprise applications and databases. This platform uses flash-based memory technology to significantly increase data center efficiency with enterprise-class performance, reliability, availability, and manageability. Some of the important features of Fusion-io iodrive2 are: Consistent, low-latency performance: The Fusion-io iomemory platform provides consistent, low-latency access for mixed workloads. The sophisticated Fusion-io iomemory architecture enables nearly symmetrical read and write performance, with best-in-class low-queue-depth performance. This feature makes Fusion-io iodrive2 devices, powered by Fusion iomemory, excellent solutions for a wide variety of real-world, highperformance enterprise environments. Industry-leading capacity: Fusion-io iodrive2 is available in 365-GB, 785-GB, 1,205-GB, and 3-TB capacities to help increase CPU efficiency in even the most data-intensive environments. Cisco UCS and 3-TB Fusion-io iodrive2 adapter specifications are as follows: 3-TB multilevel cell (MLC) flash-memory capacity 1.5-GBps bandwidth (1 MB read) 1.1-GBps bandwidth (1 MB write) 143,000 I/O operations per second (IOPS; 512 bytes random read) 535,000 IOPS (512 bytes random write) 136,000 IOPS (4K random read) 242,000 IOPS (4K random write) 15 microseconds of write latency and 68 microseconds of read latency Hardware supported: All Cisco UCS M3 servers Software supported: Cisco UCS Manager 2.1 and later Hardware and Software Components The configuration presented in this document is based on the following main software and hardware components. Hardware Requirements One Cisco UCS C240 M3 Rack Server (Intel Xeon processor E v2 at 2.4 GHz) with 256 GB of memory One Cisco UCS 1225 VIC Two Cisco UCS 6248UP Fabric Interconnects Two Cisco Nexus 5548 switches Two 3-TB Fusion-io iodrive2 cards Client Hardware Requirements One Cisco UCS C210 M2 Rack Server 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 9 of 32

10 Software Requirements Cisco UCS Firmware Release UCS 2.2(1b) Cisco Integrated Management Controller (IMC) Release 1.5(4) Red Hat Enterprise Linux (RHEL) Version 6.4 Oracle Database 11g Enterprise Edition Release (64-bit) Oracle ASM Library: Generic Linux Version Swingbench Release Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 10 of 32

11 Cisco UCS C240 M3 Server BIOS Settings The BIOS tests and initializes the hardware components of a system and boots the operating system from a storage device. In a typical computational system, several BIOS settings control the system s behavior. Some of these settings are directly related to the performance of the system. Figure 5 shows the BIOS settings that control performance. Figure 5. BIOS Settings 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 11 of 32

12 Infrastructure Setup This section provides important design and configuration information for deployment of Oracle Database 11g on Cisco UCS C240 M3 servers configured with Fusion-io iodrive2 cards. Cisco UCS Manager 2.2 supports integration of Cisco UCS C-Series Rack Servers with Cisco UCS Manager through direct connection to the fabric interconnects. Fabric extenders are not required in this configuration. This option enables Cisco UCS Manager to manage the Cisco UCS C-Series servers using a single cable for both management traffic and data traffic. For complete details about single-wire management through direct-connect mode, please refer to Integration_UCSM2-2/b_C-Series-Integration_UCSM2-2_chapter_0110.html. Network Design Figure 6 shows the network connectivity from the Cisco UCS C240 M3 to the Fabric Interconnects. The server is configured using direct-connect mode. Each port on the Cisco UCS VIC1225 on the Cisco UCS C240 M3 server is attached to the server ports on Fabric Interconnects A and B through a 10-Gbps Small Form-Factor (SFP) cable. The paths shown in multiple colors in the figure carry both management and data traffic. Figure 6. Logical Connectivity of Cisco UCS C240 M3 and Cisco UCS 6248UP Fabric Interconnect 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 12 of 32

13 Fusion-io iodrive2 Configuration Oracle Database was configured on a mirrored group of 3-TB Fusion-io iodrive2 cards. Some of the main steps for configuring Fusion-io iodrive2 cards on the Cisco UCS C240 M3 server are presented here. Step 1. Verify that the latest drivers for RHEL 6.4 are installed for each virtual NIC (vnic) interface on the Cisco UCS C240 M3. The latest Ethernet NIC (enic) drives can be downloaded from Extract the driver from the downloaded ISO file and extract the driver for Cisco VIC 1225 from the Linux\Network\Cisco\12x5x\RHEL\RHEL6.4 directory. root@fusionio-c240-1 downloads]# rpm -ivh kmod-enic rhel6u4.el6.x86_64.rpm Preparing... ########################################### [100%] 1:kmod-enic ########################################### [100%] [root@fusionio-c240-1 downloads]# [root@fusionio-c240-1 downloads]# modinfo enic filename: /lib/modules/ el6.x86_64/extra/enic/enic.ko version: license: GPL v2 author: Scott Feldman <scofeldm@cisco.com> description: Cisco VIC Ethernet NIC Driver srcversion: F55DFDB B7E94F5F5 alias: pci:v d sv*sd*bc*sc*i* alias: pci:v d sv*sd*bc*sc*i* alias: pci:v d sv*sd*bc*sc*i* depends: vermagic: el6.x86_64 SMP mod_unload modversions Step 2. Install the latest firmware and utilities for the Fusion-io iodrive2 cards. root@fusionio-c240-1 Software Binaries]# rpm -ivh iomemory-vsl el6.x 86_ el6.x86_64.rpm Preparing... ########################################### [100%] 1:iomemory-vsl ########################################### [100%] [root@fusionio-c240-1 Utilities]# ls fio-util el6.x86_64.rpm [root@fusionio-c240-1 Utilities]# rpm -ivh fio-util el6.x86_64.rp m Preparing... ########################################### [100%] 1:fio-util ########################################### [100%] [root@fusionio-c240-1 Firmware]# ls fio-firmware-fusion noarch.rpm fusion_ fff [root@fusionio-c240-1 Firmware]# rpm -ivh fio-firmware-fusion n oarch.rpm Preparing... ########################################### [100%] 1:fio-firmware-fusion ########################################### [100%] [root@fusionio-c240-1 Firmware]# 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 13 of 32

14 Step 3. Format the Fusion-io card and attach it. ~]# fio-format /dev/fct0 /dev/fct0: Creating block device. Block device of size GBytes ( GiBytes). Using block (sector) size of 4096 bytes. WARNING: Formatting will destroy any existing data on the device! Do you wish to continue [y/n]? y WARNING: Do not interrupt the formatting! If interrupted, the fio-sure-erase utility may help recover from format errors. Please see documentation or contact support. Formatting: [====================] (100%) / /dev/fct0 - format successful. [root@fusionio-c240-1 ~]# fio-format /dev/fct1 /dev/fct1: Creating block device. Block device of size GBytes ( GiBytes). Using block (sector) size of 4096 bytes. WARNING: Formatting will destroy any existing data on the device! Do you wish to continue [y/n]? y WARNING: Do not interrupt the formatting! If interrupted, the fio-sure-erase utility may help recover from format errors. Please see documentation or contact support. Formatting: [====================] (100%) /dev/fct1 - format successful. [root@fusionio-c240-1 ~]# [root@fusionio-c240-1 ~]# fio-attach /dev/fct0 Attaching: [====================] (100%) / fioa - attached. [root@fusionio-c240-1 ~]# fio-attach /dev/fct1 Attaching: [====================] (100%) / fiob - attached. [root@fusionio-c240-1 ~]# 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 14 of 32

15 Step 4. Format each of the Fusion-io cards (/dev/fioa and /dev/fiob) into four partitions of 675 GB each. ~]# parted /dev/fioa GNU Parted 2.1 Using /dev/fioa Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt (parted) mkpart primary 2048s 675GB (parted) print Number Start End Size File system Name Flags kB 675GB 675GB primary (parted) mkpart primary 675GB 1350GB (parted) mkpart primary 1350GB 2025GB (parted) mkpart primary 2025GB 2700GB (parted) print Model: Unknown (unknown) Disk /dev/fioa: 3000GB Sector size (logical/physical): 4096B/4096B Partition Table: gpt Number Start End Size File system Name Flags kB 719GB 719GB primary 2 719GB 1439GB 719GB primary GB 2158GB 719GB primary GB 2878GB 719GB primary (parted) quit Information: You may need to update /etc/fstab Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 15 of 32

16 Oracle Database 11g Configuration Oracle Database was configured on a mirrored group of 3-TB Fusion-io iodrive2 cards. Some of the main steps for configuring Oracle Database 11g are presented here. Step 1. Before installing Oracle Database, verify that all the prerequisite components are installed on RHEL 6.4 (see Step 2. Download and install the Oracle ASM packages on the Red Hat kernel (see Step 3. Verify that Oracle ASM is configured with the following parameters: [oracle@fusionio-c240-1 ~]$ oracleasm configure ORACLEASM_ENABLED=true ORACLEASM_UID=oracle ORACLEASM_GID=oinstall ORACLEASM_SCANBOOT=true ORACLEASM_SCANORDER="fi" ORACLEASM_SCANEXCLUDE="sd" ORACLEASM_USE_LOGICAL_BLOCK_SIZE="true" Step 4. Set the asm_diskstring parameter (alter system-set asm_diskstring='orcl:*' scope=both sid='*';) Step 5. Create an Oracle ASM disk with the four partitions created on each of the Fusion-io cards. /usr/sbin/asmtool -C -l /dev/oracleasm -n fioa1 -s /dev/fioa1 -a force=yes /usr/sbin/asmtool -C -l /dev/oracleasm -n fioa2 -s /dev/fioa2 -a force=yes /usr/sbin/asmtool -C -l /dev/oracleasm -n fioa3 -s /dev/fioa3 -a force=yes /usr/sbin/asmtool -C -l /dev/oracleasm -n fioa4 -s /dev/fioa4 -a force=yes /usr/sbin/asmtool -C -l /dev/oracleasm -n fiob1 -s /dev/fiob1 -a force=yes /usr/sbin/asmtool -C -l /dev/oracleasm -n fiob2 -s /dev/fiob2 -a force=yes /usr/sbin/asmtool -C -l /dev/oracleasm -n fiob3 -s /dev/fiob3 -a force=yes /usr/sbin/asmtool -C -l /dev/oracleasm -n fiob4 -s /dev/fiob4 -a force=yes Step 6. Before creating the Oracle ASM disk group, verify that all the disks appear as candidate disks. SQL> select header_status,sector_size from v$asm_disk; HEADER_STATU SECTOR_SIZE CANDIDATE 4096 CANDIDATE 4096 CANDIDATE 4096 CANDIDATE 4096 CANDIDATE 4096 CANDIDATE Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 16 of 32

17 CANDIDATE 4096 CANDIDATE rows selected. Step 7. Create the Oracle ASM disk group. CREATE DISKGROUP fiodg NORMAL REDUNDANCY FAILGROUP fio1 DISK 'ORCL:fioa1', 'ORCL:fioa2', 'ORCL:fioa3', 'ORCL:fioa4' FAILGROUP fio2 DISK 'ORCL:fiob1', 'ORCL:fiob2', 'ORCL:fiob3', 'ORCL:fiob4' ATTRIBUTE 'sector_size'='4096', 'compatible.asm' = ' ', 'compatible.rdbms' = ' '; Step 8. After the Oracle ASM disk group is configured, create Oracle Database with both data files and log files on the same disk group. Because the Fusion-io iodrive2 3-TB cards have a default sector size of 4 KB, the Oracle ASM disk group was created with a sector size of 4096 bytes. Step 9. Install Oracle Database and Grid (Release ) infrastructure on the Cisco UCS C240 M3 server with the Oracle user. Oracle binaries are installed on the local disk of the server. The data and redo log files reside in the Oracle ASM disk group created on the Fusion-io cards. Step 10. To enable restoration during database failure, verify that the archive redo logs and backup files are configured on a separate NAS device. Performance and Failover Analysis Workload Description In this solution, Swingbench is used for OLTP workload testing. Swingbench is a simple-to-use, free, Java-based tool used to generate a database workload and perform stress testing using various benchmarks in the Oracle Database environment. Swingbench provides four separate benchmarks: Order Entry, Sales History, Calling Circle, and Stress Test. In the workload testing described in this section, the Swingbench Order Entry benchmark was used. The Order Entry benchmark is based on the Swingbench Order Entry schema and resembles the TPC-C benchmark with respect to types of transactions. The workload uses a very balanced read-to-write ratio of 60 to 40 and can be designed to run continuously and test the performance of a typical order-entry workload against a small set of tables, producing contention for database resources Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 17 of 32

18 Test Scenarios In an enterprise solution deployment, system administrators and solution architects need to identify both the performance of databases and the server restoration process in the event of a hardware or software failure. This section discusses three test scenarios that evaluate the performance of Fusion-io cards deployed on Cisco UCS C240 M3 servers and the best practices for backing up and restoring the database in the event of any hardware failure. Performance validation using OLTP workload: This test scenario evaluates the main server characteristics, such as CPU, memory, and the total IOPS value generated when scaling from 100 to 450 OLTP users. Performance impact of failure of one Fusion-io card: This test scenario analyzes the impact on end-user response time and throughput of the failure of one of the Fusion-io cards. Restoration of database from archive and backup logs: In the unlikely event of a server failure, database administrators (DBAs) and other administrators should have a strategy to restore the database. The scenario measures the total restoration time for a database of about 1.8 TB. Performance Validation Using OLTP Workload This scenario evaluates the performance of Oracle Database deployed on a Cisco UCS C240 M3 server with two Fusion-io cards and from 100 to 450 users. System resource use, such as CPU, memory, and disk I/O use, was measured using the nmon analyser tool. The number of transactions per second (TPS) was measured from Oracle Automatic Workload Repository (AWR) reports generated at 10-minute intervals for OLTP users, with the number of users ranging from 100 to 450. Oracle Enterprise Manager was configured for the database to measure I/O, throughput, and disk latency. Figure 7 shows linear scalability when scaling from 100 to 450 OLTP users. The total number of transactions per second at 450 users as measured in the Oracle AWR report was about 14,000 TPS. Figure 7. Transaction Throughput for OLTP Workload 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 18 of 32

19 CPU Utilization The Cisco UCS C240 M3 server was equipped with a 2-socket Intel Xeon processor E v2 running at a 2.4- GHz clock frequency. As shown in Figure 8, the total CPU utilization ranged from about 18 percent to a maximum of about 80 percent when scaling from 100 to 450 OLTP users. CPU utilization in the system gradually increased, reflecting the linear scalability of the workload. Figure 8. CPU Utilization for OLTP Workload Figure 9 shows the CPU utilization captured by the nmon analyser tool during scalability testing for 450 OLTP Swingbench users Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 19 of 32

20 Figure 9. CPU Utilization for 450-User OLTP Workload I/O Performance The Cisco UCS C240 M3 server was equipped with two 3-TB Fusion-io iodrive2 cards configured as a mirrored Oracle ASM disk group. As shown in Figure 10, the maximum IOPS value generated was about 39,400 IOPS when the system was scaled to 450 OLTP users. The total disk read-and-write wait time was less than 1 millisecond (ms) when the system was scaled from 100 to 450 users. The workload read-to-write ratio was about 60:40 for 450 users. These IOPS values were measured both through nmon analyser and Oracle AWR reports Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 20 of 32

21 Figure 10. I/O Performance 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 21 of 32

22 Figure 11 shows the Oracle AWR data captured during scalability testing for 450 OLTP Swingbench users. The highlighted numbers shows the read and write I/O generated during the 450-user OLTP test. Figure 11. Oracle AWR Data for 450 Users 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 22 of 32

23 Oracle Enterprise Manager Snapshot Figure 12 shows the Oracle Enterprise Manager snapshot for 450 OLTP users. The data corroborates the statistics presented in the previous section. As shown in the figure, the total number of IOPS was about 39,000, whereas I/O throughput was about 350 MBps. Figure 12. Oracle Enterprise Manager Statistics Performance Impact of Failure of One Fusion-io Card Oracle Database deployed on a Cisco UCS C240 M3 server was configured with two 3-TB Fusion-io iodrive2 cards with mirrored Oracle ASM disk groups. This section shows the variation in end-user throughput during the failure of one of the Fusion-io cards. It details the system characteristics during the following events: Normal phase: Both the Fusion-io cards are in normal working condition with the OLTP user workload. Failure phase: One of the Oracle ASM disk groups was taken offline, and the impact on system characteristics such as CPU use and IOPS was analyzed. Restore phase: The failed card was brought back online, and the system characteristics were analyzed until normal working conditions were achieved Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 23 of 32

24 Figure 13 shows the TPS and IOPS values during each of the three phases. Figure 13. Performance Impact During Fusion-io Card Failure 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 24 of 32

25 Figure 14 shows server CPU utilization for all phases during the attach and detach processes for one Fusion-io card. Figure 14. CPU Utilization During Fusion-io Card Failure 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 25 of 32

26 Normal Phase: 450-User Workload Figure 15 shows a snapshot from Oracle Enterprise Manager during a system load of 450 OLTP users. It shows total IOPS of about 39,000 IOPS, throughput of about 380 MBps, and less than 1 ms of disk latency. There were about 70 active sessions at this point. The total TPS measured through an Oracle AWR report were about 13,800 TPS. Figure 15. Normal Conditions: 450-User Workload 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 26 of 32

27 Failure Phase: 450-User Workload In this phase, the disk group on one of the Fusion-io cards was taken offline. Swingbench was running a 450-user OLTP workload. As shown in Figure 16, there was no change in total IOPS or total throughput per second. Disk latency increased slightly, but it remained less than 1 ms, and there was a very small increase in total active user sessions. The total TPS value measured by an Oracle AWR report was about 13,400 TPS. Figure 16. Failure Condition: 450-User Workload 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 27 of 32

28 Restore Phase: 450-User Workload This phase shows the system characteristics when the Oracle ASM disk group on the Fusion-io card was brought back online. This phase consists of two main events: Oracle ASM synchronizes with the online Cisco ASM disk group. After synchronization, the database works under normal conditions. As shown in Figure 17, when Oracle ASM synchronizes with the online Oracle ASM disk group, a high disk latency of about 5 ms occurs. The number of active sessions on the database increases from about 70 to 170 sessions. The disk IOPS value drops to about 30,000 IOPS, and the total TPS value measured through Cisco AWR was about 10,500 TPS. The synchronization phase was completed in about 50 to 60 minutes for a database of about 1.8 TB Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 28 of 32

29 Figure 17. Restore Condition: 450-User Workload with Oracle ASM Disk Synchronization After synchronization, the database works under normal conditions. As shown in Figure 18, the system achieves the expected 39,000 IOPS and throughput of about 380 MBps. The number of transactions was about 13,500 TPS Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 29 of 32

30 Figure 18. Restore Condition: 450-User Workload After Synchronization with Normal Conditions 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 30 of 32

31 Restoration of Database from Archive and Backup Logs Oracle Database (single instance) deployed on a Cisco UCS C240 M3 server with Fusion-io cards was analyzed for several types of failures. The previous section analyzed the failure of one Fusion-io card (two Fusion-io cards are installed with Oracle ASM mirroring) and detailed the performance during the failure and database synchronization stages. Some failures may cause the system to become unusable and require intervention by system administrators and DBAs for business continuity. These failures are discussed here: Fusion-io card failure: A single Fusion-io card is installed on a Cisco UCS C240 M3 server. Oracle Database failure: The database needs restoration from backup. Local disk failure: The operating system and Oracle Relational Database Management System (RDBMS) binaries need rehosting. CPU or memory failure: Workload may need to be moved to another server to reduce downtime. The last two cases are outside the scope of this document. Case 3 requires disk replacement and installation of OS and RDBMS binaries. Case 4 may require another server. However, the use of Cisco UCS service profiles can greatly reduce the downtime from these two failure scenarios. In the first case, a single Fusion-io card is installed on the server, and a failure occurs. Upon this failure, the Fusion-io card is replaced on the server, and the Oracle database files are restored from a NAS backup. In the second case 2, an Oracle Database failure occurs. There are different types of Oracle Database failures. Some can be mitigated with simple steps taken by DBAs or with the help of Oracle support. However, in some cases the only option is to restore the database files, requiring full tablespace recovery using Oracle RMAN. In the test bed, because of the high I/O throughput of PCIe-based Fusion-io cards, multiple tests were run to ascertain the backup and restoration of Oracle Database on external storage. The total time taken for RMAN full database restoration was about 52 minutes for a 1.8-TB database. Conclusion The Cisco UCS C240 M3 Rack Server provides high performance and internal storage from a single vendor known for its long history of innovation in architecture, technology, partnerships, and services. The Cisco UCS C240 M3 is designed for organizations with increasing data and storage demands. The enterprise-class Cisco UCS C240 M3 is equipped with Intel Xeon processor E technology, which delivers top-class performance, energy efficiency, and flexibility. Cisco engineers have designed the Cisco UCS C240 M3 with Cisco UCS 1225 VIC to handle a broad range of applications, including workgroup collaboration, virtualization, consolidation, massive data infrastructure, and small and medium-size business (SMB) databases. The Cisco UCS C240 M3, with its comprehensive stack of technological offerings, offers excellent computing and networking performance and provides internal storage in one cohesive system to meet the challenges of virtualization technologies and the growing demands of SMBs. The combination of the Cisco UCS C240 M3 server, Fusion-io iomemory platform, and Oracle Database dramatically improves performance. The solution delivers fast access to information, enabling organizations to capture important insights to help guide business decisions. Even with a single Oracle Database instance, the solution helps organizations achieve exceptional performance with a failover strategy that meets today s needs, at a lower cost. The solution achieves about 39,000 IOPS with a read-write ratio of 60:40, which is comparable to performance achieved by enterprise-class storage targeted at SMBs. In the event of a single Fusion-io card failure, 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 31 of 32

32 the server sustains the throughput and IOPS achieved during normal conditions. During a server failure, the system downtime is decreased by direct attachment to the Cisco UCS rack server, unleashing the full capabilities of Cisco UCS statelessness. By deploying Cisco UCS, Fusion-io, and Oracle Database together, companies gain an enterprise-class, high-performance, low-latency, reliable, and scalable solution. The solution can help increase customer and user loyalty, gain competitive advantage, and lower operating costs. The performance study discussed in this document showed that the Cisco UCS C240 M3 Rack Server with Fusion-io cards improves the application performance and operation efficiency of Oracle Database. For More Information Printed in USA C / Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 32 of 32