Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

Transcription

1 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN This white paper describes how the performance of an Oracle 12c Online Transaction Processing (OLTP) database can be accelerated using Dell Fluid Cache for SAN. Technical White Paper Krishna Kamal Kapa Dell Database Solutions Engineering April 2014

2 THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell. Dell, the Dell logo, PowerEdge, PowerVault, OpenManage, EqualLogic, Compellent, KACE, FlexAddress, are trademarks of Dell Inc. Other Dell trademarks may be used in this document. Intel, Pentium, Xeon, Core and Celeron are registered trademarks of Intel Corporation in the U.S. and other countries. Microsoft, Windows, Windows Server, Internet Explorer, MS-DOS, Windows Vista and Active Directory are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Red Hat and Red Hat Enterprise Linux are registered trademarks of Red Hat, Inc. in the United States and/or other countries. Novell and SUSE are registered trademarks of Novell Inc. in the United States and other countries. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Citrix, Xen, XenServer and XenMotion are either registered trademarks or trademarks of Citrix Systems, Inc. in the United States and/or other countries. Broadcom and NetXtreme are registered trademarks of Broadcom Corporation. Qlogic is a registered trademark of QLogic Corporation. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and/or names or their products and are the property of their respective owners. Dell disclaims proprietary interest in the marks and names of others. April 2014 Version Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

3 Table of contents Executive summary... 4 Introduction... 4 Dell Fluid Cache for SAN overview... 5 Solution design and reference architecture... 6 Traditional rotational hard drive-based storage solution... 6 Fluid Cache for SAN solution... 7 Test methodology... 8 Performance results and analysis... 9 Conclusion Acknowledgement...12 References...12 Improving Oracle OLTP database performance with Dell Fluid Cache for DAS...12 Hardware configuration details...13 Server configuration...13 Storage configuration Network Switching Hardware Software configuration details Oracle configuration settings Kernel parameter settings User Security Limits Settings Database parameter settings Udev settings Settings for Baseline Configuration Udev Settings for Fluid Cache Configuration Server Configuration Profile Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

4 Executive summary Application performance may be limited by multiple factors such as low interconnect speeds, slow drive response times, cumbersome software stacks, and maxed out storage controllers. Application performance acceleration is key to business and customer success, and can be achieved in multiple ways. Adopting SSD technology and server caching software in your data center are a few ways to improve application performance. Dell has built an enterprise-class caching solution named Dell Fluid Cache for SAN that can radically accelerate application performance. Dell Fluid Cache for SAN is a highly available write-back server-side caching accelerator that utilizes high speed PCIe SSDs as a shared, caching solution with an open compute cache pool layer. Fluid Cache for SAN runs in a flexible and low latency Dell Compellent SAN environment. The key differentiations of Fluid Cache for SAN from other host-side caching solutions are its global cache pool distributed across multiple servers, write back cache policies, easy access media and fully integrated with the storage array features and management. This white paper can help you configure the Dell Fluid Cache for SAN solution and demonstrates how the performance of Oracle Database Online Transaction Processing (OLTP) can be accelerated using this solution compared to the traditional hard disk-based solution. Figure 1: Advantages of Fluid Cache for SAN in a 3 node Dell lab test of a hardware stack enabled with Dell Fluid Cache for SAN on an OLTP cluster running on an Oracle database 4.4x more transactions per second 9.6x reduction in average response time at one sec SLA ~4x more concurrent users at one second or less average response time Introduction This white paper studies the performance of an Oracle OLTP database in a Fluid Cache for SAN solution. It compares the performance of the Fluid Cache solution against a traditional storage solution (without any host based caching). The key benefits of Fluid Cache for SAN compared to the traditional solution are: Increased transactions per second (TPS) Decreased average response times (ART) On-the-fly scaling to meet growing business demand No single point of failure from server to SAN. 4 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

5 Dell Fluid Cache for SAN overview Dell Fluid Cache for SAN is server-side caching accelerator software. Dell Fluid Cache for SAN makes high-speed PCIe SSDs a shared, distributed cache resource. Dell Fluid Cache for SAN is deployed on clusters of Dell PowerEdge systems within a SAN environment and connected using RoCE-enabled Ethernet adapters. A minimum of three compatible Dell PowerEdge servers systems supporting Dell PCI Express Flash SSDs are required to configure a Fluid Cache cluster. Dell validates up to eight systems per cache pool with a maximum of 1.6TB of cache per server, and a maximum capacity of 12.8TB of cache per cache pool. Figure 2: Fluid Cache for SAN Components Dell Fluid Cache for SAN uses three types of networks: Cache Network: A high speed, low latency private network used by Fluid Cache to connect the PowerEdge systems in the Fluid Cache cluster. Management Network: The connection to Dell Compellent Enterprise Manager, which manages Fluid Cache and the SAN. SAN Network: The network that Dell Compellent Storage Center uses to handle data connectivity within the SAN. 5 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

6 Solution design and reference architecture This section provides the solution design of Fluid Cache for SAN and traditional rotational hard drivebased storage solution used as the baseline configuration. Traditional rotational hard drive-based storage solution The baseline configuration is deployed on two PowerEdge R820 systems connected to Compellent SC8000 controller and SC220 expansion storage using Fibre Channel Brocade Gbps switches. Redundant Dell Networking S GbE switches are used for Oracle RAC Cache Fusion. The SC220 enclosure is populated with 20 x 300GB 15k RPM SAS drives as Tier 1 and 4 x 1TB SATA 7.2 RPM drivers as Tier 2. The two systems are installed with Red Hat Enterprise Linux 6 Update 4 and Oracle Cluster Database 12c with a pluggable database deployed on the two nodes. The DATA diskgroup containing database files, controlfiles, and redolog files are placed in Tier 1 storage and FRA diskgroup containing backups and archive logs are placed in Tier 2. Figure 3: Baseline Configuration Architecture 6 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

7 Fluid Cache for SAN solution Dell Fluid Cache for SAN cluster is deployed with Dell Fluid Cache software on two PowerEdge R820 systems and one PowerEdge R620 system which is added as a management server for Fluid Cache. The two PowerEdge R820 systems act as cache servers by hosting 2 x 350GB Express Flash PCIe SSDs in each server. The Fluid cache clustered servers are connected to Dell Networking S GbE low latency network using Mellanox ConnectX card for the cache network of Fluid Cache. Using this network, the cached blocks are transferred across the Fluid Cache cluster. Dell Compellent Enterprise Manager monitors, manages and analyzes storage center and Fluid Cache clusters from a centralized management console. Compellent Enterprise Manager is installed on a dedicated Windows system for monitoring and enabling the cache to the Compellent disks. Fluid cache for SAN utilizes Dell Express Flash PCIe SSDs and Dell Compellent storage and moves the most frequently accessed data closer to compute resources minimizing data travel from storage through the network, improving response time and accelerating both reads and write performance. Figure 4: Fluid Cache for SAN Architecture For more details on the hardware configuration, see the Hardware configuration details section of this white paper. 7 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

8 Test methodology This section describes the test methodology when running the baseline and Fluid Cache analysis. To show case the Dell Fluid Cache for SAN solution benefits, several benchmark tests were conducted in the Dell Solutions lab simulating a production online transaction processing OLTP database. The steps below provide a detailed understanding of the tests performed during the Dell Fluid Cache performance study An Oracle OLTP TPC-C workload is simulated using Quest Benchmark Factory tool with a scale resulting in 1TB data being populated in the Oracle 12c pluggable database. A Compellent replay is taken on the database volumes after the data load as backup and recovery strategy. This replay is used for restoring the data after each set of OLTP Benchmark iteration to have consistent results. Baseline benchmarking is performed on the two node cluster with a 1/50 th keying and think-time values in the Benchmark factory tool to stress the database. Fluid Cache for SAN software is enabled on all the three nodes and using Compellent Enterprise Manager enabled the caching for the disks. Fluid Cache for SAN benchmarking is then performed on the cache enabled disks using the same 1/50 th keying values. Keying Time creates a delay before a transaction executes, simulating activities such as data entry that a user performs before executing the transaction. Think Time creates a delay after a transaction executes, simulating a user s decision-making process before a user proceeds to the next transaction. Figure 5: Cached Disks and Non-Cached Disks 8 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

9 Figure 6: Solution Design Performance results and analysis This section presents the performance results and analysis of the two solutions described in the previous pages. The baseline performance without caching provided a TPS of 449 and for the same configuration after enabling the Fluid Cache software with a redundant cache of 700GB resulted in a significant number of 1979 TPS. Baseline configuration cannot scale due to the storage disk bottleneck which resulted in less number of TPS. The Fluid Cache result provides a dramatic increase in TPS of 4.4x compared to the baseline results Figure 7 shows a 4.4x improvement in TPS for Fluid Cache compared to the baseline. 9 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

10 ART TPS Figure 7: OLTP Transactions Per Second 2500 OLTP Transactions/Sec (TPS) Baseline Fluid Cache 0 User Load As shown in Figure 8, 1 second = 1000ms (milliseconds) response time is considered as the valid benchmark threshold (SLA). The baseline configuration scaled upto 300 users with the response time of 281 ms. The response time of the fluid cache solution is 29 ms at the same user load which provides a 9.6x times better response time than the baseline configuration Figure 8 shows a 9.6x improvement in response time for Fluid Cache compared to the baseline configuration for an ART of 1 sec. Figure 8: OLTP Average Response Time OLTP Average Response Time (ART) Baseline Fluid Cache User Load 10 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

11 User Load The below analysis provides the maximum user load that the two solutions are able to deliver while keeping the response time at one seconds or less SLA. The baseline user load scaled up to 500 users and the Fluid Cache solution to 1900 users. So, Fluid Cache delivered ~4x more user load compared to the baseline configuration at 1 sec SLA. Figure 9 shows a ~4x increase in user load for Fluid Cache compared to the baseline. Figure 9: User Load at One Second SLA User Load at 1 sec SLA Baseline Fluid Cache Conclusion The Dell Fluid Cache solution brought data closer to compute by adding shared caching layer. It is a best in class solution offering an optimal and cost effective way of accelerating application performance. It supports global cache pool distributed across multiple servers, write-back cache policies, and the easy access media that make it distinct from other host side caching solutions. Essentially it enables customers to radically improve business by exceeding end user needs and expectations. Dell Fluid Cache offers multiple features - boosted performance, easy to adopt, flexible and scalable to implement etc., which make it a compelling choice for customers to adopt. Dell Fluid Cache for SAN provides a reliable, efficient, and effective architecture, and key takeaways are: Boosted transactional performance while increasing concurrent users with low latency. Reduce response time by bringing data closer to compute. Maximize computational horsepower with the Dell Fluid Cache for SAN solution utilizing Dell PCI Express Flash drives. Scale database performance on the fly with the addition of Express Flash drives. Ensure data integrity with write-back SAN caching technology. 11 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

12 Acknowledgement I would like to thank the following Dell team members for providing significant support during development and review of this whitepaper: Balamurugan B, Makransky Karen, Phani MV, Reddy Mahesh, Reddy Ramamohan, Gopu, Narasimha Reddy References Dell.com/Support is focused on meeting your needs with proven services and support. DellTechCenter.com is an IT Community where you can connect with Dell Customers and Dell employees for the purpose of sharing knowledge, best practices, and information about Dell products and installations. Referenced or recommended Dell publications: Oracle Best Practices on Compellent Storage Center Improving Oracle OLTP database performance with Dell Fluid Cache for DAS How to deploy Oracle 12cR1 on RHEL6/Oracle Linux 6: Referenced or recommended Oracle publications: Oracle 12c documentation Oracle Database Storage Administrators Guide Oracle Database Performance Tuning Guide Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

13 Hardware configuration details Server configuration 2 x Dell PowerEdge R820 servers (12 th generation) are used as database servers with the below components. Component Database server CPU Memory Fibre Channel interfaces Network interfaces Operating system disk Description 2 X PowerEdge R820 BIOS: idrac7 ESM FW: v (build 4) LC2: v x Intel Xeon CPU E5-4657L v2 (12 cores per processor, 2.40GHz; 30MB cache) 512GB per server (32 X 16GB at 1600MT/s) QLogic QLE 2662, dual-port 16GB Fibre Channel HBA per server Driver version: 5.47 Firmware version: Broadcom x10Gb SR/SFP+ + 2x1Gb BT network daughter card per server Driver version: Firmware version: x 146GB 15k SAS Disks in RAID10 Dell PERC H710p RAID controller Driver version: rh1 Firmware version: PCIe SSDs 2* 350GB Dell PowerEdge Express Flash PCIe SSD, 2.5 PCIe-SSD backplane PCIe extender Adapter card Mellanox adapter for Fluid Cache PCIe-SSD extender 0 Firmware version 1.0 FW: Driver: rh2 (native) MT27500 Family [ConnectX-3] Version Firmware version: Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

14 1 x Dell PowerEdge R620 used as the Fluid Cache management server with the below components Component Fluid Cache management server CPU Memory Fibre Channel Interfaces Network interfaces Operating system disk Mellanox adapter for Fluid Cache Description 1 X PowerEdge R620 BIOS: idrac7 ESM FW: v (build 7) LC2: v x Intel Xeon CPU E (8 cores per processor, 2.20GHz ; 20MB cache) 64GB per server (4 X 16GB at 1600MT/s) Qlogic QLE 2662, dual-port 16GB Fibre Channel HBA per server Driver version : 5.47 Firmware version: Intel x 520 DP 1Gb BT +DP 10Gb SFP+ Rack NDC Driver version: k Firmware version: Family x 146GB 15k SAS disks in RAID10 Dell PERC H710p Mini Driver version rh1 Firmware version MT27500 Family [ConnectX-3] Version Firmware version: Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

15 Storage configuration Storage is provided by a dual-controller Compellent array connected to a single SC220. Each controller is deployed with 2 dual-port 16 Fibre Channel HBAs that serve front-end connectivity and I/O. Backend connectivity is provided by a quad-port 6Gbps SAS controller per controller. Component Storage controller Description 2x SC8000 Storage firmware revision Storage Center 6.5 Enclosure 1 X SC220 (24 x 2.5 drives) 20 X 300GB 15K SAS (HS) disks 4 X 1 TB 7.2 SATA (HS) disks Backend connectivity Frontend connectivity 1 x quad-port 6Gb SAS per controller Two dual-port 16Gb FC QLE 2662 HBA per controller Network Switching Hardware Below table provides the list of switching components used for SAN and LAN connectivity. Component Fibre Channel switch Dell Fluid Cache switch RAC cluster interconnect switch Client IP switch Description 2 x Brocade 6510 Gen5 16Gb FC switches 2 x Dell Networking S4810 switches 2 x Dell Networking S4810 switches 2 x Dell Networking S55 switches Software configuration details This section provides the software versions, software installation, and software configuration steps for all the storage solutions described in the Solution design and reference architecture section. Component Operating system Database type Description RHEL 6.4 Kernel el6.x86_64 (x86_64) Oracle 12cR1 ( ) 2-node RAC 15 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

16 Dell Fluid Cache for SAN v Database size 1.2TB deployed on a pluggable database ASM Yes with external redundancy Oracle System Global Area (SGA) 60GB per node Load generator Quest BenchMark Factory TPC-C style workload Storage management software Compellent Enterprise Manager Version : 6.5 Oracle configuration settings Kernel parameter settings kernel.msgmnb = kernel.msgmax = fs.file-max = fs.aio-max-nr = net.ipv4.ip_local_port_range = net.core.wmem_default = net.core.wmem_max = net.core.rmem_max = net.core.rmem_default = kernel.shmmni = 4096 kernel.shmmax = kernel.shmall = kernel.sem = Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

17 User Security Limits Settings oracle soft nofile 1024 oracle hard nofile oracle soft nproc oracle hard nproc oracle soft stack oracle hard stack grid hard nofile grid soft nproc * soft memlock unlimited * hard memlock unlimited Database parameter settings db_writer_processes 4 processes sessions open_cursors transactions undo_retention Udev settings Settings for Baseline Configuration This section describes the udev rules that need to be set for the Baseline solution. 1. It is recommended to set the udev rules for the block devices to be used for ASM disk using their WWIDs. Identify the WWIDs for all the virtual disks that are to be configured for Oracle ASM by running the following command $> scsi_id --page=0x83 --whitelisted --device=/dev/sdx where sdx is the name of the block device to be used for Oracle 17 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

18 Example output: sdc b e000008a54ea5356a 2. Edit the /etc/udev/rules.d/99-dell_oracle.rules file (create a new file if it doesn t already exist) and add the following lines to set the permissions needed for the baseline configuration. Replace each of the <WWID_for_sdX> below with the correct equivalent WWID result obtained from step 1 for that sdx block device. KERNEL=="dm*", PROGRAM="scsi_id --page=0x83 --whitelisted -- device=/dev/%k", RESULT=="36000d31000ece ", NAME="fldc0", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm*", PROGRAM="scsi_id --page=0x83 --whitelisted -- device=/dev/%k", RESULT=="36000d31000ece ", NAME="fldc1", OWNER="grid", GROUP="asmadmin", MODE="0660" KERNEL=="dm*", PROGRAM="scsi_id --page=0x83 --whitelisted -- device=/dev/%k", RESULT=="36000d31000ece ", NAME="fldc2", OWNER="grid", GROUP="asmadmin", MODE="0660" Udev Settings for Fluid Cache Configuration For setting the permissions for the Fluid Cache disks, edit the /etc/udev/rules.d/99-dell_oracle.rules file as below: KERNEL=="fldc0", OWNER:="grid", GROUP:="asmadmin", MODE="0660" KERNEL=="fldc1", OWNER:="grid", GROUP:="asmadmin", MODE="0660" KERNEL=="fldc2", OWNER:="grid", GROUP:="asmadmin", MODE="0660" Server Configuration Profile Dell idrac7 with Lifecycle Controller provides the ability to generate a human readable snapshot of server configuration using the XML feature. This single file contains all BIOS, idrac, LC, Network, and Storage settings. The file captured from one server can be applied to other servers, even across different server models. 18 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN

19 Configuration XML operations are performed via both RACADM and WS-MAN commands and can be directed to or from network shares (NFS / CIFS). The Improving Oracle OLTP 12c database performance with Dell Fluid Cache for SAN whitepaper also provides a server configuration profile (xml) to directly import the Dell Engineered configuration into Dell servers. You can download the configuration profile here. 19 Accelerating Oracle OLTP 12c Database Performance with Dell Fluid Cache for SAN