Rick Stehno Principle Product Marketing Engineer / DBA. Replicated Database

Transcription

1 LSI Nytro WarpDrive Acceleration Card Increases Performance of an Oracle MySQL Replicated Database Rick Stehno Principle Product Marketing Engineer / DBA SYNOPSIS This document is a guide to help increase MySQL database performance on the master server while simultaneously reducing replication lag time on the slave server by implementing the PCIe flash based LSI Nytro WarpDrive storage acceleration card. By offering high performance with low latency and minimal CPU burden, the Nytro WarpDrive card can help maximize transactional I/O performance for applications like online transaction processing (OLTP), data warehousing (DW), and data mining. By delivering the I/O performance of hundreds of traditional hard disk drives (HDDs), the Nytro WarpDrive card is designed to improve user satisfaction and productivity by enabling applications that use the MySQL database to meet challenging service- level agreements (SLAs) for response times and throughput. This solution guide describes a reference architecture as a basis for planning and deploying PCIe compatible flash storage, including a typical use case scenario. Advantages of Using Flashbased Storage with a MySQL Replicated Database As OLTP and DW applications have demanding requirements for short response times and high throughput, it makes it difficult for database administrators (DBAs) to maintain these systems as the number of users grows and the amount of data increases. During the application life cycle, performance bottlenecks might originate in one or more areas, including the network, processor, or storage devices, such as HDDs. Correcting a bottleneck in one area might cause a bottleneck in another area. In addition, if a customer wants to take advantage of MySQL s built-in replication features to horizontally scale out their database, replication process performance can become a concern. MySQL offers different methods of replicating a database to provide for high availability, disaster recovery, scalability for high performance, or a combination of any of these. MySQL replication provides the capability for all the slave database(s) to synchronize data to the master database. The default replication operation mode in MySQL is asynchronous, meaning that the master server pushes changes to the slaves once the updates are committed. Replication has thus little overhead in transaction execution latency and the slaves eventually become consistent with the state of the master. In the following use case, the semi-synchronous replication method was used. When deploying semi-synchronous replication, the master server halts the commit of a transaction, similar to synchronous replication, until the master receives at least one acknowledgement from any of the slave servers. Another benefit of using semi-synchronous replication is that the acknowledgement from the slave server only states that the new data has been received and saved to the relay log file, it doesn t mean that the slave server has actually applied the data to the database which results in better performance. Flash-based storage provides a new tool for DBAs to help solve their MySQL database performance issues. For example, 5 ms is the typical response time for a small data read from a HDD. Flash-based devices such as the Nytro WarpDrive card can complete the same reading <50μs on average, an improvement of several orders-of-magnitude in response time. Flash-based storage also provides performance levels that fall between the performance levels of HDDs and DDR3 memory. Initial implementations of flash-based drives (SSDs) were intended to replace HDDs in directattach storage or RAID subsystems. The recent innovation of mounting SSDs on a PCIe card alleviates throughput constraints that are caused by the storage interface. The Nytro WarpDrive card supports bandwidth up to 4GB/s, up to 260,000 input/output operations per second (IOPs) and capacity up to 3.2TB.

2 LSI Nytro WarpDrive Card LSI Nytro WarpDrive Card Application Acceleration Built for Enterprise Workloads Host Offloaded Architecture LSI SandForce Driven Flash Modules Market segment leading performance DuraClass Technology for enhanced reliability and endurance Offloaded architecture does not burden the host which is critical as the Flash begins to wear out Enterprise Ready Most widely deployed SAS Drivers in the industry Extensive qualification testing/rdt Consistent performance through time and across CPU speed bins Bootable Battery-less Power Fail Protection Management Tools Future Benefits Workload based tuning Enhanced Architectural optimizations and integration Industry Hardened Drivers Breadth of capacity: 200 GB to 3.2 TB SLC & MLC Major OEM Wins in Production Image 1: the LSI Nytro Warpdrive provides multiple benefits Overview of the LSI Nytro WarpDrive Application Acceleration Card The Nytro WarpDrive card offers high performance with low latency and minimal CPU burden. The Nytro WarpDrive card is designed to maximize transactional I/O performance for MySQL databases and for other applications that require high-performance computing cycles. The Nytro WarpDrive card performs consistently across reads and writes, regardless of workload, by using industry-standard and widely deployed LSI SAS software for easier system integration and management, as well as a faster time-to-market. Configuring the Linux Operating System and the MySQL Database for Optimum I/O Performance You can achieve significant database I/O improvement by following some simple guidelines in setting up the Linux operating system and the MySQL database with or without replication. This section will describe the configuration steps for both the Linux operating system and the MySQL replicated database with the Nytro WarpDrive acceleration card. Configure the Nytro WarpDrive Application Acceleration Card to use as a Filesystem This section explains some actions you can take to tune the Nytro WarpDrive acceleration card for enhanced performance using the Oracle Linux with Unbreakable Enterprise Kernel (UEK) operating system. The configuration steps that follow will work in both the UEK and Oracle Linux operating systems. The following steps configure the Nytro WarpDrive card as a single filesystem for each of the master and slave servers for MySQL databases. Other options would be to use multiple Nytro WarpDrive acceleration cards and apply RAID to provide fault tolerance for extra data protection. 1. Align the Nytro WarpDrive card on a 1MB boundary in the Linux operating system. The following example shows how to partition the Nytro WarpDrive card as a single partition starting on a 1MB boundary: echo 2048,, sfdisk -us /dev/sda Disk /dev/sda: cylinders, 255 heads, 63 sectors/track Units = sectors of 512 bytes, counting from 0 Device Boot Start End #sectors Id System /dev/sda Linux /dev/sda Empty /dev/sda Empty /dev/sda Empty LSI Nytro WarpDrive Acceleration Card Increases Performance of an Oracle MySQL Replicated Database 2

3 2. Bypass journaling when you create a file system. Instead of using EXT-3 for the file system, use EXT-4 with journaling turned off, which eliminates double writes to the Nytro WarpDrive card in certain cases. The reduction in writes increases performance and prolongs the life of the Nytro WarpDrive card. 3. For creating an EXT-4 filesystem with journaling turned off: $ mkfs t ext4 /dev/sda1 or mkfs.ext4 /dev/sda1 After creating the EXT-4 filesystem, journaling will be on as the default. To verify this, execute the tune4fs command: $ tune4fs l /dev/sda1 grep Filesystem features Notice the has_journal feature should be listed. To turn off journaling, execute: $ tune4fs -O ^has_journal /dev/sda1 4. Verify that journaling is disabled, execute the following command and make sure has_journal is not listed. $ tune4fs l /dev/sda1 grep Filesystem features 5. When mounting the new EXT-4 device (noatime option is described below): $ mount -o noatime /dev/sda1 /mountpoint 6. Modify the kernel I/O scheduler. The I/O schedulers in the latest Linux operating system releases have new I/O capabilities, including options to modify these settings at boot time. In the tests described in this document, the DEADLINE I/O scheduler was used with the noatime file system mount option. To invoke the DEADLINE scheduler for the entire system, add this line to the /etc/grub.conf file, and bounce the system. $ kernel /vmlinuz el6uek.x86_64 ro root=/dev/mapper/vg_nytro-lv_root elevator=deadline An alternative way to invoke the DEADLINE IO scheduler is to modify the /etc/rc.local file with the following command: $ echo deadline > /sys/block/sda/queue/scheduler 7. To verify that the DEADLINE scheduler is enabled, issue the following statement as root. $ cat /sys/block/sda/queue/scheduler noop anticipatory [deadline] cfq 8. In addition to changing the I/O scheduler, the noatime file system mount option was invoked, which was added to the /etc/fstab file. This option eliminates the need for the system to create writes to the file system when objects are only being read. This option also enables faster access to the files, plus the benefit of less wear on the Nytro WarpDrive card. This example shows how the /etc/fstab entry invokes the noatime option: /dev/sda1 /osfc ext4 defaults,noatime 1 2 An alternative to invoking the noatime option is to specify it when executing the mount command: $ mount o noatime /dev/sda1 /osfc 9. Performance of the Nytro WarpDrive card can be improved by increasing Queue Depth (QD) from the default of 128 to 256 or higher, depending on the database workload. Since the latency of the Nytro WarpDrive card is so small, more I/O operations/commands can be run in parallel on the Nytro WarpDrive card compared to a HDD. In order to modify queue depth, the nr_requests parameter will need to be modified to the same or larger value of the new queue depth value. Here are examples of modifying both the nr_requests and queue_depth parameters for /dev/sda which was used in our tests: $ echo 512 > /sys/block/sda/queue/nr_requests $ echo 512 > /sys/block/sda/device/queue_depth LSI Nytro WarpDrive Acceleration Card Increases Performance of an Oracle MySQL Replicated Database 3

4 10. Further performance benefits can be recognized by telling the operating system to complete an I/O request on the same CPU that initiated it. In some workloads this can provide a nice jump in performance. To enable this feature on the Nytro WarpDrive device: echo 1 > /sys/block/sda/queue/rq_affinity 11. Another operating setting that can provide more performance is setting the rotational setting to 0 which means that the device is non-rotational, it is a flash device: echo 0 > /sys/block/sda/queue/rotational Note: To persist these settings across reboots, place all of these commands in the /etc/rc.local file. Configuring the MySQL Database for Semi-Synchronous Master/Slave Replication This section describes the modifications needed to enable semi-synchronous replication using MySQL. 1. To enable semi-synchronous master/slave replication in MySQL 5.6, the following configuration settings need to be applied. Master Database: Plugins:The semi-synchronous plugin will need to be installed before starting the master server with the my.cnf changes for replication. From a mysql prompt: INSTALL PLUGIN rpl_semi_sync_master SONAME semisync_master.so ; These are the database settings for the master to enable replication used for this test case: /etc/my.cnf [mysqld] log_bin=/u04/binlog/mysql-bin sync_binlog=1 binlog_format= MIXED server_id = 10 # semi-sync replication rpl_semi_sync_master_enabled=1 datadir = /u04/datadir max_connections = 600 innodb_flush_log_at_trx_commit=1 innodb_buffer_pool_size = 4G innodb_thread_concurrency=49 slow_query_log=1 slow_query_log_file = /u04/slow_query_logfile.log innodb_log_group_home_dir = /u04/logs innodb_data_home_dir = /u04/data innodb_file_per_table=1 innodb_flush_log_at_trx_commit=1 innodb_log_files_in_group=4 innodb_log_file_size=2g innodb_thread_concurrency=0 innodb_flush_method = O_DIRECT innodb_write_io_threads=64 innodb_io_capacity=20000 expire-logs-days=1 LSI Nytro WarpDrive Acceleration Card Increases Performance of an Oracle MySQL Replicated Database 4

5 Slave Database: Plugins: The semi-synchronous plugin will need to be installed before starting the slave server with the my.cnf changes for replication. From a mysql prompt: INSTALL PLUGIN rpl_semi_sync_slave SONAME semisync_slave.so ; These are the database settings for the slave to enable replication used for this test case: /etc/my.cnf [mysqld] log_bin=/u04/binlog/mysql-bin binlog_format= MIXED datadir = /u04/datadir server_id = 2 # for semi-sync replication rpl_semi_sync_slave_enabled=1 innodb_flush_log_at_trx_commit=1 innodb_buffer_pool_size = 4G innodb_thread_concurrency=49 slow_query_log_file = /u04 innodb_log_group_home_dir = /u04/logs innodb_data_home_dir = /u04/data innodb_file_per_table=1 innodb_log_files_in_group=4 innodb_log_file_size=2g innodb_flush_method = O_DIRECT innodb_io_capacity=20000 relay_log = /u04/logs/mysql-relay-log log_slave_updates = 1 skip_slave_start read_only = 1 expire-logs-days=1 2. To start the replication process after starting both the master and slave databases, execute the following commands from each database: Master Database mysql prompt: mysql> show master status\g; File: mysql-bin Position: 120 Binlog_Do_DB: Binlog_Ignore_DB: executed_gtid_set: 1 row in set (0.00 sec) LSI Nytro WarpDrive Acceleration Card Increases Performance of an Oracle MySQL Replicated Database 5

6 Slave Database mysql prompt: Benchmark Results mysql> Reset slave; mysql> change master to master_host= master_hostname, master_user= repl, master_password= mysql, master_log_file= mysql-bin , master_ log_pos=120; mysql> start slave; mysql> start slave IO_THREAD; Before adding the Nytro WarpDrive card to the test configuration, the Linux operating system and the MySQL database were configured using techniques described in the above sections. OLTP benchmarks were executed to establish a baseline before and after implementing the Nytro WarpDrive card. Quest Benchmark Factory software was used for these benchmarks with transactions per second (TPS) and average response time results documented. The slave metric Seconds_Behind_Master, indicating the number of seconds that the slave is behind the master in processing, was captured during these tests as well. Baseline Results The results from the benchmark before installing the Nytro WarpDrive acceleration card into the servers were the following: Average TPS: 312 Average Response Time:.32 seconds Slave Seconds_Behind_Master: 1172 seconds These tests were executed with a configuration that resulted in the best results from HDD right before performance started to drop off due to overloading the HDDs. Implementing Nytro WarpDrive After implementing the Nytro WarpDrive card on both servers to replace the HDD for the database files, the following benchmarks were executed: Test 1: Modification of the average TPS and average response time benchmarks to add more load to the system to take advantage of the lower latency which the Nytro WarpDrive card provides The Slave Seconds Behind Master test using the same configuration as with HDD Average TPS: 996 Average response time:.1 second Note: By installing the Nytro WarpDrive card in both servers, it was possible to double the userload on the system, while improving average response times using the same 4G buffer size of the database. LSI Nytro WarpDrive Acceleration Card Increases Performance of an Oracle MySQL Replicated Database 6

7 MySQL TPC-C Benchmark with Semi-Synchronous Replication MySQL TPC-C Benchmark with Semi-Synchronous Replication Transactions Per Second Over 3x more TPS Seconds Response times are a third less 0 HDD NWD 0 HDD NWD TPS Avg Response Time(s) Test 2 Slave Seconds_Behind_Master: 0 Conclusion In our benchmark testing with a Nytro WarpDrive card and a MySQL database utilizing the master/slave semi-synchronous replication feature, large performance gains were realized and could directly improve enterprise environments using the configuration presented in this guide for workloads that are: 1. I/O bound due to large number of HDDs 2. I/O bound due to large HDD throughput requirements (MBps) By implementing the Nytro WarpDrive card in a MySQL replicated environment, not only will the Nytro WarpDrive card increase performance on the master, but it will also reduce the number of seconds that the slave is behind the master in replicating data. For more information and sales office locations, please visit the LSI website at: lsi.com North American Headquarters San Jose, CA T: (within U.S.) T: (outside U.S.) LSI Europe Ltd. European Headquarters United Kingdom T: [+44] LSI KK Headquarters Tokyo, Japan T: [+81] LSI and the LSI & Design logo are trademarks or registered trademarks of LSI Corporation in the United States and/or other countries. All other brand and product names may be trademarks of their respective companies. LSI Corporation reserves the right to make changes to the product(s) or information disclosed herein at any time without notice. LSI Corporation does not assume any responsibility or liability arising out of the application or use of any product or service described herein, except as expressly agreed to in writing by LSI Corporation; nor does the purchase, lease, or use of a product or service from LSI Corporation convey a license under any patent rights, copyrights, trademark rights, or any other of the intellectual property rights of LSI Corporation or of third parties. LSI products are not intended for use in life-support appliances, devices, or systems. Use of any LSI product in such applications without written consent of the appropriate LSI officer is prohibited. Copyright 2014 by LSI Corporation. All rights reserved. > 0514 WP