LAMP Performance on Dell PowerEdge 1855 Blade Servers Running VMware ESX Server

Transcription

1 LAMP Performance on Dell PowerEdge 1855 Blade Servers Running VMware ESX Server The Dell Scalable Enterprise Technology Center Labs team ran performance characterization and sizing tests on a Dell PowerEdge 1855 blade server running VMware ESX Server software to simulate a virtualized LAMP (Linux, Apache, MySQL, and PHP) environment. Systems engineers and IT architects can use these test results to help determine the appropriate number of virtual machines (VMs) to host on ESX Server for a typical transactional LAMP-based workload as well as how to optimize the configuration for VMs and achieve effective workload and VM scaling. BY AMRESH SINGH; J. CRAIG LOWERY, PH.D.; RUDRAMUNI B.; AND SCOTT STANFORD Related Categories: Characterization Dell PowerEdge blade servers Dell/EMC storage Virtualization VMware Visit for the complete category index. The open source LAMP application stack which consists of Linux, Apache, MySQL, and PHP is a commonly used, reliable platform for Web servers and provides Web services. Running the LAMP stack on virtual machines (VMs), powered by virtualization software such as VMware ESX Server, can offer a solid, robust platform for small and medium-sized enterprises hosting Web-based application development. These virtualized LAMP environments can provide exceptional price/performance ratios thanks to the flexibility of virtual platforms, which are designed to enhance server consolidation and disaster recovery. To further explore these factors, the Dell Scalable Enterprise Technology Center Labs team conducted performance characterization and sizing tests of VMs hosted on ESX Server running on Dell PowerEdge 1855 blade servers. The LAMP stack components, PowerEdge 1855 server, and Dell/EMC storage were configured to support an online transaction processing (OLTP) workload environment. To generate the load, a Dell-developed tool the online DVD Store application 1 was used. The number of VMs running on the ESX Server host was gradually increased along with the OLTP workload per LAMP VM with the goal of achieving a high level of VM and ESX Server utilization. By running a simultaneous number of VMs, each under an OLTP workload, Dell engineers were able to demonstrate how ESX Server running on a single server blade within the Dell PowerEdge 1855 chassis could support increasing amounts of work while still effectively utilizing the underlying physical server and storage resources. The results from these performance and sizing tests can help systems engineers and IT architects determine effective workload throughput and VM utilization for their data centers. 1 The online DVD Store is an e-commerce test application created by Dell engineers. This application comprises a database component, a Web application layer, and a multithreaded driver program to drive the OLTP workload. The online DVD Store can be downloaded from linux.dell.com/dvdstore. Reprinted from Dell Power Solutions, May 26. Copyright 26 Dell Inc. All rights reserved. DELL POWER SOLUTIONS 1

2 Component Server Processor Memory Storage Configuration Building the test environment This section describes how the Dell engineers designed and deployed ESX Server on the Dell PowerEdge 1855 server blade. Figure 1 lists the hardware components used, and Figure 2 shows the test environment architecture. Specific steps used to deploy and configure VMware ESX Server 2.5.1, the guest operating systems, and the LAMP stack components as well as the steps taken to set up and create the database are also explained in this section. Configuring the SAN After setting the appropriate Fibre Channel switch zoning and EMC Navisphere storage group configurations for the storage area network (SAN), the Dell team created four logical units (LUNs). Each LUN was configured as RAID- and consisted of three 66 GB, 1, rpm disks. One of the LUNs was used to create a RAID-5 VMware ESX Server File System (VMFS) volume for the ESX Server system, while the other three LUNs were dedicated to the VMs using raw device mapping (RDM). 2 One of the key advantages of using raw devices is that databases can deploy sophisticated data-management algorithms for maximum performance. Installing ESX Server One server blade within the Dell PowerEdge 1855 chassis Intel Xeon processors at 3.6 GHz with 1 MB or 2 MB L2 cache and Intel Hyper-Threading Technology 4 GB of system memory Dell/EMC CX4 with 15 Fibre Channel disks Switch Brocade SilkWorm 385 Clients Dell PowerEdge 75 Network Dell PowerConnect 5224 Figure 1. Hardware components used in test environment The Dell team used the recommended procedures described in the Dell VMware ESX Server Deployment Guide 3 to install ESX Server software on the Dell PowerEdge 1855 server blade. ESX Server software was installed on the internal 73 GB SCSI hard disks of the PowerEdge The two 73 GB disks were configured as a mirror for fault tolerance. One of the two on-board Gigabit Ethernet 4 network interface cards (NICs) was dedicated to the service console; the other NIC was dedicated to the VMs. Creating VMs and installing the guest operating systems After the installation of ESX Server software, two VMs were created using VMware VirtualCenter software. Red Hat Enterprise Linux 3 Update 4 was installed on both VMs. Each VM was configured with one virtual CPU and 512 MB of RAM, one virtual disk (a.vmdk file residing on the VMFS volume), and one physical disk with direct access to a system LUN using RDM. VMTools was installed on all guest operating systems. Installing MySQL and PHP MySQL 5. was installed from the prebuilt binaries distributed as Red Hat Package Manager (RPM ) packages obtained from the MySQL Web site. These RPM packages are optimized, and Dell and MySQL recommend using them for production environments. PHP was compiled from sources available at the PHP Web site. The Apache Web server, which is included by default with the Red Hat Enterprise Linux 3 Update 4 distribution, was used to provide the Web application layer. The details of these installations are shown in the Configuring the LAMP stack sidebar in this article. Figure 3 lists the software versions used in the test environment. Setting up the database MySQL data files were stored on the system LUN using RAID- (striping only, with no redundancy). The index and table files were VMWare VirtualCenter Brocade SilkWorm 385 switch RAID-5 VMFS RAID- VM1 RAID- VM2 Fibre Channel module on chassis RAID- VM3 Dell PowerEdge 1855 Dell/EMC CX4 storage array ESX Server Dell PowerConnect 5224 Dell PowerEdge 75 running driver program for generating stress Figure 2. Test environment architecture Red Hat Enterprise Linux 2 Starting with the ESX Server 2.5 release, RDM allows a special file in a VMFS volume to act as a proxy for raw devices. The mapping file contains the metadata used to manage and redirect disk access to a physical device. This type of mapping merges VMFS manageability with raw device performance. For more information about RDM, visit 3 For the ESX Server best practices for Dell PowerEdge servers, visit 4 This term does not connote an actual operating speed of 1 Gbps. For high-speed transmission, connection to a Gigabit Ethernet server and network infrastructure is required. 2 DELL POWER SOLUTIONS Reprinted from Dell Power Solutions, May 26. Copyright 26 Dell Inc. All rights reserved. May 26

3 Component Version VMware ESX Server VMware VirtualCenter Apache MySQL PHP stored on the local disk. Because the local disk was a mirrored pair (RAID-1), index and table files resided on a protected volume. For production-level LAMP deployments, system designers should consider using SAN-based RAID-5 or RAID-1 LUNs for storing data files in addition to conducting regularly scheduled backups. 5 Log file writes are generally sequential, and thus the log files were stored on the local disk only. For the database sizes tested, the I/O capabilities of the local disks were sufficient to handle the transaction log I/O activity. The DVD Store application uses both MyISAM and InnoDB tables. The database parameters were tuned as described in the Configuring the LAMP stack sidebar in this article (Web server included with Red Hat Enterprise Linux 3 Update 4) 5..7 (binaries distributed as RPM packages from the MySQL network at dev.mysql.com/downloads) 5..4 (tarball available from Figure 3. Version information for software used in the test environment To test the MySQL database performance in an OLTP environment, the Dell team built a medium-sized database (1 GB) using the build scripts provided with the online DVD Store utility. These data files were then loaded into the database. A multithreaded Microsoft Windows OS based driver application supplied with the online DVD Store utility was used to simulate an OLTP workload. This driver program was run on Dell PowerEdge 75 clients running Microsoft Windows Server 23, Enterprise Edition. To simulate a real-world scenario, the think time was set for two seconds and the percentage of new users was configured at 2 percent. The parameter used for measuring the performance and scalability of ESX Server software and the LAMP VMs was orders per minute (OPMs). OPMs can also be thought of as transactions per minute. To scale the workload, the test team gradually increased the number of threads (or simulated users) from 1 to 2, and then to 3. Testing LAMP performance and sizing Various factors can affect and benefit ESX Server installations and their hosted VMs. The Dell team examined how Intel Hyper- Threading Technology and various processor cache sizes affect LAMP performance. Analyzing these test results can help systems engineers and IT architects make informed decisions about the appropriate number of VMs to deploy for OLTP workloads and VM scaling. Test procedures The DVD Store driver program was launched from the PowerEdge 75 clients after the database load completed. The driver program writes the orders per minute to the console in plain-text format. For ESX Server, the CPU utilization was measured using esxtop and vmkusage, 6 while VM utilization was captured with VirtualCenter. The tests were repeated for each of the scenarios described in this section. In each scenario, the number of threads in the driver program was increased by increments of 1 users, starting with 1 users and increasing up to 3 users per VM. Effect of Intel Hyper-Threading Technology Dell PowerEdge 1855 blade servers support Intel Xeon processors with Intel Hyper-Threading Technology, which enables one physical processor to act as two virtual processors. To measure the effect of Hyper-Threading Technology on a two-processor server, the Dell team conducted two sets of tests: one without Hyper-Threading Technology and one with Hyper-Threading Technology enabled. In the first scenario, each VM was scheduled to run on physical processors and 1. The service console also used physical processor. In the second scenario, with Hyper-Threading Technology enabled, the VMs were scheduled to run on all four virtual processors. Figure 4 shows the results from these tests; in each scenario, the OPMs are measured per VM. As shown in Figure 4, OPMs per VM increased approximately 8 percent with Hyper-Threading Technology enabled on a two-processor PowerEdge server blade. Hyper-Threading Technology can support more OPMs because ESX Server can take advantage of four logical CPUs (LCPUs) and thus allocate more CPU cycles to the virtual processors assigned to the VMs. In addition, more processing power is available to the OPMs Hyper-Threading Technology disabled Hyper-Threading Technology enabled Figure 4. Effect of Intel Hyper-Threading Technology on ESX Server performance 5 For higher levels of database I/O activity, system designers may want to consider using RAID-1 LUNs. 6 The esxtop and vmkusage monitoring tools are available with ESX Server software. For more information, visit Reprinted from Dell Power Solutions, May 26. Copyright 26 Dell Inc. All rights reserved. DELL POWER SOLUTIONS 3

4 CONFIGURING THE LAMP STACK To configure the LAMP stack used in the test environment, the Dell team used the following procedures. Setting up raw disk mapping. After creating the VMs, the Dell team used the ESX Server Management User Interface (MUI) to add a second hard disk to each VM. When prompted for the type of virtual disk, they selected System LUN/Disk. In the following screen, they choose the appropriate LUN and assigned a meaningful metadata file name.* This newly added hard drive was then visible as /dev/sdb when the guest OS booted up. After the guest OS installation completed, the Dell team logged in as root and created a 2 GB partition using a partition utility such as fdisk. This partition was then recognized as /dev/sdb1. Next, the team issued the following command to enable the MySQL user with read and write access to the raw device: # chown mysql.mysql /dev/sdb1 Setting up MySQL. After installing Red Hat Enterprise Linux 3 Update 4, the Dell team took the following steps: 1. Stop any running MySQL service by issuing the following command as root: # /etc/init.d/mysql stop 2. Remove any previous installation of MySQL: # rpm -nodeps e 'rpm qa grep i MySQL' 3. Install the MySQL binaries: # rpm ivh MySQL-client i386.rpm MySQL-devel i386.rpm MySQL-server i386.rpm 4. Edit the /etc/my.cnf file and tune the parameters as shown in Figure A. (If the /etc/my.cnf file is not present, copy the file /usr/share/mysql/my-large.cnf to /etc/my.cnf.) 5. Initialize the raw device by starting the MySQL database. This process may be time-consuming because MySQL must initialize the 2 GB of raw space. The progress can be monitored from the /var/lib/mysql/hostname.err file: # /etc/init.d/mysql start 6. Change the parameter innodb_data_file_path from /dev/sdb1:2gnewraw to /dev/sdb1:2graw. Notice that Parameter key_buffer_size table_cache read_buffer_size read_rnd_buffer_size myisam_sort_buffer_size the word new has been removed from the parameter value. Then, restart MySQL: # /etc/init.d/mysql restart Configuring PHP. Next, the Dell team obtained the php-5..4 tarball from and untarred the PHP source as root: # tar zxvf php-5..4.tar.gz They then built the PHP module from source as root: # cd php-5..4 #./ configure --with-apxs2=/usr/sbin/apxs2 --with-mysqli=/usr/bin/mysql_config # make # make install Configuring the Web server. The Dell team added the following lines in /etc/httpd/conf/httpd.conf: KeepAlive On KeepAliveTimeout 5 ServerName hostname:8 AddType application/x-httpd-php.php They then restarted the Apache service: # service httpd restart Value 16M 32M Setting up the database. Finally, the Dell team obtained the online DVD Store application from linux.dell.com/dvdstore. They untarred the archive and followed the step-by-step procedure described in the readme file available in the DVD Store directory. 2M 4M 16M thread_cache 25 innodb_data_file_path Figure A. Parameters for the /etc/my.cnf file /dev/sdb1:2gnewraw * Detailed information about raw device mapping (RDM) is outside the scope of this article. For more information about RDM, visit 4 DELL POWER SOLUTIONS Reprinted from Dell Power Solutions, May 26. Copyright 26 Dell Inc. All rights reserved. May 26

5 CPU utilization (percent) ESX Server kernel and service console related operations when Hyper-Threading Technology is enabled. In the Dell tests with Hyper-Threading Technology enabled, the VM scheduler scheduled the two VMs to run on all four LCPUs. When Hyper-Threading Technology was not used, the ESX Server system had to balance the requirements of the service console, ESX Server kernel, and VM operations on the two physical CPUs. The online DVD Store application is a CPU-intensive workload. When Hyper-Threading Technology is disabled, there are fewer OPMs per VM Figure 5. ESX Server CPU utilization as number of VMs increase Number of VMs per ESX Server system For the next test scenario, the Dell team created a third VM with a virtual hardware configuration identical to the other LAMP VMs. All the VMs were configured without the VMware Virtual SMP add-on module, which enables a VM to use multiple processors. Figure 5 shows the test results. As shown in Figure 5, with three VMs the ESX Server host reaches a CPU utilization of approximately 9 percent. Although an ESX Server system can sustain more than 9 percent CPU utilization, best practices recommend keeping spare CPU cycles available to accommodate sudden increases in workload. With CPU cycles in reserve, ESX Server systems can provide enough CPU resources to the VMs for situations such as a VMotion event or a temporary spike in OPMs on one or more VMs on the same host. For some IT organizations, a maximum targeted ESX Server host CPU utilization threshold may be 75 percent, while for others the goal may be 95 percent. These maximum CPU utilization levels can be determined in various ways, one of which is throughput. To illustrate how throughput can be used as a determining metric for establishing maximum desired ESX Server system CPU utilization, Figure 6 shows the decrease in OPMs per VM as the number of VMs and number of users per VM increase. OPMs decrease slightly when the number of VMs increases from two to three. 1 VM 2 VMs 3 VMs This decrease in OPMs per individual VM as the number of VMs increases is expected because the ESX Server host must sustain more work with a finite pool of system resources. As shown in Figures 5 and 6, if this small decrease in OPMs per VM is acceptable, a maximum of three VMs can be hosted on a single PowerEdge server blade for an OLTP workload similar to that generated by the online DVD Store application. Another way to use throughput as a metric to set desired system utilization is to evaluate the overall OPMs per ESX Server host rather than at the individual VM level. As shown in Figure 6, at two active VMs with each VM sustaining 3 simulated users, the aggregate ESX Server host OPM throughput is 1,53 (765 per VM). At three active VMs, the overall OPM throughput rate increases to 2,4 OPM (68 per VM). By adding a third VM and incurring slightly less throughput per VM, the ESX Server system can be more effectively utilized and support 25 percent more OPMs as compared to only running with two VMs. As indicated in Figure 5, this 25 percent increase in aggregate system throughput is achieved by an increase in ESX Server system average CPU utilization. At two active VMs, the ESX Server host CPU utilization was 69 percent, and at three active VMs, the host CPU utilization was 88 percent. These two methods measuring throughput per VM and throughput per ESX Server host use throughput to help determine efficient ESX Server system utilization and to establish quantifiable metrics that can be used to determine appropriate CPU utilization thresholds. However, the data presented in this article represents average values. With any random workload, throughput and utilization will vary during the normal course of a business day, so allowing for spikes in resource utilization is one key criterion that system administrators should consider when determining how many VMs to run on an ESX Server host. Effect of increasing L2 cache To observe the effect of level 2 (L2) CPU cache size, the Dell team replaced the 3.6 GHz 1 MB L2 CPU cache processors in the Dell PowerEdge 1855 server blade with two 3.6 GHz 2 MB L2 CPU cache processors. All the tests were rerun with this set of processors, which had double the amount of L2 cache compared to the previous configurations. Intel Hyper-Threading Technology was enabled on both processors, and all three VMs were configured with one virtual CPU. OPMs Figure 6. ESX Server orders per minute as number of VMs increase 1 VM 2 VMs 3 VMs Reprinted from Dell Power Solutions, May 26. Copyright 26 Dell Inc. All rights reserved. DELL POWER SOLUTIONS 5

6 Number of users ESX Server CPU utilization for 3 VMs with 1 MB L2 cache ESX Server CPU utilization for 3 VMs with 2 MB L2 cache 1 34% 3% 2 6% 52% 3 88% 72% Figure 7. Impact of L2 CPU cache size on ESX Server host CPU utilization CPU utilization (percent) Room for extra VM 1 MB L2 cache 2 MB L2 cache Figure 7 shows that as the number of users (threads) increased, the benefits of the larger CPU cache appeared as a reduction in ESX Server host CPU utilization when compared to the configuration with a smaller L2 cache. The online DVD Store workload is random in nature and creates reads and writes to the underlying database system disks; because the 2 MB CPU L2 cache can hold more data than the 1 MB cache, the online DVD Store accesses data on slower underlying disk subsystems less often. Data can be accessed more quickly in processor cache than in the disk subsystems, so CPU cycles are free to handle other ESX Server kernel scheduling tasks. Examining the results shown in Figure 8 reveals that when three VMs are hosted on a system powered by a CPU with a 2 MB L2 cache, the CPU utilization on the ESX Server system was approximately 7 percent in contrast to 88 percent with a 1 MB L2 cache. As a general rule, increasing L2 cache can improve the ESX Server software s ability to host an additional VM on the ESX server. To test this, an additional VM was added to the ESX Server and its workload was increased to 3 users as with the other three VMs. Test results from running four LAMP VMs with a varied number of users reveal that with the larger L2 cache CPUs, the ESX Server system can sustain up to four VMs while still supporting acceptable OPM throughput rates (see Figure 9). As shown in Figure 9, even when four VMs are hosted on the ESX Server, each VM can support sustainable OPM transaction rates. The OPM amounts shown in Figure 9 are the average per VM, rather than the aggregate for the ESX Server host. For the tests presented in this article, the VMs were under moderate to heavy loads. VMs running under lighter utilization levels could result in a higher consolidation ratio of VMs per ESX Server host. As discussed in the Effect of Intel Hyper-Threading Technology section in this article, the optimal number of VMs regardless of VM utilization levels that can be run on an ESX Server host depends ultimately upon desired factors such as response times, throughput rates, and the overall host utilization levels that meet an IT organization s service-level agreements (SLAs). Thus, even though running four VMs on one host is possible as shown in Figure 9, system administrators running ESX Server hosts in production-level environments may choose not to deploy a fourth VM so that enough Figure 8. ESX Server CPU utilization with larger CPU cache size Average OPMs resources remain available to support utilization spikes or a VM failover via VMotion. Determining appropriate settings for ESX Server performance on Dell blade servers The Dell PowerEdge 1855 blade server with ESX Server software can provide a platform that is well suited for hosting a minimum of two VMs running a medium-sized (1 GB) OLTP workload on a LAMP stack. When Intel Hyper-Threading Technology is enabled, ESX Server hosts can benefit from performance improvements as the number of simultaneous users accessing the databases increases. Best practices also recommend increasing the L2 cache size of physical processors when hosting multiple VMs on one ESX Server host or trying to decrease ESX Server host CPU utilization Figure 9. Average orders per minute processed on each VM given 3 VMs per host and 4 VMs per host The series of tests and analysis described in this article show that, as the number of supported users per VM and the number of VMs increased in the test environment, the ESX Server system continued to provide acceptable levels of OPM throughput. Throughput per VM and aggregate ESX Server host throughput metrics can be used to evaluate and set appropriate ESX Server system CPU utilization levels. Understanding and tracking ESX Server utilization levels and using the methods presented in this article can help IT organizations to meet their SLA targets. 3 VMs per host 4 VMs per host 6 DELL POWER SOLUTIONS Reprinted from Dell Power Solutions, May 26. Copyright 26 Dell Inc. All rights reserved. May 26

7 Acknowledgments The authors extend their thanks to Dave Jaffe and Todd Muirhead for their support and willingness to lend expertise with the online DVD Store application. Amresh Singh is an engineering analyst on the Virtualization Solutions Engineering team within the Dell Enterprise Solutions Engineering Group. Amresh has a bachelor s degree in Information Technology from Indira Gandhi National Open University in India. He is a Red Hat Certified Engineer and a VMware Certified Engineer. His primary areas of interest are virtualization and data center consolidation. J. Craig Lowery, Ph.D., is a senior engineering development manager on the Enterprise Solutions Engineering team within the Dell Product Group. His team is currently responsible for developing products that realize the Dell vision of the scalable enterprise. Craig has a B.S. in Computing Science and Mathematics from Mississippi College and an M.S. and a Ph.D. in Computer Science from Vanderbilt University. Rudramuni B. is an engineering manager for the Enterprise Solutions and OS Engineering groups of the Dell Product Group in Bangalore, India. His responsibilities include delivery of cluster-based solutions, testing, and certification of operating systems on PowerEdge servers. Rudramuni B. has an M.Tech. in Systems Analysis and Computer Applications from Regional College, Surathkal, India. His primary areas of interest include clustering, multi-tiered solutions, and operating systems. Scott Stanford is a systems engineer on the Virtualization Solutions Engineering team within the Dell Enterprise Solutions Engineering Group. Scott s primary areas of interest are virtualization, data center capacity planning and sizing, project management, writing, and team leadership. He has a B.S. from Texas A&M University and an M.S. in Community and Regional Planning from The University of Texas at Austin, and he is pursuing an M.S. in Computer Information Systems at St. Edward s University. Reprinted from Dell Power Solutions, May 26. Copyright 26 Dell Inc. All rights reserved. DELL POWER SOLUTIONS 7