RACKSPACE ONMETAL I/O V2 OUTPERFORMS AMAZON EC2 BY UP TO 2X IN BENCHMARK TESTING EXECUTIVE SUMMARY Today, businesses are increasingly turning to cloud services for rapid deployment of apps and services. Cloud deployments are frequently implemented by using virtual infrastructure, which allows for rapid, economical provisioning; but virtualization might not be the best option for businesses that require the greater control, security, and in particular the performance promised by bare-metal servers. Our engineers at Prowess decided to run a series of tests to determine if performance was significantly higher for a bare-metal cloud server than for a virtual host. We compared high input/output (I/O) database performance for Rackspace OnMetal I/O v2 a bare-metal cloud-server deployment against Amazon Elastic Compute Cloud (Amazon EC2 ) a popular virtual hosting service. Our results found that the Rackspace bare-metal cloud offering handled more than two times the peak transactions per minute (TPM) of the Amazon EC2 virtual deployment. These results are relevant for any company that is looking for high performance for high I/O workloads. Rackspace OnMetal I/O v2 Bare-metal server Single tenant Powered by the Intel Xeon processor E5-2600 v3 product family Amazon Elastic Compute Cloud (Amazon EC2 ) Type i2.4xlarge Virtual-machine servers Dedicated host or instance available as an option Powered by the Intel Xeon processor E5-2670 v2 product family Offers solid-state-drive (SSD) storage SSD Offers SSD storage Figure 1. Comparing infrastructure-as-a-service (IaaS) cloud deployments: Rackspace bare-metal server (left) versus an Amazon virtual machine offering (right) 1
Virtual Versus Bare-Metal Cloud Deployments Businesses have several choices for hosted infrastructure-as-a-service (IaaS) cloud deployments, each offering a different range of options for performance and price. Many of these choices are hosted in virtual environments provided by Amazon Web Services (AWS ), Microsoft Azure, Google Cloud Platform, Rackspace, or other vendors. Virtual machines are convenient for the hosting service because they are easier and more economical to deploy and manage. But virtual servers might not be the best option for some businesses. Many companies prefer to deploy on bare-metal machines, because physical servers can offer: Greater performance, without the overhead or complexity of a hypervisor More reliable infrastructure and consistent workload performance, without interference from noisy neighbors that can impact I/O performance Greater control of and access to the server environment to better meet business needs Improved security, because businesses know exactly what workloads are running on what kinds of physical servers Performance and scalability are of primary importance for companies looking to host online-transaction-processing (OLTP) software for payments processing, inventory control, enterprise-resource planning (ERP), customer-relationship management (CRM), or other tasks, or for businesses using online-analytical-processing (OLAP) databases for data mining and analytics. Comparing Rackspace Bare-Metal Cloud Servers to Amazon EC2 Virtual Servers OnMetal v2 Cloud Servers offer an appealing alternative to virtual hosting, because the solution offers single-tenant, bare-metal cloud servers that can be deployed and scaled quickly. The bare-metal solution eliminates the hypervisor layer and ensures single-tenancy for greater security and control. But how do OnMetal cloud servers actually stack up against a popular virtual-host solution in a real-world OLTP scenario? Which option offers customers the best performance when hundreds of users perform millions of transactions per minute? Our engineers at Prowess decided to put OnMetal v2 Cloud Servers to the test. We compared the performance of SQL databases for identical workloads on two self-service IaaS implementations: OnMetal I/O v2 bare-metal cloud server Amazon EC2 Type i2.4xlarge virtual-server instance 2
Our testing showed impressive performance for OnMetal I/O v2 in two ways: First, peak TPM and new orders per minute (NOPM) for OnMetal I/O v2 were more than twice those for the Amazon EC2 deployment. OnMetal I/O v2 processed 3,865,351 TPM compared to only 1,580,656 TPM for Amazon EC2. Second, the OnMetal I/O v2 implementation scaled faster, reaching maximum performance at 16 concurrent users, compared to 8 for Amazon EC2. Because businesses running on OnMetal I/O v2 can accommodate more users with the same or fewer instances, those businesses will be able to scale more efficiently to meet future demand, compared to businesses running on Amazon EC2. Figure 2 summarizes the TPM results of our testing. 4,000,000 Transactions per Minute Transactions per Minute 3,500,000 3,000,000 2,500,000 2,000,000 1,500,000 1,000,000 500,000 0 2 8 16 24 48 64 128 256 Virtual Users Rackspace OnMetal I/O v2 Amazon Elastic Compute Cloud (Amazon EC2 ) Figure 2. Transactions per minute (TPM) for Rackspace OnMetal I/O v2 (red) versus Amazon Elastic Compute Cloud (Amazon EC2 ) (blue) Applying Test Results to Real-World Environments Because Prowess s tests verified raw TPM without any database-specific tuning, our results suggest that users can expect similar relative performance gains whatever the workload, from OLTP and relational database workloads to any other high-i/o use cases involving large caches, search indexes, or analytics applications. For example, companies using transactional databases for payments processing, ERP, inventory control, operations, finance, or other typical enterprise functions would likely see significant performance benefits from bare-metal cloud servers. Test Methodology To measure performance, we compared TPM and NOPM between: A hosted Rackspace OnMetal I/O v2 bare-metal cloud server running Ubuntu Server 14.04 LTS with MySQL 5.6 An Amazon EC2 virtual host running Ubuntu Server 14.04 LTS with MySQL 5.6 We performed each test run three times, and we fully reset the databases between each test run. We deployed MySQL out of the box without any tuning or optimization. Overall transaction performance for both scenarios would likely be higher with standard tuning techniques for 3
both platforms, but it would be difficult or impossible to compare results if each test bed was optimized individually. By avoiding any changes to the default deployment environment, the scenarios were comparable, allowing for an unbiased evaluation. For each scenario, we configured access, increased the user limit to 256, and then used the HammerDB application to build individual MySQL database schemas and to populate the databases with data. Workload Execution After configuring and populating the database, we verified the configuration by executing pre-trial runs with five users for ten minutes. After resetting the database, we used HammerDB to generate TPC-C benchmark workloads, and we then scaled up virtual users over a five-minute period, from two to a maximum of 256 users. We ran each test for a total of ten minutes after the ramp-up period. During the testing, we monitored resource use for any unexpected patterns by using MySQL Workbench on the Rackspace OnMetal I/O v2 implementation and Amazon CloudWatch on the Amazon EC2 deployment. All test results were recorded using the HammerDB app on both platforms. About TPC-C Benchmark Testing The goal of our testing was to come as close as possible to testing real-world I/O performance in the lab. Common benchmark tests run a standard workload, and then test how quickly a given platform can complete the workload. In contrast, TPC-C benchmark testing is specifically designed to simulate a number of users (for testing purposes, concurrent threads) executing an increasing number of TPM and NOPM against a database until performance degrades. We selected the HammerDB app for testing because the tool was designed based on the TPC-C specification. It is intentionally not optimized for any particular database implementation or system hardware in order to avoid bias. Although our tests were designed to simulate OLTP workload performance, the results offer a valid indication of how any relational database or other high-i/o operations might perform under load for the tested hardware. Test Results In our testing, the OnMetal I/O v2 implementation executed significantly more TPM and NOPM across the entire range of virtual users tested, compared to the Amazon EC2 setup. At peak performance of 16 users, the OnMetal implementation performed 2.4 times more TPM and NOPM than the Amazon EC2 solution (see Table 1). 4
In addition, the OnMetal I/O v2 solution scaled better than the Amazon EC2 solution, reaching peak performance at 16 virtual users, compared to only eight for Amazon EC2. Table 1 and Figure 3 show median performance for increasing numbers of virtual users executing transactions across both configurations. Table 1. Total transactions per minute (TPM) and new orders per minute (NOPM) for Rackspace OnMetal I/O v2 and Amazon Elastic Compute Cloud (Amazon EC2 ); results are the median values for three test runs Users 2 8 16 24 48 64 128 256 Transactions per Minute (TPM) Rackspace Amazon Elastic OnMetal Compute Cloud I/O v2 (Amazon EC2 ) 906,496 355,105 3,401,811 3,865,351 3,500,591 3,086,215 2,767,527 2,604,333 2,367,658 1,589,831 1,580,656 1,485,267 1,342,491 1,278,088 1,221,875 1,169,432 New Orders per Minute (NOPM) Rackspace OnMetal I/O v2 13,881 52,088 59,104 53,553 47,157 42,304 39,819 36,236 Amazon Elastic Compute Cloud (Amazon EC2) 5,413 24,330 24,139 22,714 20,527 19,547 18,700 17,935 4,000,000 Transactions per Minute 60,000 New Orders per Minute Transactions per Minute 3,500,000 3,000,000 2,500,000 2,000,000 1,500,000 1,000,000 500,000 New Orders per Minute 50,000 40,000 30,000 20,000 10,000 0 2 8 16 24 48 64 128 256 0 2 8 16 24 48 64 128 256 Virtual Users Rackspace OnMetal I/O v2 Virtual Users Amazon Elastic Compute Cloud (Amazon EC2 ) Figure 3. TPM and NOPM performance for increasingly large numbers of virtual users for Rackspace OnMetal I/O v2 (red) and Amazon Elastic Compute Cloud (Amazon EC2 ) (blue) Conclusion The results of these tests demonstrate the significant performance advantages of using an OnMetal I/O v2 cloud solution over a virtual deployment for OLTP or other high-i/o operations. The bare-metal cloud server is a good option for companies looking to achieve high performance without competition for shared resources, the overhead of a hypervisor, and the potential security risks involved with virtual hosted solutions. 5
Appendices Appendix A: Test Environment MySQL Host Rackspace OnMetal I/O v2 bare-metal cloud server OS Ubuntu Server 14.04 LTS Ubuntu Server 14.04 LTS CPU 2 x Intel Xeon processor E5-2660 v3 (10 cores at 2.60 GHz) Amazon Elastic Compute Cloud (Amazon EC2 ) Virtual Host Instance Type: i2.4xlarge 16 vcpus RAM 128 GB 122 GB Boot Media MySQL Database Storage 2 x 240 GB hot-swappable solid-state drives (SSDs) configured in a RAID 1 mirror 2 x 1.6 TB Peripheral Component Interconnect Express (PCIe ) storage devices Provisioned I/O operations per second (IOPS) SSD 800 GB SSD (Amazon Elastic Block Store [Amazon EBS] optimized) Network 2 x 10 gigabit Ethernet (GbE) network-interface controllers (NICs) Virtual network interface Engine MySQL 5.6 MySQL 5.6 Tenancy Not applicable (N/A) Private cloud MySQL Load Driver Rackspace OnMetal I/O v2 bare-metal cloud server Amazon Elastic Compute Cloud (Amazon EC2 ) Virtual Host Instance Type: t2.medium OS Windows Server 2012 R2 Standard Windows Server 2012 R2 Standard vcpu 4 2 RAM 4 GB 4 GB Storage 80 GB (local SSD) 120 GB (general-purpose SSD, 360 IOPS; not Amazon EBS optimized) Tenancy N/A Private cloud Appendix B: Test Methodology SCHEMA GENERATION Use the HammerDB application to build individual MySQL databases on each host by using an identical method. InnoDB storage engine (default) Two warehouses (a simulated company selling products from two warehouses) WORKLOAD EXECUTION (ALL SCENARIOS) PRE-TEST/TRIAL RUNS Static-user timed test: Virtual users: 5 Ramp-up: 5 minutes Test time: 10 minutes Key and Think : False Test repetition: 3 Scaled-user timed test: Virtual users: 2, 8, 16, and 24 Ramp-up: 5 minutes Test time: 10 minutes Key and Think : False Test repetition: 3 TPC-C WORKLOADS Scaled-user timed test: Virtual users: 2, 8, 16, 24, 48, 64, 128, 256 Ramp-up: 30 minutes Test time: 30 minutes Key and Think : False Test repetition: 3 RESOURCE MONITORING MySQL Workbench Amazon CloudWatch 6
Appendix C: Test Results Rackspace OnMetal I/O v2 Run 1 Amazon Elastic Compute Cloud (Amazon EC2 ) Run 1 Time (UTC) Virtual Users TPM NOPM Time (UTC) Virtual Users TPM NOPM 21:44 2 914,208 13,951 21:11 2 355,105 5,413 22:01 8 3,009,811 46,110 21:28 8 1,666,382 25,507 22:18 16 3,344,261 51,130 21:45 16 1,580,656 24,139 22:35 24 3,134,291 47,968 22:02 24 1,487,239 22,729 22:52 48 2,928,323 44,710 22:19 48 1,330,539 20,350 23:09 64 2,760,501 42,172 22:37 64 1,265,696 19,368 23:27 128 2,604,333 39,819 22:54 128 1,216,037 18,604 23:45 256 2,422,346 37,044 23:12 256 1,163,293 17,805 Rackspace OnMetal I/O v2 Run 2 Amazon Elastic Compute Cloud (Amazon EC2 ) Run 2 Time (UTC) Virtual Users TPM NOPM Time (UTC) Virtual Users TPM NOPM 17:55 2 906,496 13,881 18:19 2 348,634 5,328 18:12 8 3,615,601 55,337 18:36 8 1,448,773 22,123 18:29 16 4,064,572 62,150 18:53 16 1,515,771 23,186 18:46 24 3,591,415 54,964 19:11 24 1,450,761 22,176 19:03 48 3,164,092 48,361 19:28 48 1,342,491 20,527 19:21 64 2,879,007 44,082 19:45 64 1,278,088 19,547 19:38 128 2,624,467 40,112 20:03 128 1,221,875 18,700 19:56 256 2,367,658 36,236 20:21 256 1,169,432 17,935 Rackspace OnMetal I/O v2 Run 3 Amazon Elastic Compute Cloud (Amazon EC2 ) Run 3 Time (UTC) Virtual Users TPM NOPM Time (UTC) Virtual Users TPM NOPM 16:56 2 819,540 12,570 15:54 2 405,806 6,193 17:13 8 3,401,811 52,088 16:11 8 1,589,831 24,330 17:30 16 3,865,351 59,104 16:28 16 1,624,340 24,850 17:47 24 3,500,591 53,553 16:45 24 1,485,267 22,714 18:05 48 3,086,215 47,157 17:03 48 1,368,914 20,957 18:22 64 2,767,527 42,304 17:20 64 1,290,983 19,744 18:40 128 2,454,407 37,538 17:38 128 1,238,079 18,937 18:58 256 2,280,844 34,867 17:56 256 1,178,524 18,049 Rackspace OnMetal I/O v2 Median of Three Runs Amazon Elastic Compute Cloud (Amazon EC2 ) Median of Three Runs Virtual Users TPM NOPM Virtual Users TPM NOPM 2 906,496 13,881 2 355,105 5,413 8 3,401,811 52,088 8 1,589,831 24,330 16 3,865,351 59,104 16 1,580,656 24,139 24 3,500,591 53,553 24 1,485,267 22,714 48 3,086,215 47,157 48 1,342,491 20,527 64 2,767,527 42,304 64 1,278,088 19,547 128 2,604,333 39,819 128 1,221,875 18,700 256 2,367,658 36,236 256 1,169,432 17,935 7
The analysis in this document was done by Prowess Consulting and commissioned by Rackspace US, Inc. The analysis was completed between February and April 2016. Results have been simulated and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Prowess, the Prowess logo, and SmartDeploy are trademarks of Prowess Consulting, LLC. Rackspace, Fanatical Support, and OnMetal are trademarks or registered trademarks of Rackspace US, Inc. in the United States and other countries. Copyright 2016 Prowess Consulting, LLC. All rights reserved. Other trademarks are the property of their respective owners. 8