IBM Emulex 16Gb Fibre Channel HBA Evaluation Evaluation report prepared under contract with Emulex Executive Summary The computing industry is experiencing an increasing demand for storage performance and bandwidth due to increases in virtual machine density, increasing demands for application performance and continual data growth. Fibre Channel storage area networks (SANs) carry the bulk of storage traffic in the enterprise data center and are beginning to feel the stresses of these increased demands. In many cases, enterprises are currently constrained by the available bandwidth between the servers and storage, or foresee a constraint as they observe their growing data consumption patterns. The IBM Emulex 16Gb Fibre Channel (16GFC) host bus adapter (HBA) addresses these increasing demands on storage performance by providing double the bandwidth of previous generation Fibre Channel HBAs. Demartek deployed an IBM System x3650 M4 server with the IBM Emulex 16GFC HBA (81Y1662) and connected this server to an all-flash storage array with four 8GFC host ports in the Demartek lab in Colorado. We ran a read-intensive data warehousing workload to determine if this type of workload could take advantage of the increased bandwidth and performance that 16GFC provides. We repeated the database workload test with previous generation IBM Emulex 8GFC HBA and compared the results. Key Findings We found that for this database workload, the 16GFC HBA exceeded the performance of the 8GFC HBA and provided the additional bandwidth needed by the database workload allowing the job to be completed in less time. The 16GFC HBA: Completed the real database workload 33% faster than the 8GFC HBA Allows for instantaneous doubling of throughput when needed, providing peak workload headroom Incorporates a simple, plug-n-play performance upgrade for the replacement of older 4GFC and 8GFC HBAs
Page 2 of 8 The Need for More Bandwidth Today s data centers face a variety of challenges brought by seemingly insatiable demands on server and storage infrastructure. At the same time, new technologies are being introduced that offer both challenges and possible solutions to meet these growing challenges. VM Density When Demartek presents to users about next-generation storage networking technologies at various industry events, we usually ask the audience of primarily technical users and first-line managers a few questions about their environments. Among the responses are that virtual machine (VM) density has been increasing over the last few years, with higher numbers of guest operating systems running on one physical server than in the past. We expect this trend to continue. 8GFC Saturation SSD During the year 2012, when we asked the end-users in our audiences about saturation of Fibre Channel links, we consistently heard from a few users who indicated that they had saturated their 8GFC links and needed something faster. The applications consistently identified as needing this higher bandwidth are database applications, regardless of the brand of database. These include single database instances running on physical hardware, multiple database instances running on physical hardware and multiple database instances running in VMs. These users are generally looking for something compatible with their existing infrastructure but that provides higher bandwidth to meet their growing demands. Solid State Disk (SSD) technology is another driver of bandwidth growth. Although relatively early in the deployment cycles, we have found that those who deploy any form of SSD technology in the enterprise experienced significant storage performance improvements. Many of these SSD deployments are in SAN environments, which drive up storage networking bandwidth consumption. Based on comments from users and many of the tests we have performed in our own lab, we concluded that SSD technology and faster storage networking technology such as 16GFC are well suited for each other. New IBM Servers In March 2012, IBM introduced its M4 servers that support the newest Intel Xeon E5-2600 processors ( Romley ) and PCI Express (PCIe) 3.0. In May 2012, IBM extended this server family with the introduction of servers with the Intel Xeon E5-4600 processors. These servers provide not only higher numbers of cores and performance improvements in processor power but also provide significant increases in I/O throughput. PCIe 3.0 doubles the maximum possible I/O rates and processors that support PCIe 3.0 support approximately double the number of PCIe lanes available to each processor. As a result, the total I/O bandwidth available in one of these new servers is approximately quadruple that of the previous generation of servers. Windows Server 2012 and Hyper-V Windows Server 2012 with Hyper-V was recently released, and it also addresses growing demands on computing infrastructure. Windows Server 2012 Hyper-V supports 320 logical processors and 4
Page 3 of 8 TB of physical memory. It supports 64 virtual processors, along with 1 TB of memory per VM. This enables virtualization environments not previously possible. When coupled with today s newer server hardware environments and new technologies such as 16GFC, much heavier workloads can be supported. A new feature for Windows Server 2012 Hyper-V is the support for virtual Fibre Channel, also known as Synthetic FC. This allows guest VMs to connect directly to Fibre Channel storage LUNs, allowing guests to take advantage of existing Fibre Channel infrastructure. This includes the ability for guest operating systems to be clustered over Fibre Channel. In order to take advantage of this feature, newer Fibre Channel HBAs that support virtual Fibre Channel are required. The IBM Emulex 16GFC HBA supports this feature and provides up to four virtual Fibre Channel ports per VM. Also required for virtual Fibre Channel is NPIV in the switch and HBA, which the IBM Emulex Fibre Channel HBA supports. Hyper-V in Windows Server 2012 supports the use of multipath I/O (MPIO) and virtual SANs, both of which are also supported by the IBM Emulex 16GFC HBA. Bandwidth Growth Summary When using storage intensive applications like backup/restore, database transactions, virtualization and rich media, there is clearly a need for higher storage networking bandwidth and performance. The improved I/O performance of 16GFC enables faster storage and retrieval of data. For those enterprises that don t believe that they need this higher performance yet, now is the time to start planning for these eventualities. When we discuss storage networks with enterprise users, we find that Fibre Channel is still the dominant storage interface in large-scale data centers, and is expected to remain dominant as a SAN interface for the foreseeable future.
Page 4 of 8 IBM Emulex Series Technology advances including the IBM Emulex series of 16GFC HBAs can help address these growing constraints on current server and storage infrastructure. Whether the environment includes mission critical stand-alone database workloads or increasing numbers of VMs running on a single server, IBM Emulex 16GFC HBAs enable higher workloads and more applications and VMs to run on a single server and port, resulting in reduced cabling and higher return on IT investment. The IBM Emulex series adapters provide several features designed for supporting enterprise I/O workloads: Twice the performance of 8GFC adapters Backward compatible with 4GFC and 8GFC infrastructure Support for Windows Server 2008 and 2012 with and without Hyper-V, VMware ESX and ESXi, Red Hat Linux, SUSE Linux Enterprise Server (SLES) In-box drivers for Windows Server 2012 and VMware vsphere 5.1 N_Port ID Virtualization (NPIV) support standard An IBM-branded solution which has undergone extensive IBM interoperability testing for connecting System x servers into storage and networking environments
Page 5 of 8 Test Description and Environment Demartek ran a read-intensive data warehouse workload in the Demartek lab in Colorado with the configuration shown below. This test was run with the 16GFC HBA running at 16Gb/sec. The test was repeated with the same HBA running at 8 Gb/sec and then repeated with the IBM Emulex 8GFC HBA. Server IBM x3650 M4 2x Intel Xeon E5-2690, 2.9GHz, 16 total cores, 32 logical processors 32GB RAM Host firmware 1.10, 6/20/2012 Boot drive: IBM 500GB 10K RPM SAS HDD Microsoft Windows Server 2012 Fibre Channel HBA IBM Emulex 16GFC dual-port HBA (81Y1662) IBM Emulex 8GFC dual-port HBA (42D0494) Fibre Channel Switch IBM SAN48B-5 16GFC Switch Storage Array Nimbus Data S-Class, 4x 8GFC host ports 24x 100GB 6Gb SAS SSD, configured as RAID0
Page 6 of 8 Data Warehouse Workload and Performance Results Database Workload This read-intensive database workload consists of a suite of business oriented ad-hoc queries and concurrent data modifications. The queries and the data populating the database have been chosen to have broad industry-wide relevance. This benchmark illustrates decision support systems that examine large volumes of data, execute queries with a high degree of complexity, and give answers to critical business questions. This particular workload consists of 22 different queries, each exercising a different area of the database. The workload runs the same queries in the same order for each run, so a lower elapsed time for each query indicates a faster system. Real vs. Synthetic Workload This workload uses a real database (Microsoft SQL Server) with database tables, indexes, etc., and performs actual database transactions. As a result, the I/O rate varies as the workload progresses because the database performs operations that consume varying amounts of CPU and memory resources, in addition to I/O resources. These results more closely resemble a real customer environment. This is unlike a synthetic benchmark that performs the same I/O operations repeatedly resulting in relatively steady I/O rates which although potentially faster, do not resemble real customer environments. Hardware and Software Specifications The specifications for the software used for this test are listed below. The RAM allocated to the database application was limited to 4GB in order to force the I/O through the Fibre Channel HBA and out to the storage system, with minimal host memory caching. Database Windows Server 2012 Microsoft SQL Server 2008 R2 RAM allocated to SQL Server: 4GB Database size: 30GB Total size of all database files: 54GB The server was rebooted between runs to clear any host memory caching. Bandwidth Results For this set of tests, we used a single host connection to the SAN. This allowed us to make a simple comparison of the 16GFC adapter with the previous generation 8GFC adapter. This data warehouse workload was able to achieve more than 8 Gb/sec of bandwidth for some of the queries, and in some cases, nearly line-rate with the 16GFC adapter. For those queries that show a flat top on the graph below using the 8GFC adapter, this indicates that more performance is available, but the 8GFC adapter is throttling the performance. Also note that when running this test with the 16GFC adapter, the time to complete the run was 67% of the time required by the 8GFC adapter, or 33% faster.
Seconds 1 20 39 58 77 96 115 134 153 172 191 210 229 248 267 286 305 324 343 362 381 400 419 438 457 476 495 514 533 552 571 MB/s Demartek Page 7 of 8 1800 1600 1400 1200 1000 800 600 400 200 0 MB/s per Server and Adapter 16 Gb Workload Completed 33% faster than 8 Gb Flat tops indicate throttling by 8GFC adapter Workload Duration (seconds) IBM Emulex 16Gb IBM Emulex 16Gb @ 8Gb IBM Emulex 8Gb Time to Completion All Queries The following chart shows the aggregate time to complete each of the 22 queries, with each of the adapters. Tests were run with the IBM Emulex 16GFC HBA and then repeated with the IBM Emulex 8GFC HBA. For this workload, on average, the 16GFC adapter completed its workload in 67% of the time required by the 8GFC adapters, or 33% faster. 700 600 500 Full Query Sets Time to Execute (Single FC HBA port: 16 Gb vs. 8 Gb) 16GFC adapter completed real database workload 33% faster than the 8GFC adapter 400 300 200 100 385 538 575 0 IBM Emulex 16Gb IBM Emulex 16Gb @ 8Gb IBM Emulex 8Gb
Page 8 of 8 Summary and Conclusion With the availability of IBM Emulex 16GFC PCIe 3.0 Fibre Channel HBA, environments with growing performance requirements such as virtualization servers, database applications, SSDs, flash caching and more have an excellent choice in Fibre Channel HBA. These database workload test results using the IBM Emulex 16GFC HBA show approximately 33% better performance than the same server and storage using 8GFC HBAs. Testing revealed that 8GFC HBAs throttle the performance and cause the application to run longer than necessary. IBM Emulex 16GFC HBAs enable the doubling of throughput when needed, alleviating bottlenecks under peak workload scenarios. 16GFC provides the performance horsepower for both new environments and existing environments that demand higher performance than are available today with older technologies. For existing environments with 4GFC or 8GFC HBAs, installing IBM Emulex HBAs provides a simple plug-and-play performance upgrade. Emulex is a registered trademark of Emulex Corporation. IBM and System X are trademarks of IBM. Intel and Xeon are registered trademarks of Intel Corporation. Microsoft, Windows Server and Hyper-V are registered trademarks of Microsoft. VMware, ESX and ESXi are registered trademarks of VMware. Demartek is a trademark of Demartek, LLC. All other trademarks are the property of their respective owners.