Performance of Mellanox ConnectX Adapter on Multi-core Architectures Using InfiniBand. Abstract

Transcription

1 Performance of Mellanox ConnectX Adapter on Multi-core Architectures Using InfiniBand Abstract...1 Introduction...2 Overview of ConnectX Architecture...2 Performance Results...3 Acknowledgments...7 For more information...7 Abstract Multi-core platforms are becoming standard in compute cluster environments. The performance of the next generation host channel adapter (HCA) from Mellanox called ConnectX achieves more than six times better latency over the previous generation InfiniBand HCA when utilizing all cores in a quadcore dual-socket node. This performance improvement demonstrates that ConnectX is well suited for multi-core systems across all HP product lines.

2 Introduction Multi-core, where two or more computational engines (cores) are contained within a single processor, is the dominant architecture in compute cluster environments. Multi-core allows more computational processing power per server (when measured in GFLOP/S) however, it also can create bottlenecks in a parallel application running across multiple servers by placing additional load on the interconnect adapters. Assuming one process per core, a single-core dual-socket server would only create interconnect traffic for two processes, whereas the same dual-socket server with quad-cores would create interconnect traffic for eight processes. If processes are primarily communicated off host, one can easily see how the interconnect adapter could become a bottleneck. Early in 2007, the HP High Performance Computing (HPC) Division launched its Multi-core Optimization Program. The program s goal is to investigate and implement performance improvement techniques for HPC applications on HP servers that use multi-core processors. This paper looks at the multi-core performance of the next generation interconnect adapter from Mellanox called ConnectX an adapter that is part of the program. Overview of ConnectX Architecture ConnectX is the fourth generation server and storage adapter architecture from Mellanox Technologies. It supports eight lane PCI Express v1.1 and 2.0 interfaces in a dual port package. Either port can be configured as a 4x InfiniBand (IB) port or a 10 Gigabit Ethernet port, but this report only discusses IB performance. ConnectX implements a layered architecture to provide maximum flexibility and offers up to 16 million flow interfaces for traffic class assignment. Hardware-based forward and backward congestion notification mechanisms avoid congestion via flow control. Figure 1. The ConnectX HCA architecture. 2

3 ConnectX also uses the same silicon in adapters regardless of whether they are used in blades or rack-mounted servers providing uniform interconnect performance across product lines. Performance Results In order to isolate the impact of multi-core architecture on HCA performance, we used the Multiple Latency (multilat.c) benchmark from the Network-Based Computing Laboratory at Ohio State University. This benchmark was designed to evaluate aggregate performance between multiple pairs of processes. The processes execute a number of simultaneous pingpongs via matching MPI_Send/MPI_Recv pairs. We ran this benchmark on the following two configurations: 1) an HP BL460c dual-socket quad-core Clovertown system with Infinihost III Lx 4x DDR IB adapters 2) an HP BL460c dual-socket quad-core Clovertown system with ConnectX using 4x DDR IB. In both tests the blades were each in a single enclosure resulting in a single switch hop. Figure 2. Multi-core performance on Infinihost III Lx Figure 2 shows the measured latency on the Infinihost III Lx configuration when running simultaneous pingpongs between two nodes as the number of cores utilized per node increases. The plot clearly shows that although latency is less than 4 μsec when only using one core per node, performance degrades to over 12 μsec when using all cores in a node. 3

4 Figure 3. Multi-core performance on ConnectX In marked contrast to this behavior, Figure 3 shows the same benchmark results when using ConnectX. Note the scale for the latency axis is drastically reduced, and the measured latency remains well under 2 μsec regardless of the number of cores used. This shows that the architectural enhancements in ConnectX directly result in vastly improved multi-core performance. To show the improvement between the previous generation adapter and ConnectX, the eight core results are plotted using the same scale in Figure 4. It is clear ConnectX provides over six times improvement in short message latency when utilizing all cores in a quad-core dual-socket node. 4

5 Figure 4. Performance comparison of Infinihost III Lx and ConnectX using all eight cores in the blade. Although these results are quite dramatic, the question remains how this translates into real application performance on multi-core systems. Figure 5 contains results from the two popular ISV applications LS-DYNA and ABAQUS. The benchmark configuration consisted of 64 BL460c dualsocket Intel Xeon 5160 nodes. The benchmark was first run using Infinihost III Lx adapters. These adapters were then removed and replaced with ConnectX adapters. The benchmark cases were rerun on the exact same system allowing complete isolation of HCA effects. All benchmark cases were run utilizing all cores on a node. The LS-DYNA benchmark was LSTC LS-DYNA V This is the 3-vehicle model of 791,780 elements, simulating the crash of 3 vehicles for 150 msec, run in single precision floating point. The input data set is provided on the TOPCRUNCH public performance site at The ABAQUS benchmark was SIMULIA Abaqus Explicit V6.6 running three of the ISV standard performance benchmarks Data set E2 is a simplified 45,785-element model of a cell phone impacting a fixed rigid floor. Data set E3 consists of forming a sheet metal part by the deep drawing process with 34,540 elements. Data set E5 consists of a stiffened steel plate subjected to a high intensity blast load with 50,000 elements. According to Figure 5, all cases except Abaqus E5 show a significant speedup when moving to ConnectX, even though the benchmark platform was only dual-core. One expects further increases when running on a quad-core system. 5

6 Figure 5. Performance comparison of Infinihost III Lx and ConnectX for LS-DYNA and Abaqus 5000 Total Runtime (seconds) Infinihost III Lx ConnectX LS-DYNA 64 cores LS-DYNA 128 cores ABAQUS E2 ABAQUS E3 ABAQUS E5 6

7 Acknowledgments The idea for this project originated in HP s High Performance Computing Division. It is one of the results of HP s Multi-Core Optimization Program, which seeks ways to improve total application performance and per-core application performance on servers using multi-core processors. For more information ConnectX Architecture Brief, Mellanox Technologies. Shainer, Gilad: Connectivity in the Multi-Core Environment, HPC Forum, MVAPICH: MPI over InfiniBand and iwarp, Network-Based Computing Laboratory, Department of Computer Science and Engineering, Ohio State University Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. AMD and AMD Opteron are trademarks of Advanced Micro Devices, Inc. Intel and Xeon are registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Itanium is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and other countries. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Linux is a U.S. registered trademark of Linus Torvalds. 4AA1-6092ENW, November 2007