Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide

Size: px

Start display at page:

Download "Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide"

Arron Joseph
5 years ago
Views:

1 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Publication # Revision: 1.00 Issue Date: June 2018

2 2018 Advanced Micro Devices, Inc. All rights reserved. The information contained herein is for informational purposes only, and is subject to change without notice. While every precaution has been taken in the preparation of this document, it may contain technical inaccuracies, omissions and typographical errors, and AMD is under no obligation to update or otherwise correct this information. Advanced Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the contents of this document, and assumes no liability of any kind, including the implied warranties of noninfringement, merchantability or fitness for particular purposes, with respect to the operation or use of AMD hardware, software or other products described herein. No license, including implied or arising by estoppel, to any intellectual property rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD s products are as set forth in a signed agreement between the parties or in AMD's Standard Terms and Conditions of Sale. Trademarks AMD, the AMD Arrow logo, EPYC, and combinations thereof, are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. Microsoft and Windows are registered trademarks of Microsoft Corporation. PCIe is a registered trademark of PCI-SIG corporation. Advanced Micro Devices

3 56288 Rev June 2018 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Revision History Date Revision Description June Initial public release. Revision History 3

4 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Rev June 2018 Contents Overview... 5 Initial Test Systems Preparation... 5 Download and Install the Mellanox Driver... 5 Initial Steps... 6 Determine the NUMA Node That the Mellanox Card is Installed On... 7 Find the Index Value of the Network Interface... 8 Adding Registry Values... 9 Changing Advanced Settings in the Driver Test Optimizations Conclusion

5 56288 Rev June 2018 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Overview This guide is focused on tuning Mellanox ConnectX-4 cards and Windows 2016 for high bandwidth runs. First, the latest Mellanox driver will be installed, and then the driver is tuned. Tuning the driver results in a significant performance improvement as we optimize the operating system. Initial Test Systems Preparation 1. Both systems should be configured as follows: a. Hardware i. AMD EPYC 7601 for best CPU performance. ii. One DIMM per channel using the highest supported memory configuration by your platform provider. iii. ConnectX-4 in a 16-bit PCIe slot running at PCIe Gen 3 (8GT/s) one of the steps below will show how to verify this iv. Both ConnectX-4s directly connected and configured with IP addresses. b. BIOS i. C-States disabled (see next step) c. Set the operating system power plan option found in the control panel to high performance. Download and Install the Mellanox Driver 1. Download the Mellanox driver from: You will need the latest WinOF-2 Driver version. 2. Double-click on the downloaded file and accept all the defaults. It may update the firmware if the ConnectX-4 card that is installed is behind on its firmware version. 3. After it completes reboot the machine. 5

Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide 56288 Rev. 1.00 June 2018 Initial Steps 1. Open Device Manager and select the Mellanox ConnectX-4 that you wish to tune.

6 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Rev June 2018 Initial Steps 1. Open Device Manager and select the Mellanox ConnectX-4 that you wish to tune. In this example it is the ConnectX-4 #3. 2. Right-click on the card, select Properties, then select the Information tab. 3. Make sure the card negotiated to PCIe 8.0 GT/s x16 and it negotiated to Gbps. 6

56288 Rev. 1.00 June 2018 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Determine the NUMA Node That the Mellanox Card is Installed On 1.

7 56288 Rev June 2018 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Determine the NUMA Node That the Mellanox Card is Installed On 1. Go to Device Manager and select the Mellanox ConnectX-4 that you wish to tune, in this example it will be the Mellanox ConnectX-4 Adapter #3: 2. Right click on the selected card, go to Properties, then select the Details tab. 3. From there click the Property drop down and press the letter N which will then display the NUMA node property. Document the NUMA node that the card is attached to. This example shows NUMA node 6. This number is dependent on the PCIe slot the ConnectX-4 is installed in. 7

8 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Rev June 2018 Find the Index Value of the Network Interface 1. Open Device Manager 2. Right click on the selected card, go to Properties and then select the Details tab. 3. Select the Driver key and document the value. 4. The driver key value in your system may be slightly different from the example shown. 8

56288 Rev. 1.00 June 2018 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Adding Registry Values 1. Adding these registry values is necessary with Mellanox v1.80 driver. The newer driver v1.

9 56288 Rev June 2018 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Adding Registry Values 1. Adding these registry values is necessary with Mellanox v1.80 driver. The newer driver v1.90 (and newer) automatically sets the registry values when installed 2. Add the registry values to the following key: HKLM\SYSTEM\CurrentControlSet\Control\Class\{4d36e972-e325-11ce-bfc be10318}\0006\ The part in bold text, 0006, will be specific to your machine. 3. Right-click on 0006 and then add the following new string values: RscMaxPacketSize RfdBufferSize RscTimeoutPeriodUsecs 64 9

10 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Rev June This example shows Select the folder which corresponds to driver key identified earlier. In our case, it was The example photo shows It will likely be different in your case. Changing Advanced Settings in the Driver 1. Open Device Manager and select the Mellanox ConnectX-4 that you wish to tune. In this example it is Mellanox ConnectX-4 Adapter #3. 10

11 56288 Rev June 2018 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide 2. Right-click on the selected card, go to Properties, then select the Advanced tab. 3. Change "Receive Buffers" from Advanced property page to

Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide 56288 Rev. 1.00 June 2018 4.

12 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Rev June Change "Receive Completion Method" to Polling in the Advanced property page. 12

13 56288 Rev June 2018 Microsoft Windows 2016 Mellanox 100GbE NIC Tuning Guide Test Optimizations On the server side, from a command prompt window enter start /node 2 /affinity 0xAAAA iperf -s from the folder in which iperf resides. This example shows Node 2. You should select the node that corresponds to the NUMA node identified above. On the TX side run start /Node 2 /Affinity 0xAAAA iperf -c <server ip> -P 8 -t <number of seconds>. Run this for at least three minutes as there is an important step to complete before iperf completes. Conclusion Following these procedures, AMD achieved transfer rates of up to 94.5 Gb/s in our laboratories using version 2.09 of IPERF. 13