Mellanox WinOF-2 Release Notes. Rev 2.10 Package v Driver v

Transcription

1 Mellanox WinOF-2 Release Notes Rev 2.10 Package v Driver v Mellanox Technologies

2 NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Mellanox Technologies 350 Oakmead Parkway Suite 100 Sunnyvale, CA U.S.A. Tel: (408) Fax: (408) Copyright Mellanox Technologies Ltd. All Rights Reserved. Mellanox, Mellanox logo, Connect-IB, ConnectX, CORE-Direct, GPUDirect, LinkX, Mellanox Multi-Host, Mellanox Socket Direct, UFM, and Virtual Protocol Interconnect are registered trademarks of Mellanox Technologies, Ltd. For the complete and most updated list of Mellanox trademarks, visit All other trademarks are property of their respective owners. Mellanox Technologies 2

3 Table of Contents Table of Contents List of Tables Document Revision History Chapter 1 Overview Mellanox WinOF-2 Package Contents Supported Operating System Versions Certifications Supported Network Adapter Cards and MFT Tools Supported Network Adapter Cards MFT Versions Chapter 2 Changes and New Features in Rev Chapter 3 Known Issues SR-IOV Support Limitations Chapter 4 Bug Fixes History Chapter 5 Change Log History Rev 2.10 Mellanox Technologies 3

4 List of Tables Table 1: Document Revision History Table 2: Supported Operating System Versions Table 3: Certifications Table 4: Supported Network Adapter Cards Table 5: Firmware Versions Table 6: MFT Versions Table 7: Changes and New Features in Rev Table 8: Known Issues Table 9: SR-IOV Support Limitations Table 10: Bug Fixes History Table 11: Change Log History Mellanox Technologies Rev 2.10

5 Document Revision History Table 1 - Document Revision History Release Date Description Rev 2.10 December 3, 2018 This is the initial version of WinOF-2 v2.10. This version includes new feature and bug fixes. For further information see Section 2, Changes and New Features in Rev 2.10, on page 10) and Section 4, Bug Fixes History, on page 16). Rev 2.10 Mellanox Technologies 5

6 1 Overview These are the release notes of Mellanox WinOF-2 Rev 2.10 Ethernet and InfiniBand drivers. Please note that WinOF-2 supports ConnectX-4 onwards adapter cards only. 1.1 Mellanox WinOF-2 Package Contents The Mellanox WinOF-2 Rev 2.10 for Windows package contains the following components: Diagnostic Tools Documentation Drivers Management Tools Performance Tools 6 Mellanox Technologies Rev 2.10

7 Overview 1.2 Supported Operating System Versions The following describes the supported operating systems and their roles in a virtualization environment. Table 2 - Supported Operating System Versions Supported Host OS Supported Guest OS Virtualization Mode: None Windows Server 2012 N/A Windows Server 2012 R2 N/A Windows Server 2016 N/A Windows Server 2019 N/A Windows 8.1 Client (64 bit only) N/A Windows 10 Client 1809 (64 bit only) N/A Virtualization Mode: VMQ Windows Server 2012 Any supported guest OS for Hyper-V Windows Server 2012 R2 Any supported guest OS for Hyper-V Windows Server 2016 Any supported guest OS for Hyper-V Windows Server 2019 Any supported guest OS for Hyper-V Virtualization Mode: SR-IOV (Ethernet only) Windows Server 2012 R2 Windows Server 2012 R2 Windows Server 2016 Windows Server 2019 Windows 10 Client 1809 (64 bit only) Windows Server 2016 Windows Server 2012 R2 Windows Server 2016 Windows Server 2019 Windows 10 Client 1809 (64 bit only) Inbox RHEL 7.3 LIS_ Inbox CentOS 7.3 LIS_ Inbox SLES default Inbox FreeBSD 10.4 Inbox Ubuntu Virtualization Mode: SR-IOV (Ethernet and InfiniBand) Windows Server 2016 Windows Server 2016 Windows Server 2019 Inbox CentOS/RHEL LIS_ Inbox SLES12 SP Inbox RHEL el7.x86_64 Inbox Ubuntu Rev 2.10 Mellanox Technologies 7

8 Table 2 - Supported Operating System Versions Supported Host OS Supported Guest OS Windows Server 2019 Windows Server 2016 Windows Server 2019 Inbox CentOS/RHEL LIS_ Inbox SLES12 SP Inbox RHEL el7.x86_64 Inbox Ubuntu Certifications The following describes the driver s certification status per operating system. Table 3 - Certifications Operating System Logo Certification SDDC Premium Certification Windows Client 8.1 Certified N/A Windows 10 Client 1809 Certified N/A Windows Server 2012 Certified N/A Windows Server 2012 R2 Certified N/A Windows Server 2016 Certified N/A Windows Server 2019 Certified N/A This section is updated in accordance with the certifications obtainment. 1.4 Supported Network Adapter Cards and MFT Tools Supported Network Adapter Cards Mellanox WinOF-2 Rev 2.10 supports the following Mellanox network adapter cards: Table 4 - Supported Network Adapter Cards NICs Supported Protocol Supported Link Speed ConnectX -4 Ethernet/InfiniBand 10, 25, 40, 50 and 100GbE QDR, FDR and EDR ConnectX -4 Lx Ethernet 10, 25, 40, and 50GbE ConnectX -5/Ex Ethernet/InfiniBand 10, 25, 40, 50 and 100GbE QDR, FDR and EDR BlueField SmartNIC Ethernet 25GbE 8 Mellanox Technologies Rev 2.10

9 Overview Firmware Versions Mellanox WinOF-2 Rev 2.10 provides the following firmware for Mellanox NICs: Table 5 - Firmware Versions NICs Recommended Firmware Rev. Additional Firmware Rev. Supported ConnectX a ConnectX-4 Lx ConnectX-5 / ConnectX-5 Ex a. For SR-IOV support, please refer to Table BlueField MFT Versions Mellanox WinOF-2 Rev 2.10 is compatible with the following MFT versions: Table 6 - MFT Versions Product Recommended Firmware Rev. Additional Firmware Rev. Supported MFT Rev 2.10 Mellanox Technologies 9

10 2 Changes and New Features in Rev 2.10 This package version is and the driver version is Table 7 - Changes and New Features in Rev 2.10 Adapter Cards Category Adapter Cards Description Debugability Debugability Dump PDDR Dump-Me-Now Description Added support for ConnectX-6 Ready adapter cards. For further information, please refer to the Mellanox Support. Updated the device description of the adapter cards to match their marketing name. Added debug counters to debug PHY. Added debug information to debug link state issues. Enabled dumping of the PDDR information into the ETL file every time the link state is reported to the OS. Now the PDDR information and diagnostics (pddr.txt file) will be part of DMN. Added the ability to collect Dump-Me-Now data on link down event. Dynamic VMMQ Added support for Dynamic VMMQ in Windows Server Mellanox WinOF-2 Port Diagnostics Counters Mellanox WinOF-2 VF Internal Traffic Counters Miniport Profiling Mechanism ND/NDK Registry Keys RoCE Sniffer Utility in IPoIB Added new set of counters: Mellanox WinOF-2 Port Diagnostics counters. This set contains physical layer statistical counters and exists for every adapter in PF. It is not supported in VF. For further information, refer to section Mellanox WinOF-2 Port Diagnostics in the User Manual. WinOF-2 v2.10 enables the user to expose the virtual network adapter's internal traffic counters in a hypervisor. Currently this capability is supported in Windows Ethernet virtual network adapters as of WinOF-2 v2.10 and Linux Ethernet virtual network adapters as of upstream kernel For further information, refer to section VF Internal Traffic Counters in the User Manual. Added miniport profiling mechanism to collect timing information on driver major flows. Modified the IPoIB ND port traffic counters (increased the counters size) to avoid any functionality issues. Enabled dynamic setting changes for some of the registry keys. For further information, please refer to the User Manual QoS Flow Control is now configured per adapter. To configure QoS Flow Control globally use the -Global flag. To disable restart after configuration, run -NoRestart. Note: NetQoSPolicy configuration is global. Changing NetQoSPolicy configuration for a specific adapter affects other adapters configuration. Added support for Sniffer Utility in IPoIB. 10 Mellanox Technologies Rev 2.10

11 Changes and New Features in Rev 2.10 Table 7 - Changes and New Features in Rev 2.10 Category Description SR-IOV VF Counters Enabled the host driver to show SR-IOV VF counters. Bug Fixes See Bug Fixes History on page 16. Rev 2.10 Mellanox Technologies 11

12 3 Known Issues The following table provides a list of known bugs and limitations in regards to this release of WinOF-2. For a list of old Know Issues, please see the WinOF-2 Archived Known Issues file (WinOF-2_Archived_Known_Issues.pdf). Table 8 - Known Issues (Sheet 1 of 4) Internal Ref Description: There are no event log messages on dynamic changes for EXT_QP_MAX_RETRY_PERIOD and EXT_QP_MAX_RETRY_LIMIT registry keys, although they are dynamic Workaround: N/A Keywords: Registry keys Detected in version: Description: Ethernet Sniffer utility is currently not functional in SR-IOV configuration on ConnectX-5 and above adapter cards. Workaround: N/A Keywords: Sniffer utility Detected in version: Description: Occasionally the driver updates the PKeys wrongly when it receives simultaneously 2 events or more from UFM/SM to change the PKeys. Issue Workaround: Restart the driver or the UFM/SM Keywords: PKeys, IPoIB Detected in version: Description: When working with SR-IOV in Windows Server 2019, the vnic that is working in SR-IOV mode status will be displayed as "Degraded (SR-IOV not operational)" although the SR-IOV VF is fully operational. The message can be safely ignored. Workaround: N/A Keywords: SR-IOV IB, Windows Server 2019 Detected in version: Description: When using the IPv6 protocol, a wrong IP presentation format will appear in the Information Tab. Workaround: N/A Keywords: IPv6 Detected in version: Description: iscsi boot over IPoIB is currently not supported. Workaround: N/A Keywords: iscsi Boot, IPoIB Detected in version: Mellanox Technologies Rev 2.10

13 Known Issues Table 8 - Known Issues (Sheet 2 of 4) Internal Ref Description: Occasionally the "RDMA Active Connections" counter cannot be reset after the connection is closed in some error flows thus it cannot display the active connections. Workaround: Run mlx5cmd -NdStat/-NdkStat to see active connections. Keywords: "RDMA Active Connections" counter Detected in version: Description: The RscIPv4 and RscIPv6 keys values are set to 0 for the host in Windows Server As the values for those keys are already written by the Inbox Driver in Windows Server 2019, they will not be changed when upgrading. Workaround: N/A Keywords: RscIPv4, RscIPv6, Windows Server 2019 Detected in version: Description: On servers with large number of VMs, (typically more than 40), after restarting the NIC on the host, VMs IPv6 global address is not retrieved back from the DHCP. Workaround: Restart the NIC inside the VM. Keywords: VMQ, SR-IOV Detected in version: Description: On servers with a large number of VMs (typically > 40) after a NIC restart on the host, VMs IPv6 global address cannot be retrieved from DHCP. Workaround: Restart Microsoft NIC inside the VM. Keywords: VM, IPv6 address, DHCP Detected in version: Description: Due to an OID timeout, the miniport reset is executed. Workaround: Increase the timeout value in such way that 2 * CheckForHangTOInSeconds > Max OID time. For further information, refer to section General Registry Keys in the User Manual. Keywords: Resiliency Description: The ModerationInterval will always be rounded down to its limit (depending on the cards capability), thus the ModerationCount will never solely control the interrupt moderation on the CQ. Workaround: N/A Keywords: CQ moderation Issue Rev 2.10 Mellanox Technologies 13

14 Table 8 - Known Issues (Sheet 3 of 4) Internal Ref Description: Multicast packets are passed via to the VM the Hyper-V (even in SR-IOV VMs). As such, the Hyper-V can decide to drop the packets based on its specific policy. Note: This issue is only related to FreeBSD OSes. Workaround: N/A Keywords: Hyper-V OS Description: Adding diagnostic counters to performance monitor might cause counters to get cleared every several seconds. Workaround: Change the time period between samples to more than 1 second. Keywords: Diagnostic Counters Description: When PXE boot is using Flexboot, the IPoIB interface is not receiving the reserved address from the DHCP using GUID reservation. Workaround: To obtain the reserved address, use a 6-byte MAC address instead of the 8-byte client ID. Keywords: PXE boot, IPoIB, Flexboot, DHCP Description: VF driver initialization fails in case of bad MSIX mapping when running Windows Server 2012 R2 Hypervisor with Windows Server 2016 VM with more than a single core CPU. As a result, performance desegregation might occur. Workaround: Run either with one CPU core, or run with different Operating Systems. Keywords: SR-IOV Description: The driver must be restarted in order to switch from RSS to NonRSS mode. Therefore, if a PowerShell command is used on a specific VM to an enabled/disabled VMMQ without restarting the driver, the RSS counters will keep increasing in Perfmon. Workaround: Restart the driver to switch to NonRSS mode. Keywords: RSS, NonRSS, VMMQ Description: In RoCE, the maximum MTU of WinOF-2 (4k) is greater than the maximum MTU of WinOF (2k). As a result, when working with MTU greater than 2k, WinOF and WinOF-2 cannot operate together. Workaround: N/A Keywords: RoCE, MTU Issue 14 Mellanox Technologies Rev 2.10

15 Known Issues Table 8 - Known Issues (Sheet 4 of 4) Internal Ref Description: In IPoIB SR-IOV setup, in the Hyper-V Manager, the address appears as "SR- IOV enabled" instead of "SR-IOV active". This does not influence any activity or functionality. Workaround: N/A Keywords: IPoIB SR-IOV setup, Hyper-V Description: In the "Network Connections" panel of Virtual Function (VF) in IPoIB SR-IOV setup, the Microsoft adapter may appear in addition to the Mellanox adapter. This does not influence any activity or functionality. Workaround: N/A Keywords: Network Connections, VF, IPoIB SR-IOV 3.1 SR-IOV Support Limitations The below table summarizes the SR-IOV working limitations, and the driver s expected behavior in unsupported configurations. Table 9 - SR-IOV Support Limitations WinOF-2 Version Earlier versions 1.50 and and onwards ConnectX-4 Firmware Ver. Up to Between 1x and 1x (IPoIB supported) 1x and onwards (IPoIB supported) InfiniBand Adapter Mode Ethernet SR-IOV On SR-IOV Off SR-IOV On/Off Driver will fail to load and show "Yellow Bang" in the device manager. Yellow Bang unsupported mode - disable SR-IOV via mlxconfig Issue No limitations For further information on how to enable/disable SR-IOV, please refer to the Single Root I/O Virtualization (SR-IOV) section in the User Manual. OK No limitations OK OK No limitations Rev 2.10 Mellanox Technologies 15

16 4 Bug Fixes History Table 10 lists the bugs fixed in this release.for a list of old Bug Fixes, please see WinOF-2 Bug Fixes file (WinOF-2_Archived_Bug_Fixes.pdf). Table 10 - Bug Fixes History (Sheet 1 of 9) Internal Ref Description: Occasionally, when the HW RSS is disabled, the incorrect RSS counter s value is increased for encapsulated traffic when in VMQ mode. Keywords: NVGRE, VXLAN, RSS Fixed in Release: Description: SR-IOV over BlueField devices is not supported when the Embedded CPU (EC) model is enabled. Keywords: SR-IOV, BlueField, ECPF Detected in version: 2.0 Fixed in Release: Description: Fixed an issue that replaced the mlx5cmd -ndstat and -ndkstat port column with the priority column. Keywords: NDK Detected in version: 2.0 Fixed in Release: Description: The Nd_write_lat utility gets stuck at the end of the operation on the server side. This behavior is not applicable to the client side. Issue Keywords: Nd_write_lat, server Side, Client Side Detected in version: 2.0 Fixed in Release: Description: Fixed an issue that prevented the mlx5cmd command ndkstat from showing the source QP state. Keywords: mlx5cmd Detected in version: 2.0 Fixed in Release: Description: Fixed an issue that caused the ETL session on Windows Server 2019 to fail. Keywords: ETL session, Windows Server 2019 Detected in version: 2.0 Fixed in Release: Mellanox Technologies Rev 2.10

17 Bug Fixes History Table 10 - Bug Fixes History (Sheet 2 of 9) Internal Ref Description: Fixed an issue in that cause mlx5cmd -Stat not to group adapters properly on Windows 2019 and display them as separate NICs. Keywords: mlx5cmd, Windows Server 2019 Detected in version: 2.0 Fixed in Release: Description: IPoIB ND port traffic counters are not functional. Keywords: IPoIB, ND, Counters, Perfmon Detected in version: 2.0 Fixed in Release: Description: ConnectX-5 Socket Direct card in IB mode is not supported. Keywords: ConnectX-5, Socket Direct, IB mode Detected in version: 2.0 Fixed in Release: Description: Fixed an issue that caused.pcap file to be corrupted when running the Sniffer with a too small buffer_size value. Keywords: Sniffer Detected in version: 2.0 Fixed in Release: Description: Fixed an issue that reported the driver as being stuck when heavy receive and send UDP multicast traffic was detected. In this case NDIS triggered the reset process on the adapter card. Keywords: NDIS reset, UDP multicast Fixed in Release: Description: Reduced the active connections counter of RDMA activity counter set when a connection encountered an error. Keywords: RDMA activity counter Fixed in Release: Description: Fixed an issue that caused NDIS reset when the MTU configuration was not the same on all hosts. Keywords: IPoIB Fixed in Release: 2.0 Issue Rev 2.10 Mellanox Technologies 17

18 Table 10 - Bug Fixes History (Sheet 3 of 9) Internal Ref Description: Fixed wrong Perf tuning printings after choosing the Restore default settings option. Keywords: Perf Tuning Fixed in Release: Description: Fixed an issue that required the driver to change the local admin traffic class count to 8. Consequently, allowed the firmware, when in willing mode, to accept remote traffic classes. Keywords: DCBX Fixed in Release: Description: Updated the internal table update method. Now the internal table is updated only when a registry key does not match its default value. Keywords: DSCP Fixed in Release: Description: Improved the forwarding mode mechanism in case of internal queue packets. Now the mechanism enables resending multiple packets per transaction instead of one by one. Keywords: RFC2544 Fixed in Release: Description: Fixed an issue that caused the driver to load with a yellow bang when working with machines with more than 256 cores / Keywords: Driver load, 256 cores Fixed in Release: 2.0 Description: Fixed an issue where the RoCE connection failed as a result of an incorrect GID when the Universal/Local (U/L) bit in the MAC was set to 1. Keywords: RoCE Fixed in Release: 2.0 Issue 18 Mellanox Technologies Rev 2.10

19 Bug Fixes History Table 10 - Bug Fixes History (Sheet 4 of 9) Internal Ref Description: Removed the ability to create mst dump files from mlx5cmd. The command -mstdump is now deprecated and dump files are created using the Dump- Me-Now -dmn command which is enabled by default. Additionally, to the -dmn command was added the -nomstdump flag that will use the - dmn command without mstdump. Keywords: mst dump Fixed in Release: Description: When in IPoIB mode, Rss counters are shown in Perfmon with zero values although they are supported only in ETH mode. Keywords: Rss counters, IPoIB, ETH Fixed in Release: Description: Improved throughput ratio in the forwarding scenario. Keywords: Performance Fixed in Release: Description: Modified the link speeds format shown when using the PDDR utility and running the mlx5cmd -dbg -pddrinfo command. Keywords: PDDR utility, link speed format Fixed in Release: Description: Fixed the Check-For-Hang mechanism report to present the correct values. Keywords: Check-For-Hang mechanism report Fixed in Release: Description: The "NDISTest [2 Machine] - Stats" HLK test may fail due to inaccurate statistics caused by the counters caching mechanism in the driver. Keywords: HLK test, statistics, caching mechanism Description: Decreased the maximum value of RemoteCMResponseTimeout to 25 to prevent any unexpected cases of the machine getting stuck. Keywords: CM, Communication Manager, ND, NS, connection Issue Rev 2.10 Mellanox Technologies 19

20 Table 10 - Bug Fixes History (Sheet 5 of 9) Internal Ref Description: Updating the non-default vport's VM network interface RSS parameter during heavy traffic in vrss mode could cause the host and non-default vports network interfaces to reset, and a momentary disruption in network connectivity. Keywords: RSS parameters, vrss, vports Description: Updated a wrong value description received when dumping the interface VlanID. The new output is VlanID instead of VlanId. Keywords: VlanID Description: Fixed an issue that caused memory leak due to a race between the driver disabling and new RDMA connection creation. Keywords: Memory leak, ND, NDK Description: Fixed an issue when using ConnectX-4 Lx adapter cards that caused packets to drop when the send packet is requested with a VLAN tag placed directly in the packet instead of putting in it in the OOB data. Keywords: VLAN tag, Send packets, ConnectX-4 Lx Description: Fixed an issue that caused the driver to fail initialize when adding Wake-on-Lan support dynamically. Keywords: ISCSI boot, PXE Boot, Ethernet Detected in version: Description: Modified the amount of requested MSI-X vectors dependency in the VM when in SR-IOV mode. Now the amount of requested MSI-X vectors depends on RSS configuration only and is no longer limited by the inf file. Keywords: MSI-X vectors Description: Fixed an incorrect device description in the Device Manager menu. Keywords: Device Manager, device name Detected in version: 1.60 Issue 20 Mellanox Technologies Rev 2.10

21 Bug Fixes History Table 10 - Bug Fixes History (Sheet 6 of 9) Internal Ref Description: Fixed an issue that caused the firmware upgrade process to fail when upgrading the driver to a newer version. Keywords: Driver upgrade/install, firmware upgrade Description: Fixed an issue that caused the system to ignore the DeviceRxStallWatermark configuration. Keywords: DeviceRxStallWatermark Description: Fixed an issue that caused a miniport reset false alarm due to a command execution starvation. Keywords: Miniport reset, command starvation Description: Fixed an issue that caused the nd_read_bw utility to hang when running on duration mode. Keywords: nd_read_bw utility, duration mode Description: Fixed an issue that caused the device to report more than the supported number of schedule queue. Keywords: Virtualization Description: Fixed an issue that prevented the network adapter IP configuration from being reset after uninstalling the driver. Keywords: Uninstallation, network adapter configurations, Windows Server 2016, Windows 10 Detected in version: Description: A system crash can occur when working with WinOF-2 and iscsi boot with IPoIB when using Windows 2016 Server. Keywords: iscsi boot, IPoIB, Windows 2016 Server, system crash Detected in version: 1.70 Issue Rev 2.10 Mellanox Technologies 21

22 Table 10 - Bug Fixes History (Sheet 7 of 9) Internal Ref Description: The description of the Send queues priority counter on the performance monitor is incorrect. The correct description can be found in Mellanox WinOF-2 Diagnostics Counters section in the User Manual. Keywords: Counters, diagnostics Description: In some scenarios, since the default LRO/RSC coalescing value is 4K, 100GbE bandwidth cannot be reached. Keywords: LRO/RSC registry key Description: When the VMQ mode is enabled in a virtual switch, the RSS-enabled network adapter supports only 60 VMNICs, in spite of the 124 advertised. Thus, only 60 VMQs for virtual NICs are available and HLK VMQ tests fail. Keywords: VMQ, RSS, VMNICs, HLK Description: Setting the *TransmitBuffers parameter to a value that is not a power of 2 causes a BSOD while sending packets. Keywords: BSOD, *TransmitBuffers Description: Removed from the Windows INF file all the device IDs that are not in use. Keywords: Windows INF file, device IDs Discovered in Release: 1.80 Fixed in Release: Description: Fixed a false alarm that indicated that a send queue was stuck and as a result a miniport reset was performed. Keywords: False alarm, send queue, miniport reset Discovered in Release: 1.80 Fixed in Release: Description: Fixed a rare case that caused a BSOD when switched from a single RQ to RSS. Keywords: BSOD, RQ, RSS Discovered in Release: 1.80 Fixed in Release: Issue 22 Mellanox Technologies Rev 2.10

23 Bug Fixes History Table 10 - Bug Fixes History (Sheet 8 of 9) Internal Ref Description: Fixed certification issues that occurred when running ConnectX-4 Lx at 25Gb/s link speed. Keywords: HLK, certification Discovered in Release: 1.50 Fixed in Release: Description: Fixed an issue that caused the firmware to get stuck during VM migration in SR-IOV mode or upon PF driver restart. Keywords: VM, SR-IOV, PF driver restart Discovered in Release: 1.80 Fixed in Release: Description: Fixed an issue that caused packets to drop on the Windows host when running a Windows host with a Linux host in IPoIB mode, and both of them were configured with the same MTU. Keywords: Windows host, Linux host, IPoIB, traffic drop, MTU Discovered in Release: 1.70 Fixed in Release: Description: Fixed an issue that could cause a crash when the PF miniport driver did not wait for all previously indicated receive packets to return to the miniport from the VPort. Keywords: Virtualization, VmNic Discovered in Release: 1.70 Fixed in Release: Description: Fixed an issue where the ND provider threw an exception when an invalid parameter was used. Keywords: ND RDMA Discovered in Release: 1.80 Fixed in Release: Description: Fixed an issue where RSS counters were incorrectly incremented for encapsulated traffic. Keywords: RSS counters Discovered in Release:1.80 Fixed in Release: Description: Fixed an issue in the getconnectiondata method, where the pcbprivatedata did not return the correct value in case the pprivatedata buffer value was lower than required. Keywords: ND Discovered in Release: 1.80 Fixed in Release: 1.80 Issue Rev 2.10 Mellanox Technologies 23

24 Table 10 - Bug Fixes History (Sheet 9 of 9) Internal Ref Description: Fixed an issue that caused an installation failure when running in modify mode with the ADDLOCAL=All parameter. Keywords: Installation Discovered in Release: 1.70 Fixed in Release: Description: Removed an incorrect event log message in VMQ NIC switch mode. Keywords: Keywords: Event log Discovered in Release: 1.70 Fixed in Release: Description: Fixed an high stress issue that caused uncontrolled memory consumption, sporadic miniport device resets and missing send completions that could result in TX Q stuck events. Keywords: Keywords: High stress, memory consumption, miniport device resets, send completions Discovered in Release: 1.70 Fixed in Release: Description: Fixed an issue where the shortcut to the system snapshot tool was broken and pointed to the wrong folder. Keywords: System snapshot tool, broken shortcut, installation Discovered in Release:1.60 Fixed in Release: Description: Moved SR-IOV VF's live migration from Beta level to full support of all flows. Keywords: SR-IOV Discovered in Release: 1.70 Fixed in Release: Description: Removed a limitation that did not allow both of the following Ethertypes to be enabled at the same time: 0x Fibre Channel over Ethernet (FCoE) 0x FCoE Initialization Protocol Keywords: Ethertypes, 0x8906, 0x8914 Discovered in Release:1.60 Fixed in Release: Description: Updated the Cnp_dscp changeable values to 0-63, as supported by the firmware. Keywords: Cnp_dscp, changeable values Discovered in Release:1.70 Fixed in Release: 1.80 Issue 24 Mellanox Technologies Rev 2.10

25 Change Log History 5 Change Log History Table 11 - Change Log History Feature/Change Description Rev Communication Manager (CM) Debugging Driver/Firmware Load Time Dump-Me-Now Dump-Me-Now Event Trace for Windows (ETW) Event Trace for Windows (ETW) InfiniBand Performance Resiliency Added to the CM the ability to check if incoming connections are not already from a connected QP, source. Added firmware trace (itrace) to other debug information dumped and logged in the Event Log file to enable easier analysis of issues. Reduced the driver s and the firmware s load and unload time. Added the following files to the DMN directory: 2 mst files Registry files Now the mstdump files directory in the same as the DMN directory. Added the option to print the driver s current configuration into the DMN logs. Added the option to log errors occurring in the memory address translation and protection mechanism to the event trace log file. Added to ETW the information below about NB and NBL in case of a CQE error: NBL and NB pointers QueueID (vportid) DMA address Packet header Partition Keys (PKeys) are used to partition IPoIB communication by mapping a non-default full-membership PKey to index 0, and mapping the default PKey to an index other than zero. Note, this feature requires SM version and above For further information, see section Default and non-default PKeys in the User Manual. Added a new Interrupt Moderation mode. The new mode enables the user to change the interrupt moderation dynamically while also decreasing latency and CPU usage. For further information, see RxIntModerationProfile and TXIntModerationProfile registry keys in the User Manual Shutting Down RDMA QPs with Excessive Retransmissions is a mechanism used to detect excessive retransmissions for an RC connection, and to close the connection in response to it. If the number of retransmissions due to a Local Ack Timeout, NAK-Sequence Error, or Implied NAK, during a specified period, exceeds the specified threshold, the QP will be handled as if the IB spec defined Retry Count was exceeded. For further information, see section Shutting Down RDMA QPs with Excessive Retransmissions in the User Manual. Rev 2.10 Mellanox Technologies 25

26 Table 11 - Change Log History Feature/Change Description RoCE Supported Devices Tools Tools Tools Provides a quick RoCE configuration method using the mlx5cmd tool. It enables the user to set different QoS RoCE configuration without any prerequirements. For further information, see section Quick RoCE Configuration (One-Click RoCE) in the User Manual. Added support for BlueField SmartNIC (Beta level). DriverVersion utility is now part of the mlx5cmd tool, and is used to display the VF's driver version. Note: For VMs with driver version 1.90 and prior, the printed data about the Windows OS might not be correct. For further information, see section DriverVersion Utility in the User Manual. NDKStat utility is now part of the mlx5cmd tool, and is used to enumerate open NDK connections. The connections can be filtered by the adapter s IP or Process ID. For further information, see section NdkStat Utility in the User Manual. Updated the name property of the mlx5cmd -Stat output to match the Get- NetAdapter PowerShell command output, and added a new property "interface_description", to show the Interface Description of the adapter card. Before: name=mellanox ConnectX-4 Adapter #4 After: name=ethernet 3 interface_description=mellanox ConnectX-4 Adapter #4 Virtualization Increased the maximum supported number of VFs to up to 100 per port. Bug Fixes See Bug Fixes History on page 16. Rev Driver Default Settings Update Dump Me Now (DMN) WinOF-2 driver v loads with the wrong default settings. This issue is fixed in WinOF-2 v In case you have installed WinOF-2 v driver, to fix the above issue the following can be applied: Install WinOF-2 v If you do not want to install the new package, revert to the correct settings: 1. Delete the following registry key: HKLM\SYSTEM\CurrentControlSet\Services\mlx5\msAzure- Setup 2. Restart the driver for changes to take effect. Added support for DMN to generate dumps and traces for: General information QP specifics SR-IOV 26 Mellanox Technologies Rev 2.10

27 Change Log History Table 11 - Change Log History Feature/Change Description Event Log Messages Datapath Statistics Mellanox WinOF-2 Diagnostic Counters Mellanox WinOF-2 VF Diagnostic Counters Port Diagnostic Database Register (PDDR) DSCP Added the following new event log message in case firmware events are reported: Transmit engine hang Receive hang Note: The resiliency flow is disabled by default for WinOF-2 v1.90 thus the driver will ignore these events when reported. Added Mellanox WinOF-2 Receive and Transmit Datapath counters. For further information, see sections Mellanox WinOF-2 Receive Datapath and Mellanox WinOF-2 Transmit Datapath in the User Manual. Added the following Mellanox WinOF-2 Diagnostic Counters: Dropless Mode Entries Dropless Mode Exits Transmission Engine Hang Events MTT entries used for QP MTT entries used for CQ MTT entries used for EQ MTT entries used for MR CPU MEM-pages (4K) mapped by TPT for QP CPU MEM-pages (4K) mapped by TPT for CQ CPU MEM-pages (4K) mapped by TPT for EQ CPU MEM-pages (4K) mapped by TPT for MR Note: These counters are not counted per VF in the Hypervisor. For further information, see section Mellanox WinOF-2 Diagnostic Counters in the User Manual Added the following Mellanox WinOF-2 VF Diagnostic Counters: Async EQ Overrun Completion EQ Overrun Current Queues Under Processor Handle Total Queues Under Processor Handle Packets Received dropped due to Steering Packets Received dropped due to VPort Down Packets Transmitted dropped due to VPort Down Invalid Commands For further information, see section Mellanox WinOF-2 VF Diagnostic Counters in the User Manual The tool provides troubleshooting and operational information that can assist in debugging physical layer link related issues. For further information, see section Port Diagnostic Database Register (PDDR) in the User Manual. Added DSCP counters per-priority. Bug fixes See Bug Fixes History on page 16. Rev Bug fixes See Bug Fixes History on page 16. Rev.1.80 SR-IOV Enabled Windows SR-IOV Enhanced eipoib (without Secure Connection). Rev 2.10 Mellanox Technologies 27

28 Table 11 - Change Log History Feature/Change Description Diagnostics RDMA Performance Installation Added 3 diagnostic counters: Send queues priority counter Current queues under processor handle counter Total queues under processor handle counter. Added the host s ability to control RDMA per VM when using Windows Server, version The host can turn RDMA on/off for a specific VM without affecting other VMs. Added NDK (Network Direct Kernel) v3 support. Enabled by default RDMA in the VF from the host, using the EnableguestRdma registry key. Removed the RssMaxProcNumber registry key from the registry, and the "RSS Maximum Processor Number" from the UI. Added the ability to upgrade the driver while ND application is running. Operating Systems Added support for Windows Server, version DPDK mlxtool SR-IOV ECN [Beta] Added support for DPDK. DPDK is a set of libraries and optimized NIC drivers for fast packet processing in user space. DPDK provides a framework and common API for high speed networking applications. DPDK runs over Linux of SR-IOV. For more information, see dpdk.org. Rev.1.70 Added the ability to query ND active connection status. Added the ability to load the driver on devices configured for SR-IOV and IPoIB, with no SR-IOV support. The default values for ECN configuration are now taken from the firmware. It is still possible to configure the ECN parameters from the driver. Virtualization Added support for Linux VF over Hyper-V (Windows PF). Quality of Service RDMA Event Log Performance General QoS classification of received packets can now consider the DSCP value in addition to the PCP value. Added RDMA transport diagnostic counters. Added an event log in case the server reaches a certain (configurable) high temperature. Added threaded DPC support. Rev.1.60 Added Beta support for ConnectX-5/ConnectX-5 Ex devices 28 Mellanox Technologies Rev 2.10

29 Change Log History Table 11 - Change Log History Ethernet Diagnostics Virtualization Security InfiniBand RoCEv2 Congestion Management (RCM) DCBX Feature/Change Receive Path Activity Monitoring Performance Debug-ability The feature below is applicable to all supported Operating System: Added Differentiated Services Code Point (DSCP), a mechanism that uses the 6-bit Differentiated Services Field (DS or DSCP field) in the IP header for packet classification. Added a resiliency mechanism that stops transmission of pauses from the NIC port if pauses are sent for a long period of time. Added the ability to handle packet bursts, while avoiding packet drops that may occur when a large amount of packets is sent in a short period of time. Added Head of Queue Lifetime Limit, a feature that enables the system to drop the packets that have been awaiting transmission for a long period of time, preventing the system from hanging. Added a new diagnostics counters set with information about RSS traffic spread per core, and new RDMA diagnostic counters. Added the ability to limit the number of memory pages in the host that are used for contexts of VF resources. Extended SR-IOV support for up to 96 VFs per port. Added an anti-spoofing mechanism for the host driver to configure filters on outgoing traffic. Rev.1.50 Added support for InfiniBand and IPoIB including EDR link speed. Switches running ECN will report congestion to the sources using CNP messages. The sources (Host), in turn, react by throttling down their injection rates, thus preventing congestion in the fabric. Added support for Data Center Bridging Exchange Protocol at GA level Monitors the devices status continuously tot when the receive pipeline is stalled for a period longer than a pre-configured timeout. Optimized MSI-X interrupt vectors usage to enable efficient work on machines with high core counts. Extended data collected by the system snapshot tool to include QoS configurations. Added a counter to detect how many times the link went down and up. Added counters for NIC PCIe errors Added low level device counters to enable performance analysis. Added counters for driver reset. Rev Description Rev 2.10 Mellanox Technologies 29

30 Table 11 - Change Log History Feature/Change Description Link Speed Added to mlx5cmd the capability to query supported link speeds according to the adapter card. Additionally, it enables the user to force set the range of speeds the adapter supports. The list of features below are applicable to Window Server 2016 OS: NDKPI Virtualization VXLAN Stateless Offload PacketDirect WPP Traces Extraction MTT Optimization Data Center Bridging Exchange (DCBX) Packet Sniffer Documentation Operating Systems Added support for NDKPI v2.0 interface. Added support for RoCE in SR-IOV VM. Added support for RoCE in virtualization mode in the hypervisor. Added support for VMMQ - RSS load-balancing offload in HW for non- SRIOV VMs. Note: RSS for SR-IOV VMs is already available in older versions. Added support for send/receive checksum, LSOv2 and VMQ hardware offloads on VXLAN encapsulated frames. Added support for PacketDirect Provider Interface (PDPI). Rev WinOF-2 Mellanox driver automatically dumps trace messages that can be used for debugging issues that have recently occurred on the machine. Hardware resource utilization improvement for RDMA applications that use contiguous memory buffers [Beta] DCBX is used by DCB devices to exchange configuration information with directly connected peers. DCBX uses Link Layer Discovery Protocol (LLDP) to exchange parameters between two link peers. Added to mlx5cmd a command that allows sniffing packets as seen by the device and creating a.pcap file. Rev Release Notes and User Manual documents were removed from the package. A new README file which includes basic installation instructions, summary of main features and requirements has replaced them. Added support for Windows 8.1 Client. RDMA Changed the default value of RoCE mode from v1 to v2. SR-IOV Tools Added the option to configure the RoCE version per adapter. Changed UDP source port calculation according to IB spec. SR-IOV is at GA level. Added support for SR-IOV mode for Windows VM over Linux KVM hypervisor. Added the option to view information of current adapter configuration in the snapshot tool. Added mstdump utility to reflect the state of a certain adapter by producing several types of events. 30 Mellanox Technologies Rev 2.10

31 Change Log History Table 11 - Change Log History Feature/Change Congestion Control QoS Performance Utilities Wake On LAN Cables VF Counters Virtualization QoS NVGRE Ethernet Network Installation Diagnostics RDMA Ethernet UI (At beta level) Added support for Explicit Congestion Notification (ECN) to avoid congestion hot spots and optimize the throughput of the fabric. Priority Flow Control (PFC) is at GA level. Enhanced Transmission Selection (ETS) is at GA level. Improved RDMA latency, RoCE with high scale and cache line alignments. Added support for mlx5cmd utility, which is a generic utility that contains the trace, information and performance tuning utilities. Added the option for a network admin to remotely power on a system or wake it up from sleep mode. Added a notification pop-up upon bad cable state in the event viewer. Added new traffic counters per Virtual Function (VF). Rev Single Root I/O Virtualization (SR-IOV) is a technology that allows a physical PCIe device to present itself multiple times through the PCIe bus. Note: SR-IOV is currently at Beta level and is disabled by default in both the driver and the firmware. In order to enable it, please refer to WinOF-2 User Manual at Added support for configuring the global pause mode on the two sides (Rx and Tx). This feature is enabled by default. Rev Network Virtualization using Generic Routing Encapsulation (NVGRE) is a network virtualization technology that attempts to alleviate the scalability problems associated with large cloud computing deployment. Enhanced Transmission Selection (ETS) provides a common management framework for assignment of bandwidth to frame priorities. Added support for downgrade. Note that Rev 1.20 is the older version to which downgrade is possible. Added the mlxstat tool to WinOF-2 package. mlxstat displays information of Mellanox NIC attributes. Rev (Beta) Added support for RoCE through NDSPI and NDKPI Added QoS support for both global and priority-based flow controls Added support for VMQ Description Added support for Receive Side Coalescing (RSC) Added the following proprietary counters as part of the Performance Monitor: Mellanox WinOF-2 Traffic Counters Mellanox WinOF-2 QoS Counters Made some fixes in the Device Manager Properties Information tab Rev 2.10 Mellanox Technologies 31