Mellanox WinOF VPI Release Notes Rev 3.2.0 www.mellanox.com
NOTE: THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT ( PRODUCT(S) ) AND ITS RELATED DOCUMENTATION ARE PRO- VIDED BY MELLANOX TECHNOLOGIES AS-IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUS- TOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CAN- NOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MER- CHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEM- PLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOT LIMITED TO, PAYMENT FOR PROCURE- MENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 350 Oakmead Parkway Suite 100 Sunnyvale, CA 94085 U.S.A. www.mellanox.com Tel: (408) 970-3400 Fax: (408) 970-3403, Ltd. Beit Mellanox PO Box 586 Yokneam 20692 Israel www.mellanox.com Tel: +972 (0)4 909 7200 ; +972 (0)74 723 7200 Fax: +972 (0)4 959 3245 Copyright 2012.. All rights reserved. Mellanox, Mellanox Logo, BridgeX, ConnectX, CORE-Direct, InfiniBridge, InfiniHost, InfiniScale, PhyX, SwitchX, Virtual Protocol Interconnect and Voltaire are registered trademarks of, Ltd. FabricIT, MLNX-OS, Unbreakable-Link, UFM and Unified Fabric Manager are trademarks of, Ltd. All other trademarks are property of their respective owners. Mellanox WinOF VPI Release Notes 2
Table of Contents Table of Contents..........................................................3 Chapter 1 Overview...................................................... 4 1.1 WinOF VPI Package Contents - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 4 1.2 Supported Operating Systems Versions - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 4 1.3 Supported Network Adapter Cards- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 5 Chapter 2 Changes and Major New Features................................. 6 2.1 New Features and Changes From Version 3.1.0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 6 2.2 New Features and Changes From Version 3.0.0 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 6 2.3 New Features and Changes From Version 2.1.3 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 6 2.4 New Features and Changes From Version 2.1.2 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 7 2.5 New Features and Changes From Version 2.1.1 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 8 Chapter 3 Beta Features.................................................. 10 Chapter 4 Known Issues.................................................. 11 4.1 General Known Issues - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 11 4.2 InfiniBand Drivers Known Issues - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 11 4.3 Ethernet Drivers Known Issues - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 11 4.4 RoCE Known Issues - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 12 3
Overview 1 Overview This is the release notes for Mellanox VPI Rev 3.2.0 InfiniBand and Ethernet drivers and it is part of the MLNX VPI release. The driver is intended for Mellanox adapter cards based on the ConnectX family that are identified on the PCI bus as one of the following PCI Device IDs (decimal): 25408, 25418, 25448, 25458, 26418, 26428, 26448, 26458 or 4099. This software package includes several software modules that can be used on a computer cluster configured as an InfiniBand subnet. Mellanox VPI is based on WinOF which was developed under Windows OpenFabrics, http://openfabrics.org. Mellanox VPI for Windows is presented as.exe packages, one for each supported operating systems. 1.1 WinOF VPI Package Contents The Mellanox WinOF VPI for Windows package contains the following components: Core and ULPs: IB HCA low-level drivers (mlx4) IB Access Layer (IBAL) Ethernet driver (ETH) Upper Layer Protocols (ULPs): IP over InfiniBand (IPoIB) NetworkDirect (ND) Utilities: OpenSM (OSM): InfiniBand Subnet Manager (beta level) Low level performance tools vstat - get the card status SdpConnect - SDP\WSD test IB Diagnostics tools SW Development Kit (SDK) Documentation 1.2 Supported Operating Systems Versions Supported Operating Systems and Service Packs: Windows 7 (x86, x64) Windows Server 2008 (x86, x64) Windows Server 2008 R2 (x64) 4
Mellanox WinOF VPI Release Notes Rev 3.2.0 1.3 Supported Network Adapter Cards Mellanox MLNX_WinOF_3.1.0_wlh_x64.exe supports the following Mellanox network adapter cards: ConnectX -2/ConnectX -2 EN /SDR/DDR/QDR ConnectX -3/ConnectX -3 EN /FDR/SDR/QDR We recommend upgrading ConnectX and ConnectX-2 adapter cards to firmware 2.9.1000 or higher to enable improved functionality while using this WinOF release. 5
Changes and Major New Features 2 Changes and Major New Features 2.1 New Features and Changes From Version 3.1.0 Ethernet Features Change the join/leave of Multicast groups in order to resolve Multicast packet drop when calling to Multicast Join or leave over Ethernet interface. During the join or leave multicast operation the user could encounter multicast packet drop for short period (30 milliseconds up to 2 seconds) General Features Fixed a blue screen that may occur when changing HighRateSteering registry value. 2.2 New Features and Changes From Version 3.0.0 Ethernet Features Optimized code performance Increased send parallelism Memory used in receive flow is now allocated with the same affinity of the handling processor for faster access Statistics parameters are now directly read from hardware instead of being calculated by software Added a performance UI to tune performance under various scenarios General Features Added a tool to tune performance under various scenarios part_man utility is currently not supported ibdiagnet utility is currently not supported 2.3 New Features and Changes From Version 2.1.3 Ethernet Features Added RoCE support Added support for 40GigE Ethernet Added support for Hyper-V with Virtual Machine Queue (VMQ) Added support for header data separation for Ethernet IPoIB Features IPoIB performance improvements Added RSS Support for IPoIB Modified the UI Full IPv6 support for IPoIB including DHCP Management Features Added support for OpenSM v3.3.11 (beta level) General Features 6
Mellanox WinOF VPI Release Notes Rev 3.2.0 Added Auto Sensing support Added ConnectX-3 support Added FDR/FDR 10 support Performance improvements for ConnectX-3 Added support for more than 64 processors Removed support for SDP/SRP/WSD Changed the installation package to.exe instead of MSI Removed the IBDiagnet tool from the Mellanox WinOF VPI v3.0.0 package 2.4 New Features and Changes From Version 2.1.2 VPI Features Added support for Windows 7 (32 and 64 bits) Added support for iscsi boot Mellanox drivers deletion (sys files) from system32\drivers directory upon uninstall Added support for Altiris booting Upgrade improvements when upgrading from 2.1.2 Fixed a bug related to link status displaying. A link was displayed as disconnected although it was connected Fixed a blue screen issue caused when the link was disconnected and the NIC configuration was IB and ETH MLNX_WinOf_VPI v3.0.0 uses OpenSM v3.3.6 InfiniBand Features Added support for software gateways Increased the system memory allocation from 256GB to 8 TB Added static IPv6 support Added support for IPv6 neighbor discovery Added support for receive methods such as: interrupts, polling and adaptive to improve network and/or system performance in certain configurations Changed the default size of receive queue to 1024 and the max size to 2048 Changed the minimum payload size to 600 bytes Changed ibnetdiscover tool to display the machine name instead of the port description Disabled WSD installation on windows 7 as it is not supported by the OS Fixed a bug that caused driver starting failure Fixed a bug in Linux related to windows interoperability Ethernet Features Added the Fault Tolerance failback option A new UI Fixed a bug resulting in a blue screen when the MCAT address was changed Fixed a bug in MAC address setting that caused setting same MAC to multiple teams 7
Changes and Major New Features 2.5 New Features and Changes From Version 2.1.1 InfiniBand Features Using OpenSM 3.3.3 Fixed OpenSM issues: Memory leak Command line parsing Fixed memory leak in Network Direct provider when querying an adapter Fixed blue screen issues caused when: the driver was disabling receiving illegal memory region address the NIC interface contains multiple IPs receiving DHCP discovery message the unmap memory was not initialized Fixed NMI BS issues that can happen on driver enabling or resetting Removed firewall disabling during MLNX_WinOF_VPI installation Added support to Repair and Upgrade while installing the package Added InfiniBand tools such as: ibroute.exe, ibtracert.exe, perfquery.exe, smpquery.exe, ibclearerrors.exe Added ibv_* performance tools Enabled IPoIB to work with Linux DHCP ibal: Set timeout and retry values for RMPP messages Improved scalability issues Fixed wrong calculation of rate in ibstat tool Added support for Dell GUIDs Added missing options to read_bw and read_lat tools Fixed issues that may have caused lost of interrupt Removed inadequate event log osmtest - created inventory file in case it does not exist when creating DB Enabled sending mad message in DCP level Perftests: Added a -d <guid> option to enable choosing the device to run on Changed the setup settings of the installation process to not disable firewall on installation, by default Fixed VPD read issues that disrupted the burned VPD Ethernet Features Added support to report the default link state when the link is connected to the ETH driver Fixed a blue screen issue caused when the driver was disabling Fixed issues that may have caused loss of interrupt Fixed VPD read issues that disrupted the burned VPD Fixed NMI BS issues that can happen on driver enabling or resetting Removed inadequate event log Removed firewall disabling during MLNX_WinOF_VPI installation 8
Mellanox WinOF VPI Release Notes Rev 3.2.0 Added support to Repair and Upgrade while installing the package Fixed RSS issues on Windows 2008 Disabled header data split by default Changed the number of RSS rings to 4 by default 9
Beta Features 3 Beta Features OpenSM is currently at beta level. We do not recommend using it in deployment setups. If OpenSM is required in your setup, please use Mellanox Managed switch or download the OpenSM from http://openfabrics.org. WinVerbs is currently at beta level. We recommend using ND over IBAL 10
Mellanox WinOF VPI Release Notes Rev 3.2.0 4 Known Issues 4.1 General Known Issues The following is a list of general limitations and known issues of the various components included in this Mellanox WinOF VPI release. On rare occasions, as a result of port configuration change (IB/ETH and enabling or disabling RoCE) the UI may get stuck for up to a few minutes. This effect does not require any user action. The UI returns to its proper functionality after a few minutes. WinOF driver versions older than 3.0.0 do not support upgrade. When installing WinOF v3.1.0 or WinOF v3.2.0 on a machine which has an older driver (v3.0.0 and below), the older driver is uninstalled prior to starting the new installation and any previous configurations is lost. This issue does not apply when upgrading WinOF 3.1.0 to 3.2.0, all configurations will be saved. 4.2 InfiniBand Drivers Known Issues The following is a list of InfiniBand limitations and known issues of the various components included 0in this Mellanox WinOF VPI release. IPoIB does not support: MAC address change QoS (packet priority) Hyper-V Load balancing and fail-over (LBFO) Connected Mode Partition Setting multiple IP address on IPoIB interface on Window 2008 R2 is not functional due to Microsoft bug. Memory Registration: Memory registration is limited to up to 20GB In an interoperability environment that has both Linux and Windows OSs, the MTU value must be the same, otherwise packets larger than the minimum will not go through Old InfiniBand application must be recompiled with latest SDK. iscsi boot over IPoIB is currently not functional. When trying to configure and adapter card that has only one port, the following message is displayed and can be safely ignored: An invalid registry PortType value was detected. The port will be initialized with default configuration 4.3 Ethernet Drivers Known Issues The following is a list of general limitations and known issues of the various components included in this Mellanox WinOF VPI for Windows release. 11
Known Issues VLAN creation: VLANs on the same machine cannot be assigned dynamic IPs (using DHCP) of the same subnet Replacing an adapter card may require reconfiguring LBFO bundles and/or VLANs Changing the port type from ETH to IB is not supported if Vlan/LBFO is defined over the port. If you change, the following error is displayed: "Upper VLANs have been detected. Please remove associated VLANs and repeat the operation. To change the port, remove the VLAN from the LBFO. Hyper-V over LBFO is not supported High multicast drop rate on multicast storming. In order to resolve this issue, please increase RX ring size to 4096, and increase RX ring number (based on the computer architecture up to the number of core number). For further information, please refer to section "Tunable Performance Parameters" in the User Manual. When using LBFO, it is able to take up to 5 seconds to fail back to primary. If you work with MS clustering over the LBFO interface you have to change the health bit time out to be more than 5 seconds. When a link comes up, it may blink for few times. This may affect the LBFO behavior. 4.4 RoCE Known Issues RoCE does not support: Routable RoCE - Traffic cannot go through the router Multicast traffic VLAN 12