Technical white paper Configuring SR-IOV with HP Virtual Connect and Microsoft Hyper-V Table of contents Abstract... 2 Overview... 2 SR-IOV... 2 Advantages and usage... 2 With Flex-10... 3 Setup... 4 Supported adapters... 4 Supported guest OS for G7, Gen8, and Gen9 ProLiant Server Blades... 5 Configuring SR-IOV... 5 Configuring Virtual Connect... 5 Enabling SR-IOV in the RBSU... 8 Configuring SR-IOV in the OS... 8 Enabling SR-IOV on the adapter... 10 Creating and configuring SR-IOV on a virtual switch... 11 Creating and configuring the virtual machine... 12 Enabling SR-IOV on the Virtual Machine... 14 Verifying the VM network adapter status... 15 Updating Integration Services... 16 Verifying VM network adapter drivers... 17 Adding multiple VFs to the same VM:... 17 Checking the number of VFs allocated to a VM... 19 Working with VLANs... 20 Configure a VLAN on the PF... 20 Configure a VLAN on the Virtual Switch... 20 Configure a VLAN on the Virtual Machine... 21 Verifying VLAN configurations... 22 Trouble Shooting and Debugging Tips... 23 Get-NetadapterSriov common error messages and reasons... 23 Troubleshooting SR-IOV enablement on VM... 23 Support and other resources... 25 Resources... 25 Documentation feedback... 25 See the latest version at hp.com/go/vc/manuals
Abstract This paper explains how to implement SR-IOV on an HP ProLiant server blade, Microsoft Hyper-V Server 2012, and Virtual Connect. It outlines the procedures and best practices for configuring the Virtual Connect Manager, the ProLiant server blade, the adapter, and the virtual machine for SR-IOV. Overview SR-IOV support has been enhanced in Virtual Connect version 4.40, giving end users an option to configure Virtual Functions on all Ethernet ports of a network adapter. This document describes SR-IOV, its benefits, and how SR-IOV support on Virtual Connect brings high performance to the virtualized world. SR-IOV The PCI-SIG defined a standard approach to creating and managing natively shared devices through the Single Root I/O Virtualization and Sharing (SR-IOV) specification. This new standard, allows a PCIe device to appear as multiple separate physical PCIe devices. SR-IOV works through physical functions (PF) and virtual functions (VF). The PCI-SIG SR-IOV specification introduces two PCIe function types: A Physical Function (PF) - This function is the primary function of the device and advertises the device's SR-IOV capabilities. The PF is associated with the Hyper-V parent partition in a virtualized environment. One or more Virtual Functions (VFs) - Each VF is associated with the device's PF. A VF shares one or more physical resources of the device, such as a memory and a network port, with the PF and other VFs on the device. Each VF is associated with a Hyper-V child partition in a virtualized environment. Microsoft has been investing in SR-IOV for device I/O from virtual machines. The key points that are addressed by SR-IOV are reduction in latency, increased throughput, lowered compute overhead, and future scalability. SR-IOV enables virtual machines to directly talk to the physical NIC. It allows a single physical connection to be utilized by multiple VMs, giving them the performance as in the old world, before hypervisors came along. In SR-IOV the network traffic bypasses the V-switch layer of the Hyper-V virtualization stack. It uses I/O memory management unit (IOMMU) to differentiate between the traffic streams and apply memory and interrupt translations between the PF and VFs. This allows traffic streams to be delivered directly to the appropriate Hyper-V parent or child partition. As a result, the I/O overhead in the software emulation layer is reduced and achieves network performance that is nearly the same as in non-virtualized environments. For details on Microsoft s Architecture for SR-IOV, see the Microsoft MSDN website. For more information on the PCI-SIG SR-IOV standard, see the SR-IOV specification (v1.1) on the PCI-SIG website. Advantages and usage In a shared I/O model, the Hypervisor presents a very simple Ethernet Controller to the VM. With SR-IOV, the Hypervisor presents the actual configuration space to a specific VF, granting the Virtual Function driver within the VM physical access to the VF resources. The key advantages of using SR-IOV for virtualization are: Increased Network Performance VMs have direct access to the network adapter. Lower CPU Utilization Switching of Ethernet frames is offloaded from the hypervisor vswitch to the adapter. Low Latency VM network traffic bypasses the hypervisor and associated data movement. Efficient sharing of PCI devices Optimized performance and capacity These results allow the support of more VMs per host, delivering increased network bandwidth utilization on the host and providing greater performance to guest VMs. For example, even a medium loaded SQL Server can be difficult to virtualize because of the intense I/O load it generates. With better I/O performance, SR-IOV can help a SQL Server achieve the higher network performance it needs to run in a VM. At the same time, financial applications and interactive applications (such as video streaming or VDI) benefit from SR-IOV for its lower latency.
SR-IOV is a key technology that will allow you to virtualize your most complicated and demanding workloads and hence is very much relevant in the HPC world. Complex workflows demand increasing flexibility from HPC platforms. With SR-IOV there is substantial improvement in latency and negligible bandwidth overhead, making fully virtualized HPC clusters viable. With Windows Server 2012 Hyper-V SR-IOV can be deployed in conjunction with key capabilities such as Live Migration. In addition, SR-IOV can be deployed on existing servers, because it is compatible with most current-generation 10 Gb NICs. With Flex-10 Using HP Flex-10 technology on Virtual Connect, a user can configure a single server blade network port to represent four physical network interface controllers (NICs), called FlexNICs. These four FlexNICs appear to the operating system (OS) as discrete NICs, each with its own driver. While the FlexNICs share the same physical port, traffic flow for each one is isolated with its own MAC address and virtual local area network (VLAN) tags between the FlexNIC and VC interconnect module. As of today, each dual-port Flex-10 device supports up to eight FlexNICs, four on each physical port. With SR-IOV technology provided on Flex NICs there is a major advantage for HP Server Blades. The SR-IOV architecture on VC allows the expansion of up to 512 PCIe functions, depending on hardware capability. For example, if you have an Ethernet adapter supporting 64VFs per physical port, on a single FlexNIC you can configure 8 to 64 VFs using Virtual Connect. The VFs can be assigned to one or more VMs each having its own IP and dynamic MAC Address. This provides a significant increase in server consolidation and performance. With SR-IOV, data handling operates in a much more native and direct manner, reducing processing overhead and enabling highly scalable PCI functionality. SR-IOV technology complements the HP Flex-10 Technology and they together improve I/O efficiency without increasing overhead burden on CPUs and network hardware.
Setup The diagram below shows an environment where three Virtual Machines route their traffic through the VFs directly to the Physical NIC, bypassing the Hypervisor. SR-IOV is configured on the network adapter from Virtual Connect. Supported adapters SR-IOV is supported on the following servers: ProLiant BL680c G7 servers ProLiant BL620c G7 servers All Gen8 and Gen9 server LOMs Virtual Functions use hardware resources. They are managed through their underlying PFs. There may be several VFs associated with a single PF. The number of VFs available per physical port on an adapter differs across vendors and devices. Typically as of today, HP SR-IOV supported NIC adapters are offering 24 or 64 VFs per PF. Below is a list of HP SR-IOV Adapters with their supported VF number (as per SPP version 2015.04.0). Network Adapter Number of SR-IOV VFs per Physical Port HP Flex-10 10Gb 2-port 530FLB Adapter 64 HP FlexFabric 10Gb 2-port 534FLB Adapter 64 HP FlexFabric 20Gb 2-port 630FLB Adapter 64 HP Flex-10 10Gb 2-port 530M Adapter 64 HP FlexFabric 10Gb 2-port 534M Adapter 64 HP NC552m Flex-10 Adapter 24 HP FlexFabric 20Gb 2-port 630M Adapter 64 HP NC553m 10Gb 2-P FlexFabric Adapter 24
Network Adapter Number of SR-IOV VFs per Physical Port HP FlexFabric10Gb 2-port 554M Adapter 24 HP Flex-10 10Gb 2-port 552M Adapter 24 HP FlexFabric10Gb 2-port 554FLB Adapter 24 HP FlexFabric 20Gb 2-port 650FLB Adapter 24 HP FlexFabric 20Gb 2-port 650M Adapter 24 HP FlexFabric 10Gb 2-port 536FLB Adapter 64 Supported guest OS for G7, Gen8, and Gen9 ProLiant Server Blades The following OSs are the supported Guest OSs for Hyper-V (as per SPP version 2015.04.0): Microsoft Windows Server 2012 Microsoft Windows Server 2012 R2 Configuring SR-IOV Use the following steps to configure SR-IOV: 1. Configuring Virtual Connect 2. Enabling SR-IOV in the RBSU 3. Configuring SR-IOV in the OS 4. Enabling SR-IOV on the adapter 5. Creating and configuring SR-IOV on a virtual switch 6. Creating and configuring the virtual machine 7. Enabling SR-IOV on the virtual machine 8. Verifying the VM network adapter status 9. Updating Interation Services 10. Verifying VM network adapter drivers 11. Adding multiple VFs to the same VM 12. Checking the number of VFs allocated to a VM Configuring Virtual Connect With VC 4.40, you can allocate SR-IOV VFs to any of the Ethernet ports of a Network Adapter. From VC 4.10 to VC 4.31, all VFs were by default allocated on the third Physical Function of each port on a Network Adapter. There was no option available on VCM for the user to configure or modify the VFs. From VC 4.40 and higher, SR-IOV can either be configured in Simplified mode or Advanced mode. Simplified Mode The SR-IOV type for all the Ethernet connections is set to DEFAULT. All VFs are allocated on the third PF. This is to maintain the backward compatibility and all other connections have zero VFs assigned. Advanced mode The SR-IOV type is set to AUTO for all the Ethernet connections corresponding to the third PF. All the remaining connections are set to DISABLED. In Advanced mode the user can configure the following three SR-IOV types to an Ethernet Connection: Custom - Allows the user to allocate the number of Virtual Functions in multiples of 8. Auto - The remaining unallocated Virtual Functions are distributed among the Ethernet connections. Disabled - No Virtual Functions are allocated to the Ethernet connection.
Use the following table to compare Simplified and Advanced Mode. Simplified Mode In this mode, you can continue with the existing SR-IOV behavior (prior to VC 4.40). All the VFs are allocated on the Third PF. VFs cannot be distributed among the PFs. The number of VFs allocated on the third PF cannot be set to zero. Advanced Mode In this mode, you can configure the SR-IOV type and allocated VFs. You can request the number of VFs to allocate on a PF using SR-IOV Type "Custom". The VFs can be distributed uniformly on all the PFs using SR-IOV type "Auto". The number of VFs can be set to zero on a PF using SR- IOV Type "Disabled". After firmware upgrade from a VC version prior to VC 4.40, all existing Profiles will be by default in Simplified Mode. Table 2: Simplified Mode versus Advanced Mode. Profiles created after firmware upgrade will be by default in Advanced mode. To configure SR-IOV from the VCM CLI, use the commands add/set enet-connection and set profile. To configure SR-IOV from the VCM GUI, use the Server Profile screen. On the VCM GUI the SR-IOV mode can be set for a server profile by clicking the SR-IOV Mode icon. A pop-up appears with the Advanced and Simplified buttons.
The SR-IOV Type can be configured from the Ethernet Connection Settings icon on the Define/Edit Server Profile Screen. Observe the following: When the SR-IOV Mode is set to Advanced then the SR-IOV Type can be set to Custom, Auto, or Disabled. When the SR-IOV Mode is set to Simplified then the SR-IOV Type is set to Default. When the SR-IOV Type is set to Custom, then you can select the number of SR-IOV VFs from the Requested SR-IOV field. The VFs assigned to a server profile can be viewed on the Device Bays screen under Device Bays. An entry is added to the system log when VFs are assigned to a server profile For more information on configuring SR-IOV with Virtual Connect Manager, see the HP Virtual Connect 4.40/4.41 User Guide in the VC Information Library.
Note: VC does not support SR-IOV configuration on FC, FCoE and iscsi connections. Enabling SR-IOV in the RBSU To configure SR-IOV in the RBSU: 1. Boot the server, and then press F9 2. Go to System Options -> Virtualization Options 3. Verify the following options are enabled: Virtualization Technology Intel Vt-D SR-IOV Adding the Hyper-V Role to the server To configure SR-IOV in the OS, add the Hyper-V Role through Server Manager: 1. Launch Server Manager. 2. Click on Add roles and features, and then click Next. 3. Under Installation type, select Role based or feature-based Installation, and then click Next. 4. Under Server Selection, select your server, and then click Next.
5. Under Server Roles, select the Hyper-V checkbox, and then click Next. 6. On the Features screen, select Next. 7. On the Hyper-V screen, select Next. 8. On the Virtual Switches screen, select a network connection for the Virtual Switch. You can also select a network connection later when setting up a Virtual Switch. 9. On the Migration screen, select the checkbox, and then click Next. Select this checkbox only if you plan to perform live migrations of VMs. 10. On the Default Stores screen, select the storage location for the VM files, and then click Next. 11. On the Confirmation screen, select the Restart the destination server automatically if required checkbox. 12. Click Install. The Hyper-V installation starts on the server. Once the installation completes, Hyper-V appears on the left side menu of the Server Manager as shown below.
Enabling SR-IOV on the adapter 1. From the Server Manager menu bar, select Tools-> Hyper-V Manager. 2. On the right, under the Actions pane, click Connect to Server... 3. Select Local Computer, and then click OK. 4. Verify SR-IOV is enabled on the network adapter of the OS. A. Under Network and Sharing Center, right click on the Ethernet Network Connection. B. Select Properties -> Configure -> Advanced. C. Find the SR-IOV property, and then set its value to Enabled.
Creating and configuring SR-IOV on a virtual switch 1. Create a virtual switch: A. Launch Hyper-V Manager. B. On the right side of the Hyper-V Manager screen, under the Actions pane, select Virtual Switch Manager. C. Under Create virtual switch pane, select External, and then click Create Virtual Switch. 2. Configure the virtual switch properties: A. Name the virtual switch. B. Under Connection type, select the correct adapter, and then check the Enable SR-IOV checkbox.
C. Click Apply, and then OK. Creating and configuring the virtual machine To create a virtual machine and connect the network adapter to the virtual switch: 1. Launch Hyper-V Manager. 2. On the right side of the Hyper-V Manager screen, select New -> Virtual Machine. The New Virtual Machine Wizard is displayed. 3. On the Before You Begin screen, click Next.
4. On the Specify Name and Location screen, name the virtual machine, and then click Next. The default VM store location can be changed, if preferred. 5. On the Specify Generation screen, select Generation1 or Generation2, depending on the usage. Generation2 allows boot options, such as PXE Boot, SCSI boot options, etc. 6. On the Assign Memory screen, enter the startup memory, and then click Next. The example below shows a Windows Server 2012 as the Guest OS with 2048MB of RAM assigned. 7. On the Configure Networking screen, select the virtual switch you created or configured with SR-IOV.
8. On the Connect Virtual Hard Disk screen, specify the size of the hard disk for the VM, and then click Next. 9. On the Installation options screen, select the option you want for the OS installation on the VM. If an OS image is mounted on the Host server, select the Install an operating System from a bootable CD/DVD-ROM option. 10. Verify the description details on the Summary screen, and then click Finish. Enabling SR-IOV on the Virtual Machine 1. Launch Hyper-V Manager. 2. Find the virtual machine on the right sidebar of the Hyper-V Manager. 3. Open the VM Settings, and then select Hardware Acceleration.
4. Select the Enable SR-IOV checkbox. 5. Click Apply, and then OK. 6. Power on, and then connect to the VM using Connect. A remote desktop connection is established to the VM. 7. Complete the OS installation. Verifying the VM network adapter status 1. Launch Hyper-V Manager. 2. On the Hyper-V Manager screen, under the Summary tab, the Heartbeat property displays OK(Application Healthy). 3. Click on the Networking tab.
4. Verify the connection status. If the connection shows a status of Degraded (Integration services upgrade required), then Windows Integration Service needs to be updated on the VM. Updating Integration Services 1. Launch Hyper-V Manager. 2. Connect to the VM. 3. On the toolbar, click Action -> Insert Integration Services Setup Disk. If the the installation does not start, be sure the image is mounted in the CD/DVD drive. 4. After the installation completes, verify the VM connection displays an OK (SR-IOV Active) status. Verify the VM aquired an IP Address and can reach or be reached from another system in the same network.
Verifying VM network adapter drivers Verify the correct driver is installed on the virtual machine. If the driver is not installed, the VF will appear as Degraded and will not show up correctly. On the guest OS, locate a degraded VF using Device Manager: Device Manager -> Other Devices -> Network Controller Once the network adapter driver is updated, the VF is displayed under Network adapters. Adding multiple VFs to the same VM: 1. Power off the VM.
2. In Hyper-V Manager, launch the VM settings. 3. Under Add Hardware, select Network Adapter, and then click Add. 4. On the Network Adapter screen, select the virtual switch. 5. On the Hardware Acceleration screen, be sure to select the Enable SR-IOV checkbox. 6. Click Apply, then OK. 7. In Hyper-V Manager, select the networking tab for the VM. There should be one more Network Adapter added with status OK (SR-IOV active).
Checking the number of VFs allocated to a VM The number of VFs allocated to a VM can be seen using the following command: get-netadaptersriov <Portname> fl *
You can also use the command get-vmswitch fl * Working with VLANs Through Virtual Connect, a user can assign a VLAN to an SR-IOV Virtual Function if the Ethernet Network has a VLAN tag attached to it. No new configuration is required on VCM. On VCM under Ethernet Networks the SUS should be assigned a VLAN ID. The same holds true If VLAN tunneling is enabled on the Network. On VCM, VLAN tunneling should be enabled on the Ethernet Network. This is an existing feature available in VC. The tagged packets are then be sent to the Guest VM running SR-IOV. The Host requires the configuration of VLAN at three places The Physical Function The virtual switch The virtual machine Configure a VLAN on the PF Under Connections go to Properties -> Configure -> Advanced. Enter the VLAN ID in the VLAN Identifier field. Configure a VLAN on the Virtual Switch 1. In Hyper-V, go to Virtual Switch Manager. 2. Select your Virtual Switch.
3. Under the VLAN ID section, check the Enable virtual LAN identification for management operating system checkbox, and then enter the VLAN value. 4. Click OK. Configure a VLAN on the Virtual Machine 1. Power off the VM. 2. Go to Settings -> Network Adapter.
3. Under VLAN ID, enable the Enable virtual LAN identification checkbox, and then enter the VLAN value. 4. Click OK. Verifying VLAN configurations Be sure an IP is assigned to the VF. 1. Launch Hyper-V Manager. 2. In Hyper-V Manager, go to the Networking Tab of the VM, and then verify the adapter settings.
Trouble Shooting and Debugging Tips To ensure SR-IOV is working correctly on your setup, use the command on the server: get-netadaptersriov SriovSupport should be Supported and Enabled should be True. For the Ethernet connection which is configured as a Virtual switch, NumVFs will display the actual number of VFs assigned to it from the VC server profile. For all other connections NumVFs will display a Default value of the maximum number of VFs, the network adapter supports. If you do not see SR-IOV as Supported and True on your server: 1. Be sure SR-IOV is enabled on the VC server profile and the profile is assigned to the server. 2. Verify the server supports SR-IOV and you have an SR-IOV supported network adapter. If your hardware supports SR-IOV check for the following: SR-IOV supported firmware and drivers are installed on the server. The latest SPP is installed on the server. SR-IOV is enabled in the RBSU. Get-NetadapterSriov common error messages and reasons Missing PF Driver Confirm that your NIC supports Windows 2012 SR-IOV. Make sure you have the latest supported NIC driver installed. NoIoMmuSupport Check Hyper-V is enabled. Make sure you have restarted your server after installing Hyper-V. NoVfBarSpace Check if SR-IOV is enabled on the OS for the port. Go to Network -> Ethernet Connection -> Properties -> Configure -> Advanced. Check if SR-IOV is enabled on the VC and VFs are assigned to the Ethernet Connection on the server profile. Troubleshooting SR-IOV enablement on VM If SR-IOV is not working properly on your VM then check the output of the following commands: 1. Run the following command on your host: get-vmhost fl Verify the following parameters: InternalNetworkAdapters ExternalNetworkAdapters IovSupport IovSupportReasons : {HP SRIOV Test} : {HP SRIOV Test_External} : True : {OK} IovSupport should display a True value. If it is False, then the appropriate reason is displayed. 2. Run the following command on your host: get-vmswitch fl Verify the following parameters: IovEnabled : True IovVirtualFunctionCount : 16 IovVirtualFunctionsInUse : 2 IovSupport : True IovSupportReasons : {OK}
IovEnabled and IovSupport should display True values. If it is False, then the appropriate reason is displayed. The IovVirtualFunctionCount displays the number of Virtual Functions assigned in the VC server profile. 3. Run the following command on your host: get-vmnetworkadapter fl Check for the following parameters: SwitchName : SR-IOVVSwitch SwitchId : 04be3e9e-3f46-4c08-9c96-262ff4d8e8d0 Connected : True VFDataPathActive : True Status : {Ok} IPAddresses : {90.1.2.132, fe80::dc07:a80d:dc9:6876, 2001::d546:ae73:3d8:8cdc} If the Status is Degraded it could be possible the Guest OS is not supported on the Host or that the latest drivers are not installed for your adapter on the VM. Learn more at hp.com/go/vc/manuals
Support and other resources Resources http://hp.com/go/vc/manuals http://hp.com/go/virtualconnect http://hp.com/go/bladesystem Documentation feedback HP is committed to providing documentation that meets your needs. To help us improve the documentation, send any errors, suggestions, or comments to Documentation Feedback (mailto:docsfeedback@hp.com). Include the document title and part number, version number, or the URL when submitting your feedback. Version 830456-001 Edition: 1 June 2015