Optimizing the GigE transfer What follows comes from company Pleora.

Transcription

1 Optimizing the GigE transfer What follows comes from company Pleora. Selecting a NIC and Laptop Based on our testing, we recommend Intel NICs. In particular, we recommend the PRO 1000 line of Intel PCI Express (PCIe) NICs, which are affordable, receive data well, and are easy to find. In cases where it is difficult to find an Intel NIC that suits your needs (for example, for some laptops) or where you already have a non-intel NIC, we also support all other brands of NICs, with some limitations. PCIe NICs versus PCI NICs Pleora recommends PCIe NICs, which are found in most modern computers (instead of PCI NICs). These NICs provide 1 Gbps of bandwidth. If you have a PCI NIC in your computer, the theoretical bandwidth of the bus is Gbps (133 MBps). However, it is important to note that the full bandwidth may not be available, due to the following constraints: The PCI bus is shared with other devices in the computer, meaning that all of the cards in your computer share the same bandwidth. There is overhead associated with PCI bus communication. Note: MISSING_PACKET errors can be an indication that there is insufficient bandwidth. Choosing a Laptop Pleora has seen good performance on laptops that have the Intel Pro chipset. When selecting a laptop, this is the recommended chipset. If selecting an add-on laptop Gigabit card, we recommend ExpressCard, which streams well at maximum bandwidth. List of NICs with Potential Issues Table 1: Marvell Yukon 88E8058 PCI-E Gigabit Ethernet Controller (On MacBook Pro) Component Vendor ID: Device ID: OS: Problem: Solution: Details 11AB 436A Microsoft Windows 7, 64 bit. Drops packets when streaming close to 400 Mbps. Use another NIC or another bandwidth range. Table 2: SysKonnect SK-9521 V2.0 10/100/1000 Base-T Adapter (Marvell Yukon) Component Details Vendor ID: 1148 Page 1

2 Table 2: SysKonnect SK-9521 V2.0 10/100/1000 Base-T Adapter (Marvell Yukon) Component Details Device ID: 4320 OS: Problem: Solution: Microsoft Windows 7, 23 and 64 bit. Number of receive (RX) descriptors is too low when used with the default driver settings. Increase the number of RX descriptors on the driver configuration page. Table 3: Intel Gigabit CT Desktop Adapter (Intel Pro 1000 CT 82574L) Component Details Vendor ID: 8086 Device ID: OS: Problem: Solution: 10D3 Microsoft Windows XP, 32 bit. Other Windows versions work properly. Does not stream properly. Do not use this adapter with the 32-bit version of Windows XP. Use another adapter or a different version of Windows. Table 4: Intel Gigabit PL Desktop Adapter (Intel Pro 1000 PL 82573L) Component Details Vendor ID: 8086 Device ID: OS: Problem: Solution: 10D3 Red Hat Enterprise Linux 5.4, 32 and 64 bit. Jumbo frames may not work properly on some laptops. By design. See section 17 "ASPM/Jumbo Frames Disabled Due to Early Receive Threshold Overrun Buffer" in Intel Family Gigabit Ethernet Controllers Specification Update, available at: Selecting the Appropriate Version of the Manufacturer s Driver The latest version of a driver is most often the best version, since it has the latest bug fixes from the manufacturer. The driver that is distributed with the operating system is often not the most recent Page 2

3 and a later version can be obtained from the manufacturer. We generally suggest using the latest version of the driver provided by the manufacturer. Configuring Jumbo Packets, Receive Buffers, and the Interrupt Moderation Rate You can configure the NIC to ensure the best possible performance. The following table lists the NIC settings that you should adjust to maximize performance. Table 5: Configuring Jumbo Packets, Receive Buffers, and Interrupt Moderation Rate Setting Details Recommendations Jumbo packets (jumbo frames) Allows for a larger Ethernet frame size, which corresponds to a larger data payload per Ethernet frame, and allows each block to be transferred using fewer packets. As a result, this setting decreases the amount of interrupts that the PC has to service and results in lower overall CPU usage. When a system exceeds 9 KB Ethernet frame size, the gain in CPU savings becomes negligible. If the image size will be smaller than the size of the jumbo frame, the use of jumbo frames should be avoided. If devices across the network support jumbo frames, then jumbo frames should be enabled and configured to support the desired frame size. Before you enable jumbo frames, ensure that all of your NICs, switches, and devices/cameras support jumbo frames (most new devices do). Receive buffers (receive descriptors) Sets the number of receive buffers used by the adapter when copying data to memory. If the system runs out of receive descriptors, the NIC will drop packets (older packets will be overwritten). We recommend that you increase the receive descriptors to the maximum possible value. While an increased number of receive buffers allows you to store more packets and improves stability (that is, fewer older packets are overwritten), more memory is required. Interrupt moderation rate (interrupt throttling) When a packet arrives, the adapter generates an interrupt that allows the driver to handle the packet. At greater link speeds, more interrupts are created and CPU usage increases, resulting in poor system performance. When you enable interrupt moderation, the interrupt rate is lower, resulting in better system performance. Note: The number of receive descriptors does not affect your system s CPU usage. To improve system performance, you can configure the interrupt moderation rate. This setting defines the number of interrupts per second. Note that decreasing this value decreases the CPU usage. For applications where low latency is critical, this setting should be approximately 8000 interrupts per second. If some latency as a trade-off for lower CPU usage is acceptable, then set the number of interrupts per second at approximately If you do not know what to set, you have two options: Page 3

4 Table 5: Configuring Jumbo Packets, Receive Buffers, and Interrupt Moderation Rate Setting Details Recommendations Use the default value set by the manufacturer s drivers. Some NICs support adaptive throttling mode. This mode allows for good overall performance based on the NIC load. Experimentation with your system will provide you with the appropriate value for this parameter. To configure jumbo packets, receive buffers, and interrupt moderation rate On the Windows Start menu, click Control Panel. Navigate to the Device Manager section of the Control Panel. In Windows 7, you can locate this section by clicking Hardware and Sound. Alternately, you can open the Device Manager section of the Control Panel by typing the following command at the Windows command line prompt: > rundll32.exe devmgr.dll DeviceManager_Execute Expand the Network adapters list, right click the NIC that you want to configure, and then click Properties. Click the Advanced tab in the Adapter Properties dialog box. In most cases, you do not need to change the network adaptor s default settings. However, you can adjust the following settings depending on your data rate and frame rate. For a description of each setting and recommended settings, see Table 5: Configuring Jumbo Packets, Receive Buffers, and Interrupt Moderation Rate. Page 4

5 Jumbo Packets (also known as Jumbo Frames) Receive Buffers (also known as Receiver Descriptors) Interrupt Moderation Rate (also known as Interrupt Throttling). The default setting (Adaptive) adjusts the interrupt rates dynamically, depending on traffic type and network usage. Choosing a different setting can improve network and system performance. At higher data rates, a High interrupt moderation setting may improve system performance. At lower data rates, a Low interrupt moderation setting is preferred, since delayed interrupts cause additional latency. Note: Because the procedures for configuring these options vary from one manufacturer to another (including the way the value is interpreted), we recommend that you refer to the documentation accompanying the NIC for instructions. Click OK. Minimizing the Impact of Windows Firewall and Other Filter Drivers Firewalls, anti-virus software, third-party filter drivers, Wireshark, and Windows networking traffic monitoring software can affect the performance of your system because they monitor the incoming packets at different network stack layers. For powerful systems or a low bandwidth stream, the impact of these filters may be negligible. For other systems, the impact can be significant and needs to be handled appropriately. To minimize the impact of Windows Firewall and other filter drivers, ensure you: Turn off the Windows firewall. Disable the anti-virus scan. Disable third-party filter drivers (on the NIC s properties page). Do not open the Networking tab in Windows Task Manager. Page 5

6 Do not run the Windows Performance Monitor (perfmon.exe). Do not run Wireshark. Only use Wireshark when it is needed for debugging purposes. Configuring Multiple NICs in a Single Computer When there are multiple NICs installed in the same computer, we recommend that you configure each NIC with an IP address on a unique subnet. The following scenario shows a computer with three NICs: NIC #1 IP address: Subnet mask: For example, an office network. NIC #2: IP address: Subnet mask: For example, connected to one GigE Vision device. NIC #3: IP address: Subnet mask: For example, connected to another GigE Vision device. With the configuration above, NIC #1 is on an office network with a DHCP server, which provides an IP address on the x subnet. NIC #2 and NIC #3 use a static IP address. To configure a static IP address for a NIC 1. In the Windows Control Panel, click Network and Internet. 2. Click Network and Sharing Center. 3. In the left-hand panel, click Change adapter settings. Page 6

7 4. Right-click the NIC and then click Properties. 5. Click Internet Protocol Version 4 (TCP/IPv4) and then click Properties. 6. Select Use the following IP address and give the NIC an IP address. We recommend you leave the Default gateway box empty. Page 7

8 7. Close the open dialog boxes to apply the changes. Note: If NIC #2 and #3 both use Obtain an IP address automatically (the default Windows setting) and there is no DHCP server on either network, both NICs automatically get an LLA IP address of x.y and a subnet mask of In this situation, the IP addresses are on the same subnet, which is an incorrect configuration. Note: If there is an unused NIC in your computer and it is not connected to a network, then you do not need to configure its IP address. Configuring Power Management For image streaming applications that run for an extended period of time, we recommend that you change the Windows power plan settings to ensure the computer and NIC never go to sleep. If you do not change the power plan settings, the computer or NIC may go to sleep when there is no activity from the keyboard or mouse for certain amount of time and the image stream may be interrupted. To configure the power plan settings 1. In the Windows Control Panel, click Hardware and Sound and then click Power Options. Page 8

9 2. Beside the selected power plan, click Change plan settings. 3. In the Put the computer to sleep list, click Never. 4. You do not need to change the Turn off the display option. You can leave it at its current setting. Page 9

10 To configure the power settings for the NIC 1. In the Windows Control Panel, click Network and Internet. 2. Click Network and Sharing Center. 3. In the left-hand panel, click Change adapter settings. Page 10

11 4. Right-click the NIC and then click Properties. 5. Click Configure. 6. Configure the power management settings for the NIC to ensure the computer does not turn off the device to save power. 7. Click OK. 8. Repeat steps 4-7 for each NIC on your computer. Page 11

12 LINUX OPERATING SYSTEM ONLY To enable jumbo packets You can change the frame size using the following commands: ifconfig eth[index] mtu [SIZE] ifconfig eth[index] up where, [INDEX] = 0,1,2,,n (depending on the number of NICs present) [SIZE] = desired frame size For example, ifconfig eth0 down ifconfig eth0 mtu 8164 ifconfig eth0 up - Or - The configuration can be done at boot time by modifying the etc/sysconfig/network-scripts/ifcfgeth[index] file with the line: MTU = [SIZE] To configure receive descriptors Note: The Intel PCI driver is named e1000 and the PCIe version is named e1000e. For the PCI version of the driver, for example (e1000): You can customize the Intel driver using the RxDescriptors parameter on the modprobe command line. This can be done by loading the driver with: modprobe e1000 RxDescriptors = 512 If you have more than one NIC that you are configuring, the parameter for each adapter must be comma-delimited: modprobe e1000 RxDescriptors = 512, 512, 512 Alternately, the configuration can be performed at boot time by modifying the /etc/modprobe.conf file with the line: options e1000 RxDescriptors = 512 For the PCI Express version of the driver, for example ( e1000e ): The tool ethtool provided by many Linux distributions can be used to configure the number of receive descriptors (buffers), as per the following command: ethtool -G ethx rx 512 where, ethx represents your NIC (for example, eth0) To configure interrupt coalescing You can customize the Intel driver using the InterruptThrottleRate parameter on the modprobe command line. This can be done by loading the driver with: modprobe e1000 InterruptThrottleRate = 8000 Page 12

13 If you have more than one NIC that you are configuring, the parameter for each adapter must be comma-delimited: modprobe e1000 InterruptThrottleRate = 8000, 8000, 4000 The configuration can be performed at boot up by modifying the /etc/modprobe.conf file with the line: options e1000 InterruptThrottleRate = 8000 Configuring Socket Buffer Size When you configure the socket buffer size, you are increasing the number of UDP datagrams that the operating system can buffer before delivering them to the network stack. You must configure the socket buffer size to ensure that it is large enough to allow the application to survive normal CPU scheduling latency without dropping packets. For high bandwidth applications (for example, streams of 800 Mbps or higher), the default value is too small and should be increased. If the value is too small, the side effect is lost packets, which could have negative side effects on your system. This should be avoided at all times. If the value is too large, the application may use excessive amounts of memory. It may also increase the latency in the system after the connection is lost. Pick this number with caution. In-house testing shows good results when this value is set to 10,485,760. This is a good starting value if you need to configure the socket buffer size. Note: Manual configuration of the socket buffer size is not available in the Windows operating system. It is only used on the Linux operating system. To configure the socket buffer size You can change the socket buffer size with the following command: sysctl -w net.core.rmem_max= sysctl -w net.core.wmem_max= The configuration can be performed at boot up by modifying the /etc/sysctl.conf file with the lines: net.core.rmem_max= net.core.wmem_max= Page 13