Network Adapter. Increased demand for bandwidth and application processing in. Improve B2B Application Performance with Gigabit Server

Transcription

1 Improve B2B Application Performance with Gigabit Server Network Adapter By Uri Elzur Business-to-business (B2B) applications and gigabit networking speeds increase the load on server CPUs. These challenges require a careful gigabit adapter design. Optimizing the complete system (network-adapter-server) is a must for the best results, since the whole system is under a heavy load. This article discusses some new mechanisms, key features, performance aspects, and tuning possibilities provided by the new NetXtreme Gigabit Ethernet Controller. Increased demand for bandwidth and application processing in the mission-critical business-to-business (B2B) trading has contributed to the growing popularity of 1 Gbps server adapters. B2B applications provide ways for direct ordering while minimizing inventory levels for providers and consumers alike. The customer data accumulated and generated by these applications becomes the lifeline of the organizations, providing added value and increased revenue. Because handling more external traffic places an additional demand on the servers, the performance and reliability of the servers, their network adapters, and the network as a whole are critical. Hence, support for multiple NICs along with features such as load balancing, link aggregation/trunking, and failover/failback is key. It is important that the CPUs of those servers process applications rather than run the gigabit network. Many enterprise servers, such as Dell PowerEdge 6400 or 4400, and departmental servers feature multiple CPUs to offer more CPU cycles for application processing. These servers often use multiple network interface cards (NICs) for increased reliability, resiliency, and network throughput. Since B2B applications involve highly valued transactions, the network requires strong security to protect data flow as well as server infrastructure. Optimized gigabit NICs combined with security accelerators, such as the SM BCM5800 family, enable servers to deploy Secure Sockets Layer (SSL) and IPSec security at full gigabit wire-speed throughput. Another issue for maintaining network reliability is the huge cable plants created by the increased connectivity and size of server farms. The information technology (IT) personnel need better ways to handle, manage, and troubleshoot large cable plants. The sections that follow describe the family of NetXtreme Gigabit Ethernet Controllers that offer capabilities to address these issues. Designing for 1 gigabit performance Connecting a server to a 1 Gbps network requires significant CPU resources. The CPU must process the communications protocols and manage data movement to/from the NIC and from/to the application (see Figure 1). For the last few years, the increase in network bandwidth has outpaced the bus and CPU speed improvements for servers. This places an even higher burden on the controller designers as they try to optimize throughput and improve host CPU efficiency. The best controller provides high throughput with minimal CPU utilization and optimal use of the limited I/O bus bandwidth. The CPU overhead in processing network data results from processing the frame headers (that is, TCP/IP protocol stack), servicing the controller s interrupt requests for moving data in/out of buffers, and notifying the host protocol stack that operations are complete (see Figure 2). PowerSolutions 111

2 Application reads 64 KB from the disk 64 KB Segment and add application header 8 KB 8 KB 8 KB 8 KB 8 KB Segment into bytes to fit in TCP segment Place data on the NIC output queue NIC transmits from queue and interrupts CPU LAN Figure 1. Moving data from application to wire Process TCP and IP for connection and add TCP and IP header #1 #2 #3 #4 #5 #6 #1 #2 #3 #N However, at 1 Gbps in a poorly designed adapter, a burst of frames can fully consume the host CPU cycles, overrun the allocated buffers, and generate nonstop interrupts. Forced retransmissions may also impact network performance. An analysis of the mechanics of how a server processes network data may be necessary for choosing the right design point for the 1 Gbps adapter. This may require changing the load split between the host CPU and the network controller with changes such as new schemes for interrupt coalescing, jumbo frame support, and TCP/IP checksum off-load. The following sections provide a detailed description of these mechanisms. Processing TCP/IP frames Header in every layer is to be processed Layer 2 (Ethernet) by hardware Layer 3 (IP), Layer 4 (TCP), and higher (RPC, NFS) by software Ethernet Interrupt coalescing The type of network protocol used and the operating system architecture determine the load of protocol stack processing. Although the controller does not directly influence this load (except for offloading when allowed), it can help lower the server load by reducing the number of service requests it issues to the host (that is, interrupts). The controller issues interrupts to the CPU to request service for outgoing or incoming frames. Each interrupt requires two context switches for the CPU: one to move into the interrupt service routine and one to return to the previous code that was running prior to the interrupt, when interrupt service is completed. The NetXtreme Gigabit Ethernet Controller coalesces the per-frame interrupts to generate a smaller number of interrupts to the host. It also provides tuning options for best wire performance or best CPU utilization. The controller tuned for best wire speed moderates the interrupts from 81,000 per second (if one interrupt is generated per frame) to a more manageable level of 3,000 interrupts per second. Servers involved mainly with moving data may be tuned to the best wire speed, while servers bottlenecked by running application code may maximize CPU cycles available for running applications while sacrificing few percentage points of the wire speed. Tuned for best CPU utilization, the NetXtreme Gigabit Ethernet Controller generates only 2,000 interrupts per second, which reduces the load on the host CPU and frees cycles for processing applications. The technique used to achieve this is interrupt coalescing, shown in Figure 3. s driver software manages sophisticated hardware mechanisms that determine the best time to issue the next interrupt. A typical NIC can interrupt its host once per-packet received, or even twice, when early indication of header reception is provided, to allow for early host protocol processing (this technique has been used with lower speed networks where CPU speed is Interrupt causes a CPU context switch Every context switch wastes cycles CPU was in the middle of processing frame #1, when interrupted for #2 CPU activity Interrupts (without coalescing) Frames on the wire Interrupt #1 #2 #3 #4 #1 #2 #3 #4 IP TCP RPC NFS Application Data Interrupts (with coalescing) Improved CPU activity #1 #2 #3 #4 CPU time saved Figure 2. Processing TCP/IP frames Figure 3. Interrupt coalescing 112 PowerSolutions

3 abundant and the network is running slower, but it is not used with high-speed network adapters). In high-speed networks like the 1 Gbps, the rate of interrupts is superfluous, because the CPU is likely to be busy processing the protocol stack or servicing the previous interrupts when the new frame (followed by the new interrupt) is received or transmitted. This intelligent coalescing of interrupts based on network and host load is very host-efficient, preventing unnecessary and nonproductive interrupts and their associated context switches. CPU cycles are freed for the mission-critical server. This scheme is also latencyaware. When the network frame rate slows down, it is less likely that the CPU is already processing previous network controller interrupts when a new frame is received. Hence, to improve the adapter latency, more interrupts are generated per incoming frame. Jumbo frames The use of jumbo frames 1 is another approach to reduce the number of interrupts per second and increase the efficiency of protocol header processing at high speeds. Bulk transfers and intense traffic, such as server to server, tend to benefit from larger frame sizes because they save the segmentation and reassembly and allow full upper layer messages (8400 for NFS) to travel in one frame. This allows a single frame to hold up to 9018 bytes instead of the 1518 bytes provided by the IEEE standard (see Figure 4). 2 Although jumbo frames provide relief to the servers, they require support from the network infrastructure. Gigabit switches from Cisco, Extreme Networks, and others support jumbo frames. Jumbo frames can reduce the frame-per-second (fps) rate from approximately 81,000 fps to less than 14,000 jumbo fps, similar to the number of frames generated by 100 Mbps Ethernet. In turn, the number of protocol headers to be processed by the CPU is about six times smaller. This reduces the CPU load and leaves more power for application processing. As shown in Figure 5, the NetXtreme Gigabit Ethernet Controller is well-suited for the jumbo frame environment and has a flexible interrupt coalescing suitable to the different parameters required by jumbo frame traffic. Such a network can be physically or virtually partitioned from other parts of the network. A physically partitioned network uses infrastructure (hosts, switches, and routers, if used) with jumbo frame support enabled and is physically separated from other parts of the network by routers. A virtually partitioned network employs the IEEE 802.1Q VLAN mechanism that uses VLANcapable switches to provide a seamless connection between the Ethernet IP TCP Data Ethernet IP TCP Data 54 Figure 4. Frames: IEEE and jumbo Performance Mbps Figure 5. Jumbo frame performance 8964 jumbo-enabled portion and other parts of the network. This enables high-performance server islands, which perform the most demanding data-crunching tasks, to be built using jumbo frames. In this scenario, sometimes the data needs to be sent from the jumbo-enabled network segment to other parts of the network, or vice versa. The protocol stacks on both ends, using the Path MTU discovery mechanism (IETF RFC 1191) auto-detect the smallest MTU along the path and adapt the frame size to it (that is, byte frames are transmitted like any regular Ethernet frame). Path MTU discovery services are available on a per-path basis to applications utilizing TCP/IP. Applications using UDP rely on IP fragmentation, which is not performance-optimized. Using jumbo frames can significantly enhance efficiency of both servers and networks by reducing the CPU processing overhead and by increasing the end-to-end throughput (see Figure 6). 814 TCP checksum In the past, NICs were limited to processing the physical and Ethernet layers. With TCP/IP as the prevalent network stack, specific TCP/IP mechanisms can be off-loaded to the NIC. Off-loading the TCP/IP checksum processing saves CPU cycles in this CPU-hungry system. 1 Originally proposed by Alteon Websystems. 2 This has not yet been standardized by IEEE, but has growing importance as network speed increases. PowerSolutions 113

4 The TCP checksum requires summation of the entire TCP payload in 16-bit units. The processor must read the data into its cache and accumulate the data into an internal register to get the required checksum value. The result is placed in the TCP frame header (see Figure 7). This requires accumulation of all the frame s data bytes before transmission of the first byte can commence adding to the end station latency. A -byte frame has 1460 bytes of TCP payload, which amounts to 365 Dwords. Since every Dword is added in 16-bit mode, three CPU cycles (assuming 1.4 clocks per instructions, or CPI) are required. It takes about 500 cycles for the whole frame, which would consume about 0.6 microseconds on an 800 MHz server. The wire time for such a frame is about 12 microseconds. Hence, expected CPU off-load should be about five percent (not accounting for other effects like cache thrashing). Throughput Percent Mbps CPU utilization Standard Jumbo 50 Unlike most NICs, the NetXtreme Gigabit Ethernet Controller provides failover/failback even for a server equipped with heterogeneous NICs. Performance improvements Using these techniques enables the NetXtreme Gigabit Ethernet Controller to demonstrate the highest performance levels in its class. The NetIQ Chariot 3.2 benchmark is used because it measures performance at the application level while using TCP/IP over Ethernet. (This benchmark is widely used by the industry to measure expected application-level performance.) The NetXtreme Gigabit Ethernet Controller exhibits this performance level in various operating systems and frame size configurations (see Figure 8). Improved server availability: teaming and failover Unlike most competing products, the NetXtreme Gigabit Ethernet Controller provides failover/failback 3 even for a server equipped with heterogeneous NICs (including adapters with different speeds). A patent-pending, switch-independent load-balancing scheme (no special configuration or specific switches required) allows the controller to load balance both the receive and the transmit traffic (see Figure 9). This scheme becomes useful for data center servers that expect more Receive traffic than the classic client/server model. NICs load balance the Transmit traffic only, while configured to the switch-independent mode. The Advanced Server Control Suite with its graphical user interface (GUI) can easily set and manage these new modes and the entire common load-balancing modes. Data offset Source port Checksum Sequence number Acknowledgment number Reserved Options Destination port Window Urgent pointer Padding 0 Standard Jumbo Data Figure 6. Jumbo frame throughput and CPU utilization Figure 7. Checksum field in the TCP header 3 Hernandez, Rich. Improving e-business Server Availability. Dell Power Solutions, Issue 3, Discusses link aggregation and load balancing. 114 PowerSolutions

5 Troubleshooting cable interfaces One annoying and time-consuming problem for IT personnel is fighting connectivity issues stemming from cable problems. Numerous factors can result in poor copper cable plants, which can cause excessive packet loss, suboptimal connection speed, reduced network efficiency, and complete connection failure. Sources of problems include poor cable plant housing, unraveling of pair twists, cinched cables, loose connectors, connectors with poor electromagnetic shielding, stretched and broken pairs, and so on. These are often difficult for IT Throughput 1000 personnel to discover. In many cases, equipment is mistakenly returned to its manufacturer for assumed failure before noticing that the problem is actually poor cabling. This outcome is not surprising, because it takes special equipment hooked to both ends of the cable to verify cable health. It might also require several trips to the wiring closet, patch panel, and the server and switch network connection to determine which cable (out of several that make the end-to-end connection) is the culprit for performance or connectivity problems. The NetXtreme Gigabit Ethernet family of products uses a specially designed integrated cable analyzer feature. It provides for a non-intrusive cable analysis at 10 Mbps, 100 Mbps, and 1 Gbps speeds. Examples of situations where the cable diagnostic can be used include: Mbps Troubleshooting during initial network equipment installation Fault determination when network problems arise Routine scanning and maintenance of cable plants Fault determination during network equipment testing 750 on production lines P/E Index The cable diagnostic enables users to identify if the cable plant is at fault. If it is, users can check connections, change cables, or try different connectors to resolve the problem. This saves IT personnel time and prevents erroneous return of equipment to vendors, network downtime, and erroneous fallout of network equipment during production line testing How does it work? When the server is first connected to the network, the PHY uses its sophisticated, digitally controlled receive amplifiers to test for signal amplitude and distortion. Based on this data, it Interrupts 7,500 6,435 Interrupts per second 6,000 4,500 3,000 1, ,831 4 Gbps Switch independent (works with any switch) All channels two way (receive and transmit) Note: For throughput and P/E index, higher is better; for interrupts, lower is better Figure 8. Transmit performance Figure 9. Switch-independent load balancing PowerSolutions 115

6 can determine whether cable connectivity is intact, and the length and quality of the cable. IEEE 802.3z specifies the use of four pairs on Category 5 cable. These pairs can demonstrate different physical characteristics, ranging from being disconnected to having different latencies end-to-end. The PHY can compare these cable pair characteristics and compensate accordingly. User benefits are remarkable in several areas: PHY achieves best-in-class ranges over Category 5 copper cables, averaging 160 meters over IEEE-compliant copper cables PHY auto-compensates for performance variation between pairs of same cable, eliminating the signal skew PHY detects and reports connectivity problems and wire length it measures, which can identify problematic spots on the cable The PHY detects and corrects the polarity of each wire pair. Because hubs and some switches require MDI-X cable, and servers and clients use MDI cables (with pairs crossed), connectivity is often an issue of using the right cable. This features guarantees connectivity with any cable as long as the pair is connected and exhibits reasonable quality Administrators can use a GUI-based application, provided by, to collect cable-related information and troubleshoot problems. The application provides Link Status and Link Quality Indicators, the length of each wire pair, pair mapping (swapped or not) and pair connections (open, shorted, or correctly connected). This is provided with no need to connect special equipment to the wires, which is costly and time-consuming. The The NetXtreme Gigabit Ethernet Controller is designed to meet the challenges of a high-speed network in a system where preserving resources and especially CPU cycles is essential. NetXtreme Gigabit Ethernet family operates over all properly connected Category 5 (CAT5) copper cables. Whenever a problem is found, the controller reports it to the user through the GUI. NetXtreme Gigabit Ethernet Controller meets the challenge The new NetXtreme Gigabit Ethernet Controller demonstrates the best-in-class performance. It is designed to meet the challenges of a high-speed network in a system where preserving resources and especially CPU cycles is essential. New features provide flexibility in using advanced server load balancing and the failover/failback mode. Improved manageability allows the use of the GUI-based Server Control Suite to tune the adapter and troubleshoot some of the most common cable problems. Sophisticated physical layer electronics provide the best operation over any Category 5 cable for an extended range of 160 meters, while providing the user with indications for many of the common cabling problems. Uri Elzur (uelzur@broadcom.com) is technical marketing manager for s network business unit and specializes in NIC architectures. Before joining, Uri was a LAN component architecture manager at Intel. Uri has a master s degree in Computer Science and a bachelor s degree in Electrical Engineering from the Technion University in Haifa, Israel. FOR MORE INFORMATION For more information, service, and support, visit PowerSolutions