by Brian Hausauer, Chief Architect, NetEffect, Inc

Transcription

1 iwarp Ethernet: Eliminating Overhead In Data Center Designs Latest extensions to Ethernet virtually eliminate the overhead associated with transport processing, intermediate buffer copies, and application context switches to make multi- Gigabit Ethernet a better option for data center designs by Brian Hausauer, Chief Architect, NetEffect, Inc The Need for Speed Organizations across the planet are experiencing rapid growth in both the volume of data they must process, and the speed at which processing must occur. As a result, the demand for networking throughput is growing faster than CPU clock speeds and memory bandwidth of servers that process data traffic. To keep up IT managers are building new data center infrastructures using a mix of the best available technologies. The result is an interconnected, heterogeneous, and often semi-interoperable, collection of network fabrics, protocols, hardware, network management software, and personnel skills. Clearly, Ethernet is one of the key connectivity technologies in the development of higher-throughput data center designs. In the past there was a clear performance gap between Ethernet and technologies like InfiniBand and Fibre Channel in storage and clustering designs. But, with the emergence of Gigabit (G-) and 10 Gigabit (10G-) Ethernet, this gap has been virtually eliminated making Ethernet a more viable option in data center applications. The move to Ethernet in the data center, however, comes with challenges. G-Ethernet and 10G-Ethernet solutions historically have had considerable overhead associated with them which, in turn, leads to increased CPU processing requirements as network bandwidth demands increase. As a result, less CPU is available for applications driving network bandwidth requirements. Enter iwarp To help combat the Ethernet-overhead challenges, Adaptec, Broadcom, Cisco, Dell, EMC, Hewlett-Packard, IBM, Intel, Microsoft, and Network Appliance formed the RDMA Consortium. After five months of work this group released the iwarp standard as an extension to TCP/IP. Ethernet has evolved yet again, addressing the performance requirements IT managers seek while meeting the rigorous data center demands for clustering, storage and networking applications. 10G iwarp Ethernet will deliver the low latency requirements for clustering, fast access for storage and high throughput for networking.

2 The iwarp standard eliminates the three major sources of Ethernet networking overhead: transport (TCP/IP) processing, intermediate buffer copies, and application context switches. These three issues collectively account for nearly 100% of CPU overhead related to networking (Table 1). iwarp's advanced techniques reduce CPU overhead, memory bandwidth utilization, and latency by a combination of offloading TCP/IP transport processing from the CPU, bypassing the OS, and eliminating unnecessary buffering. To make this happen, iwarp calls for the movement of data management and network protocol processing to a specialized Ethernet channel adapter (ECA). Table 1: Sources of Overhead A rough estimate of the CPU overhead related to networking for a given Ethernet link speed is that for each Mbit/s of network data processed, one MHz of CPU processing is required. For instance, a fully-used 20-GHz CPU would be needed to drive a 10-Gbit/s bi-directional Ethernet link. [1] A full implementation of the iwarp standard eliminates virtually all of the CPU's networking overhead, returning these CPU cycles to the application. Offloading TCP/IP Processing In Ethernet, processing the TCP/IP stack is traditionally accomplished in software, putting a tremendous load on the host server's CPU. Transport processing includes tasks such as updating TCP context (sequence numbers, etc), implementing required TCP timers, segmenting and reassembling payload, buffer management, resource intensive buffer copies, interrupt processing, etc. CPU load increases linearly as a function of packets processed. With the 10x increase in performance from G-Ethernet to 10G- Ethernet, packet processing will also increase 10x, driving CPU overhead related to transport processing to increase by up to 10x as well. As a result the CPU becomes burdened and, eventually, crippled by transport protocol processing well before reaching Ethernet's maximum throughput. iwarp Ethernet offloads transport processing from the CPU/OS to specialized hardware on the Ethernet network adapter, eliminating 40% of CPU overhead attributed to networking (see Fig. 1). Transport offload can be implemented by a standalone transport

3 offload engine (TOE) or embedded in an Ethernet channel adapter (ECA) that also supports other iwarp accelerations. Fig. 1: iwarp Protocol Calls For TCP/IP Processing To Be Offloaded To A Channel Adapter Card While TOEs alone can eliminate up to 40% of the CPU overhead associated with networking, this solution may not deliver the necessary performance increases. A system with a saturated network may see network performance increases of as much as 1.6x after implementing a TOE solution. However, if increased performance requirements are 10x, then 90% of the networking overhead would need to be eliminated. Moving transport processing to the ECA also helps eliminate a second source of overhead: intermediate TCP/IP protocol stack buffer copies (discussed in detail below). Offloading these buffer copies from system memory to the ECA's memory conserves system memory bandwidth and lowers latency. Eliminating Intermediate Buffer Copies In a traditional software TCP/IP implementation a received packet is direct memory accessed (DMAed) from the Ethernet NIC's receive buffer into server memory, then copied several times by the host CPU before arriving at its ultimate destination in an application buffer. Each buffer copy performed by the CPU requires a read followed by a write (eg, read the OS buffer, write the application buffer), requiring 2x wire rate in memory bandwidth. In the simplified illustration (Fig. 2), without RDMA, the bandwidth load on server memory for an incoming 10-Gbit/s stream would be 5x the line rate, or 50 Gbit/s.

4 Fig. 2: RDMA Enables ECA To Support Direct Memory Reads From/Writes To Application Memory Eliminating Buffer Copies To Intermediate Layers Some relief from buffer copy overhead is achieved through a fully-optimized software networking stack. For example, in some OS the need for a driver buffer-to-os buffer copy has been dramatically reduced. However, two sources of intermediate buffer copies still exist in all mainstream OS TCP/IP implementations. When packets arrive off the wire the OS temporarily stores them, performs TCP processing to check for errors, and determines the destination application, then transfers the packet(s) into the specified application buffer. The only practical way to do this is with a copy operation so the copy from the OS buffer into the application buffer cannot be eliminated. The second source of intermediate buffer copies occurs within the application itself. This copy is often required when the application pre-posts buffers and when the application's transaction protocol integrates control information and data payload within the received byte stream, a characteristic of most popular application transaction protocols. The application examines the byte stream and separates control information from data payload, requiring a scatter copy operation from one application buffer into others. An application can eliminate this buffer copy by not pre-posting application buffers, but this can dramatically reduce performance. DMA techniques have been used for years and were most recently applied to network processing by InfiniBand. Repurposed for IP by the RDMA Consortium, remote DMA (RDMA) and direct data placement (DDP) techniques are now part of the iwarp standard. RDMA embeds information into each packet that describes the application memory buffer with which the packet is associated. This enables payload to be placed directly in the destination application's buffer, even when packets arrive out of order. Data can now move from one server to another without the unnecessary buffer copies traditionally required to "gather" a complete buffer. This is sometimes referred to as the zero-copy model.

5 Together RDMA and DDP eliminate 20% of CPU overhead related to networking and free memory bandwidth attributed to intermediate application buffer copies. Avoiding Application Context Switching The third and somewhat less familiar source of overhead, context switching, contributes significantly to overhead and latency in data center applications. Traditionally, when an application issues commands to the I/O adapter (eg reads from or writes to an I/O device), the commands are transmitted through most layers of the application/os stack. Passing a command from the application to the OS requires a compute-intensive context switch. When executing a context switch, the CPU must save the application context (the information needed to return to the application when finished with the command) in system memory including all of the CPU general-purpose registers, floating point registers, stack pointers, the instruction pointer, and all of the memory management unit (MMU) state associated with the application's memory access rights. Then the OS context is restored by loading a similar set of items for the OS from system memory. The iwarp standards implement an OS bypass (also known as user-level direct access) that enables an application executing in user space to post commands directly to the network adapter. This completely eliminates expensive calls into the operating system (also known as kernel space) dramatically reducing application context switches (Fig. 3). Fig. 3: OS Bypass Eliminates Expensive Calls Into Operating System Tasks typically performed by the OS -- including protection of the I/O device, memory protection checks, and virtual-to-physical memory translation -- are handled by an iwarp-enabled ECA. Such adapters are necessarily more complex than traditional nonaccelerated NICs but can eliminate the final 40% of CPU overhead related to networking. [2]

6 The iwarp Protocols The iwarp protocol layers are formally described in a series of four specifications. They standardize how transport offload, RDMA (zero copy), and OS bypass (user-level direct access) features are implemented. Three of the iwarp specifications -- marker PDU aligned framing for TCP (MPA) direct data placement over reliable transports (DDP) and remote DMA protocol (RDMAP) -- define packet formats and transport protocol for RDMA-over-Ethernet connections. This set of specifications was publicly released by the RDMA Consortium in October The fourth specification (RDMA Protocol Verbs) defines the behavior of RDMA-enabled Ethernet network adapter hardware, firmware, and software as viewed by the host. Similar to InfiniBand Verbs, the iwarp Verbs specification provides OS bypass, enhancements for privileged consumers, better control over registered buffers, and better multi-processor event distribution. Partial implementations of 1G- and 10G- iwarp Ethernet have been in the market for a while, offering some relief from overhead and latency issues. ECAs that fully implement the iwarp standard (TOE, RDMA, OS bypass) have just become available beginning in the first quarter of Revisions to related standards, protocols, and OS networking stacks are paving the way for broad iwarp deployment in the data center. Related Activities Thanks to the efforts of the RDMA Consortium and others, the iwarp standard has been embraced by the Internet community. A brief overview of RDMA-related standards activity follows. 1. RDMA Consortium In addition to the iwarp standard, the RDMA Consortium has released two specifications that enable existing data center applications to take advantage of RDMA transparently: 1. iser (iscsi Extensions for RDMA) is an extension of the iscsi storage networking standard. It maps the iscsi application protocol over the iwarp protocol suite to take advantage of RDMA technology while preserving compatibility with the iscsi infrastructure. A draft of the iser specification was publicly released by the RDMA Consortium in July 2003

7 2. SDP (Sockets Direct Protocol) is a transaction protocol that enables emulation of sockets semantics over RDMA, allowing applications to gain the performance benefits of RDMA without changing application code that relies on sockets today. The RDMA Consortium's SDP specification was publicly released in October Internet Engineering Task Force (IETF) Several IETF working groups are preparing Internet standards related to iwarp. The RDDP working group is using relevant portions of the iwarp standard to enable RDMA capability on TCP/IP networks. Authors of the iscsi standards, the IPS working group is working to extend iscsi with RDMA using relevant portions of the iser standard. Using portions of draft specifications submitted by NetApp and Sun, the nfsv4 working group has been chartered to develop an RDMA-enabled version of NFS which will reduce overhead associated with network file systems. 3. DAT Collaborative The Direct Access Transport (DAT) Collaborative has defined a transport-independent and OS-independent set of APIs that exploit RDMA capabilities such as those present in iwarp. 4. Interconnect Software Consortium (ICSC) The ICSC was founded by Fujitsu, HP, Sun, Network Appliance, and IBM to develop RDMA-enabled extensions to UNIX APIs. It includes a working group focused on RDMA extensions to the UNIX Sockets API, a working group specifying a user-level RDMA-capable API similar to the DAT Collaborative API, and a working group developing a common API that standardizes the RNIC driver-os boundary. 5. Microsoft Microsoft enables deployment of RDMA technology today using Windows Server 2003 with Winsock Direct Protocol (a precursor to SDP). Microsoft has announced a roadmap for major enhancements to their networking stack, as part of their Scalable Networking initiative. TCP and RDMA Chimneys, were designed exclusively to support TCP/IP offload and full iwarp transport offload, respectively. The first step of the roadmap, the TCP Chimney, is supported in Windows Vista, currently in beta test. 6. Open Fabrics OpenFabrics is the former OpenIB alliance. OpenIB chose to re-brand itself with a new name based upon its expanded charter. Under OpenFabric s new charter it will integrate support for iwarp and other industry standard RDMA enabled fabric technologies. As an industry alliance of Infiniband and iwarp open source developers, OpenFabrics is a non-profit member run company, based upon the open source licensing model. Its members are comprised of system vendors, component vendors, software companies and customers.

8 Wrap Up iwarp Ethernet eliminates virtually 100% of the CPU overhead due to networking in G-Ethernet and 10G- Ethernet deployments. It gives data centers the opportunity to reap the performance benefits of a 10x increase in wire speed and lowers the total cost of ownership by leveraging the economies of a pervasive technology. Practical for deployment throughout the data center, iwarp Ethernet offers opportunities for convergence of networking, file and block storage, and clustering applications over a common transport technology and, in some cases, as a migration path to a single, highspeed networking fabric. References [1] Introduction to TCP/IP Offload Engine (TOE), (Version 1.0, April 2002) published by 10 Gigabit Ethernet Alliance [2] Laboratory tests run at NetEffect, Inc About the Author Brian Hausauer is the Chief Architect at NetEffect, Inc Brian holds 15 patents covering I/O interconnect technology and holds a BS in Electrical Engineering from Iowa State University. He can be reached at brianh@neteffect.com