Implementation of ATM Endpoint Congestion Control Protocols. Prashant R. Chandra, Allan L. Fisher, Corey Kosak and Peter A.

Transcription

1 Implementation of ATM Endpoint Congestion Control Protocols Prashant R. Chandra, Allan L. Fisher, Corey Kosak and Peter A. Steenkiste School of Computer Science and Department of Electrical and Computer Engineering Carnegie Mellon University Pittsburgh, Pennsylvania Abstract We describe the implementation of two ATM congestion control mechanisms, credit-based ow control and ATM Forum-style ABR rate control, on the Net experimental ATM LAN testbed. This document focuses specically on the implementation choices and performance tradeos available to the designer of a network interface card. We conclude that although credit protocols are more ecient and easier to implement on current hardware, ecient rate implementations are also possible with modest hardware support. 1. Introduction In order to minimize data loss due to congestion while achieving high link utilization for bursty trac, switched networks need some means of applying backpressure, or congestion control, to data sources. In the ATM context, two main approaches have been developed for managing congestion in available-bit-rate (ABR) trac. The credit approach operates on a hop-by-hop basis, providing each hop's transmitter with information on receiver buer space. The rate approach is based on estimating an end-to-end rate at which each network terminal may send. In this paper, we briey describe the Net host interface card, the platform for our implementations and experiments. We then summarize in turn the structure and performance of several credit implementations and a rate implementation, running at both OC-3 and OC-12 speeds. Along the way, we point out some of the design options and cost/performance tradeos available to the network interface designer. In particular, we consider implementations that require varying levels of hardware support. We conclude that although credit protocols are more ecient and easier to implement on current hardware, ecient rate implementations are also possible with modest hardware support. 2. Net Adapter The architecture of the Net adapter is shown in Figure 1. The adapter contains an ATM ASIC, designed by Intel Architecture Labs, that handles all of the ATM AAL5 segmentation and reassembly in hardware. The ASIC also has a programmable microcontroller which can schedule connections based on available credit. Also associated with the adapter is a i960 coprocessor board that is intended to be used for experimentation. The i960, ASIC microcontroller and the host processor can access the SRAM, which stores connection and segmentation/reassembly state, and can communicate with each other. The adapter plugs into a standard PCI bus, and two dierent versions operate at OC-3 and OC-12 rates. A detailed discussion of the design of the Net adapter can be found in a separate paper [3]. This research was sponsored by the Defense Advanced Research Projects Agency under contract number F C The authors can be reached at fprashant, alf, kosak, prsg@cs.cmu.edu.

2 PCI ATM ASIC SRAM UTOPIA OC3 / OC12 Bridge DRAM i960 Figure 1: Net adapter architecture. VC 1 Source 1 Switch 1 Switch 2 Dest. 1 VC 2 Source 2 per VC Buffer Dest. 2 N1 N2 N3 Figure 2: -based ow control concept. 3. -based Flow Control Of the several approaches proposed for ABR ow control, credit-based ow control was the rst to be proposed and built. The most important goal of ABR ow control is to prevent data loss due to congestion. As shown in Figure 2, the credit-based approach achieves this in a straightforward manner by sending explicit hop-by-hop feedback to the sender, in the form of \credit" messages, regarding the buer space available at the receiver. The sender can transmit data only when it has sucient credit. The credit based approach requires each connection to have its own separate buer at the switch and if the switches serve connections in a round-robin fashion, fair sharing is achieved. There are several variants of credit-based ow control [2, 4, 5]; the particular algorithm described in this document is the N123 algorithm [4]. In this scheme each connection's buer is divided into three zones: N1, N2 and N3. The N1 area is proportional to the credit round trip time (which includes the physical round trip time plus delays due to credit processing at the sender and the receiver) in the feedback loop. The N2 parameter controls the frequency of credit messages and allows a tradeo between credit cell overhead and the timeliness of feedback. An N2 value of 10 means that a credit cell is sent upstream for every 10 cells of data forwarded to the downstream node. The N3 area prevents data underow so that the connection can sustain its targeted bandwidth. The credit cell reects the number of empty cell buers in the N2 + N3 area (also called N23).

3 3.1. Implementations The exible architecture of the Net adapter allows several dierent implementations of credit-based ow control. The implementations can be classied along two dimensions: (a) based on where the credit cell processing is implemented, the implementations can be classied as host-based or i960-based; (b) based on the granularity of scheduling quantum the implementations can be classied as coarse grain or ne grain. Note that the implementations that use the i960 could be implemented in hardware in a production system. Hence we have four dierent implementations of credit-based ow control that are described in greater detail below. Host Coarse Grain Implementation In this case the host processor handles credit cell processing in software. The scheduling granularity is on the order of packets, i.e. the host does not schedule data for transmission until an entire packet's worth of credit is available. The great advantage of this implementation is that it is a purely software implementation and hence can be implemented on any adapter without requiring special purpose hardware. The downside to this is that the buer allocation at the switch (N23) has to be at least as large as the maximum packet size. Host Fine Grain Implementation In this case the host processor handles credit cell processing in software as before. However the scheduling granularity is on the order of cells. The ASIC microcontroller in the adapter is used to schedule cells based on availability of credit. The advantage of the ne grain approach is that the per connection cell buers at the switch need be no larger than the feedback delay to prevent underow. i960 Coarse Grain Implementation In this case the processing of incoming credit cells is o-loaded to the i960 coprocessor. The i960 polls for incoming credit cells and continuously updates a credit table that is shared with the host processor. The advantage of this implementation is that o-loading the credit cell processing to the i960 saves host CPU cycles which can lead to increased network bandwidth for applications. Another advantage is the reduced latency in handling incoming credit cells and hence faster response to feedback. As before this implementation also requires at least one MTU worth of cell buers per connection at the switch. i960 Fine Grain Implementation The i960 ne grain implementation oers the best performance of all implementations at the cost of extra hardware (i960 or custom hardware). Fine grain scheduling allows the per-connection buers at the switches to be small, and the i960 credit cell processing minimizes processing latency for incoming credit cells. The feedback loop is closer to the network in this case and hence dynamic response is much faster Performance Comparison Source 166 MHz Pentium PC OC-3/12 Net Switch Dest 166 MHz Pentium PC Figure 3: Experimental setup for comparing dierent credit implementations In this section we compare the performance of the dierent implementations and study the tradeos involved. The experimental setup is as shown in Figure 3. The hosts involved are two Pentium 166 MHz PCs equipped with the Triton PCI chipset and running NetBSD 1.1. In this experiment a single UDP stream is established between the two PCs and the throughput is measured using the application ttcp, sending 8192-byte packets. The rst experiment studies the credit cell processing overhead at the host. In this case the output link from the switch runs at full speed, with N1 set to 44 and N3 set to 200. At OC-3 speeds, as shown in Figure 4, host implementations suer from the burden of credit cell processing, especially at low values of N2. For N2 greater than 10, the throughputs of all four implementations are similar. The i960 outperforms the

4 Throughput (Mbps) Figure 4: OC-3 UDP throughput as no a FC function of N2. Host CG host slightly; the gap reects host processor capacity that the i960 Host frees FG up for UDP/IP protocol processing. The i960 coarse and ne grain implementations are nearly indistinguishable i960 CG in performance; the host coarse grain implementation is slightly faster than the ne grain implementation i960 FG due to its lower overhead. Overall, this experiment shows that credit-based ow control can be implemented in software at the host with small performance penalty, at least when credit processing can be done every ten cells or less frequently N2 value Figure 5: OC-12 UDP throughput as a function of N2. A similar graph of throughput at OC-12 is shown in Figure 5. In this case, all of the implementations run slightly faster than before, although the maximum throughput is limited to 128 Mbps due to protocol processing overhead, notably data copying. The one other dierence from the OC-3 case is that the bestperforming implementation, by a small amount, is the i960 coarse-grain algorithm. This is apparently due to ATM ASIC resource conicts between scheduling and interaction with the i960 in the ne-grain case. Again, except for very small values of N2, ow control overhead is dominated by UDP/IP protocol processing The second experiment studies the throughput of the UDP stream passing through a throttled link, running at a 48 Mbps ATM payload rate, as a function of the number of cell buers allocated to the connection at the switch. Because the speed of the output link is the limiting factor, the OC-3 and OC-12 adapters have essentially the same performance; the OC-3 data is plotted. As shown in Figure 6, the coarse grain implementations need at least one packet's worth of switch buers (about 185 cells). The ne grain implementations require signicantly fewer cell buers per connection at the switch.

5 46 Throughput (Mbps) 45 Figure 6: Throughput of the congested stream as a function of switch buers allocated to the stream (N23). Host CG Host FG The host coarse grain implementation provides good performance i960 CG with a pure software implementation at the cost of requiring at least one MTU worth of cell buers per connection at the switch. However for 44 i960 FG a LAN environment, this buer requirement is not large. The ne grain implementations overcome this limitation by scheduling on the granularity of cells at the cost of extra hardware. The i960 implementations can save host CPU cycles by o-loading the credit cell processing. Although the results are reported only for the N123 credit scheme, the tradeos indicated are valid for implementations of some other credit-based 43 schemes as well. 0 The N scheme 100[4] is an incremental 150 extension 200 of the N123 scheme and has identical end-system behavior as the N123 scheme. The N23 scheme, which has been formalized and released as the Quantum Maximum credit Flow Control Specication (QFC) [2], reduces buer requirements by eliminating the need for the N1 buer area. This is achieved by tracking the number of cells for each VC that are in ight over a given link, thus eliminating the possibility of overow. This change requires additional state to be maintained at the endsystems (per connection counters, etc.) and denes several credit transaction messages. Ecient implementation of this protocol would require support for ne-grain tracking of cells in ight. Furthermore, credit messages need to be correlated at a ne time granularity with the appropriate counters, making coarse grain implementations less attractive. For a ne grain implementation on our current hardware, QFC message processing could be implemented on the i960 while the ASIC microcontroller maintained cell counters per connection and a cell counter for the outgoing link. 4. ABR Rate Control The ATM Forum has specied a mechanism [1] for exchange and use of rate information along the path followed by a VC in an ATM network. The source of each VC periodically generates a request for bandwidth in the form of a resource management (RM) cell, which traverses the VC's path to its destination and returns via the reverse path. Along the way, switches and the destination may change the cell's contents to reect the bandwidth sustainable at each point. On the host, this mechanism has four parts: Generation of outgoing RM cells: Every 32nd cell (typically) sent on a VC is an RM cell. Interpretation of incoming RM cells: Every outgoing RM cell generated, barring losses in the network, will eventually cause a reected RM cell to be received. This cell may contain an explicit rate allowed by the network, a congestion indication, or both. A source may also receive a congestion notication cell spontaneously generated within the network. The source must adjust its transmission rate accordingly. Reecting far-end RM cells: An endpoint will also receive RM cells for connections for which it is a destination. It must update explicit rate and congestion elds, if necessary, and direct the cell back

6 to its origin. Scheduling of cells on ABR VCs: Given the rates at which each VC is allowed to transmit, a host scheduler must multiplex them over the outgoing link. On average, each of the RM cell processing operations (generation, interpretation, reecting) will happen once every 32 cell times. At OC-3 rates, this corresponds to some operation being performed every 30 microseconds. Transmit scheduling decisions need to be made at an even ner resolution. A key benet of the rate control approach is that switches are not required to do such scheduling, although doing so may help; the load falls instead on the sending terminal. If the host processor is expected to do useful work, these tasks will need to be performed by some other agent, e.g., a coprocessor or the adapter itself Implementation In our implementation, the transmit scheduling algorithm is implemented on the ASIC's microcontroller. While the various RM cell processing operations could conceivably also be implemented on the microcontroller, severe code space constraints caused us to implement them on the i960 coprocessor instead. In the following sections we discuss the design and implementation of the transmit scheduling algorithm and each of the RM cell processing operations. Generation of Outgoing RM Cells The RM cell format contains several ags and three rate elds, protected against bit error by a 10 bit CRC. Although the CRC computation would be straightforward in custom hardware, a software implementation on the i960 would perform poorly. Luckily, the only eld that varies between RM cells is the current cell rate (CCR) eld. The other two are set to constant values: the explicit rate (ER) eld is set to the link peak cell rate, and the minimum cell rate (MCR) eld is set to zero. Hence the CRC for the RM cell depends only on the CCR eld. Furthermore, since the ATM Forum rate format 1 can describe only 16K+1 dierent values, the i960 implementation is able to precompute the CRCs corresponding to all possible CCRs and store them in a table of modest size. Our current implementation, however, does not include this capability. The scheduling of outgoing RM cells is easily incorporated into the scheduler described below. Interpretation of Incoming RM Cells An incoming RM cell aects the rate of its corresponding VC by either triggering a multiplicative decrease in rate (in the case of congestion), or an additive increase (in the case of no congestion). Further, it imposes an upper bound, via the Explicit Rate (ER) eld, on the VC's rate. Because arithmetic on values expressed in the rate format is somewhat awkward, best performance is again achieved by precomputing and tabulating appropriate result values for all possible rates. The CRC can be checked by comparing it against a tabulated value in the manner described above. Similar processing occurs when various timeouts, on the order of 100 milliseconds, are exceeded. These timeouts can be eciently implemented on the i960 coprocessor. Reecting Far-End RM Cells At the destination, a forward RM cell is modied and reected toward its source. The modication consists of setting the cell's direction bit and indicating any rate constraints imposed by the destination. If the destination is capable of receiving cells at link rate, no such constraint is necessary, as it will be imposed by the edge switch. If it is necessary or desirable to use the ABR mechanism to indicate dynamic congestion at the destination, the simplest mechanism is to set the RM cell's congestion bit. The RM cell's new CRC can be looked up in a table. Scheduling of Cells on ABR VCs A simple implementation of a rate scheduler would maintain a priority queue of VCs, sorted by their next valid transmit time. Each time a VC were scheduled, it would be requeued with a priority value of the current time plus an interval whose value is (link rate/vc rate). Unfortunately, both the maintenance of a priority queue and the arithmetic complexity involved make this implementation unattractive. Worse, it has the consequence that under conditions where allowed cell rates oversubscribe the outgoing link, it slows ag. 1 Rates are expressed as 2 e (1 + m=512)nz, where e is a 5 bit exponent, m is a 9 bit mantissa, and nz is a one bit nonzero

7 down all connections in proportion, including those that are bottlenecked elsewhere in the network. We argue that it is preferable to adopt the max-min fairness criterion widely used for the rest of the network: we dene a fair share rate such that all VCs with allowed rates below the fair share receive their allowed rates, all VCs with higher allowed rates receive the fair share rate, and all link bandwidth is used as long as some VC is not receiving its allowed rate. This results in contention for resources at the link leaving the host being treated in the same way that contention at links leaving switches is (ideally) treated. Our ABR scheduler is based on this notion of fairness and on a single simplifying assumption: we assume that all VCs have an allowed rate of at least some minimum value. Slower VCs can be handled by host software if necessary. This allows us to represent the rates as a small, xed number of intervals, where the interval is the approximate reciprocal of the rate. Converting rates to intervals is straightforward; best performance is achieved by table lookup. Using intervals has two benets. First, it allows us to do scheduling arithmetic using only addition on a simple xed-point representation (our representation comprises a 12 bit integer part and a 4 bit fractional part). Second, it allows us to represent the future in which cells can be scheduled as a small wraparound table. For reasons of implementation convenience, we use intervals of up to 2047 cells. For an OC-3 link, this lets us represent payload rates between 66.4 kbps (a bit higher than an ISDN B channel) and 136 Mbps, the maximum rate for ATM over OC-3. OC-12 rates are four times larger. Note that this scheme introduces a worst-case rate error of 3%; if this is not tolerable in a particular network, a solution is to round each rate downward, with the net eect of shifting bandwidth slightly from bottlenecked to un-bottlenecked VCs. The algorithm uses three data structures: a round-robin queue, a high-priority queue, and a scheduling timetable. The timetable is large enough to accommodate the largest possible interval. The general idea is that VCs with allowed rates in excess of the fair share will be scheduled from the round-robin queue, while VCs with lower allowed rates will be scheduled from the timetable. The high-priority queue exists to ensure that low-rate VCs are serviced as rapidly as possible, to avoid falling below their allowed rates. The scheduling algorithm lets VCs shift dynamically between the two populations, as the number of active VCs, their allowed rates, and the fair share change. The fair share itself is never explicitly represented, but arises implicitly. At each time step, the algorithm operates as follows: 1. Append the VCs from the current timetable entry to the high priority queue. 2. If the high priority queue is not empty, remove the VC at the head of the queue, schedule it, and add it to the end of the round-robin queue. 3. Otherwise: (a) Remove the VC at the head of the round-robin queue. (b) If it is eligible to go, schedule it and return it to the end of the queue. (c) Otherwise, add it to the timetable at its earliest eligible time, and repeat from step 3a. Eligibility is determined by a computation involving the last time the VC was scheduled, the VC's interval, and the Generic Cell Rate Algorithm (GCRA) Limit parameter specied by the network. VCs enter and drop out as they receive or exhaust data to transmit. Note that although the loop in step 3 may be performed a large number of times, hence potentially missing a time slot, this will be a rare occurrence. Also note that the total number of steps involving a VC between successive scheduling events is bounded, so the total scheduling eort expended is proportional to the sum of the number of time steps taken and the total number of cells scheduled Performance and Complexity The code for the scheduler achieves high throughput through two additional techniques. First, it allows more than one cell at a time be scheduled, to decrease scheduling overhead. In protocol terms, this requires the GCRA Limit parameter to be greater than or equal to the Interval parameter. Second, it precomputes the next VC to send, so the scheduling computation is pipelined with the action of the segmentation process. Figure 7 shows the throughput of raw AAL5 transfers, using dierent burst sizes, for OC-3 and OC-12 links. The OC-3 link can run at line rate (136 Mbps) with a 2-cell burst; at OC-12, using a burst of 8 cells, the scheduler achieves 541 Mbps vs. a line payload rate of 544 Mbps.

8 Throughput (Mbps) 400 Figure 7: AAL5 throughput vs. ABR burst size. 300 OC-3 The code to implement this scheduler consumes 133 words ofoc-12 microcode, out of the 192 available on the Net ATM ASIC. We estimate that adding RM cell processing would inate this number by a factor 200 of two to three. (In comparison, a microcode implementation of sender-side credit, including receiving credit cells and managing their buers, consumes about 190 words). 100 The high performance of the scheduler persists for large numbers of VCs. We have run a series of tests designed to put maximum load on the scheduler: given N VCs, one has an interval of 1, and the rest have 0 intervals of N + 1=16. This causes the slow VCs to cycle through the longest path of the algorithm. Overall 0 throughput remains 2 nearly4 constant: VCs, running 8 on an OC-3 link with a burst size of 2 cells, achieve an aggregate throughput of 135 Mbps, close to the full link rate of about 136 Mbps. 100 VCs (the most we Burst size (cells) could manage due to a temporary hardware limit), on an OC-12 link with a burst size of 8 cells, achieve an aggregate throughput of 451 Mbps compared to a line rate of 544 Mbps; they reach 533 Mbps with a burst size of 16, and 538 Mbps with a burst size of Conclusions We have presented a variety of implementation techniques for credit- and rate-based ATM congestion control protocols. At OC-3 rates, it is possible to implement FCVC-style credit ow control on a host processor with modest overhead, although the coarse-grain implementation style that achieves greatest eciency requires an increase in buer resources on the switch to which the host is attached. In contrast, the ABR rate control scheme requires more computation than is economical on a host processor. A rate scheduler is somewhat more complex than a credit scheduler: the main dierence is that the ABR protocol requires the issuance of outgoing RM cells whose timing is decoupled from host notions of packet boundaries. However, modest hardware support, as modeled by our mixed microcode/i960 implementation, can solve this problem. References [1] F. Bonomi and K. Fendick. The rate-based ow control framework for the available bit rate ATM service. IEEE Network Magazine, 9(2):25{39, March/April [2] Mike Gaddis and Walker Kelt (eds.), \Quantum Flow Control", Version 2.0, FCC-SPEC-95-1, Flow Control Consortium, July [3] Corey Kosak, David Eckhardt, Todd Mummert, Peter Steenkiste and Allan Fisher, \Host and Adapter Buer Management in the Net ATM Host Interface", Proceedings of the 20th Annual Conference on Local Computer Networks, IEEE Computer Society, Minneapolis, September 1995.

9 [4] H. T. Kung and Alan Chapman, \The FCVC (Flow Controlled Virtual Channels) Proposal for ATM Networks", Proceedings of International Conference on Network Protocols, October 1993, San Francisco, California. [5] C. Ozveren, R. Simcoe, and G. Varghese. Reliable and ecient hop-by-hop ow control. In Proceedings of the SIGCOMM '94 Symposium on Communications Architectures and Protocols, pages 89{100, University College, London, UK, October ACM.