On the Transition to a Low Latency TCP/IP Internet

On the Transition to a Low Latency TCP/IP Internet Bartek Wydrowski and Moshe Zukerman ARC Special Research Centre for Ultra-Broadband Information Networks, EEE Department, The University of Melbourne, Parkville, Vic. 3010, Australia Abstract- Recently, a number of Active Queue Management (AQM) algorithms, such as REM and GREEN, have been proposed which reduce the packet queueing backlog and hence reduce the network s latency close to the propagation delay. This paper uncovers a fundamental problem that a low latency TCP/IP network faces. We call this problem the low latency efficiency collapse. With Explicit Congestion Notification (ECN) still not widely deployed, the main congestion notification remains packet dropping. By reducing the Round Trip Time (RTT) to near the propagation delay, TCP sessions become very aggressive and the packet dropping rate required for congestion notification becomes prohibitively high. In this paper, a solution to this problem is introduced. It is based on inducing latency. When applied to the new AQMs, it limits the packet loss whilst the Internet makes the transition to ECN. We demonstrate by an experiment that the proposed solution improves the efficiency by about 0%. 1. INTRODUCTION The total packet delay on the Internet is the sum of queuing delay (in routers and switches) and propagation delay (in physical medium). Currently due to poorly controlled queues, where Droptail and RED are deployed, queuing delay dominates most round trip times (RTT). A number of papers [1] [4] have envisioned a framework for a network where the network delay can be reduced to near the propagation delay. With a low latency Internet, there is no need for traffic class differentiation within the network (DiffServ) [1]. The framework proposed by [1], [4] and [6] achieve small queuing delays by controlling the congestion notification rate (packet marking or dropping) such that the effective price of bandwidth is controlled so that the arrival rate (consumption) is below the link s capacity (supply). The link price calculation is called the Active queue management (AQM) algorithm and the source algorithm follows a utility function that determines its consumption of bandwidth based on the bandwidth price. The most practicable deployment of this framework in the current Internet requires only the replacement of the AQM algorithm and must interoperate with the widely deployed TCP source algorithm. Another practical constraint is that the AQM must work with both methods of congestion notification: (1) explicit congestion notification (ECN) packet marking, and () packet dropping. Indeed operating with both constraints has been proposed by REM [4] and GREEN [6]. However, it is shown in this paper that deploying the unmodified REM and GREEN algorithms in the current network would result in what we call low latency efficiency collapse. The remainder of the paper is organised as follows. In Chapter we explain the problem of low latency efficiency collapse. In Chapter 3 we provide experimental results on a scenario involving small latency and demonstrate the efficiency collapse. In Chapter 4 the solution is presented.. INSIGHT INTO LOW LATENCY EFFICIENCY COLLAPSE To illustrate the problem faced by low latency AQMs with packet drop congestion notification, the simple and effective model for TCP proposed by S.Floyd [] is used. The maximum throughput of a TCP connection is bounded by the packet size B, round trip time R and packet loss rate p (congestion notification rate): 1.5 B T 3 R p Let us assume that the actual TCP throughput is some constant k times this upper bound. Then, the approximate TCP throughput is: (1) T = () R p

Now, the congestion notification rate p, ( 1 p 0 ) as a function of the R (RTT) and the desired throughput T is: p = (3) T R By (3), given an AQM algorithm that reduces the RTT, as R is reduced, the amount of packet dropping needed increases at rapidly increasing rate. This can be explained intuitively as follows. TCP increases its sending rate upon every positive acknowledgement. As the RTT reduces, these positive acknowledgements return at a faster rate causing TCP to increase its sending rate at a faster rate. As a result, in order for the AQM to maintain the same flow rate, with a smaller round trip time, more packet drops are necessary. In other words, decreased RTT permits a faster rate of positive acknowledgements. In order to control the rate, these must be compensated by a faster rate of negative acknowledgements, which must be signalled by dropping packets for non-ecn sources. As TCP retransmits the lost data, packet loss is not only a problem of losing the data but also loss of efficiency. The expected number of packet transmissions X needed for a successful transmission is given by 1 X =. (4) 1 p. EXPERIMENTAL RESULTS To study low latency efficiency collapse, an experiment was performed using a network of two computers running the LINUX operating system with TCP/SACK and no ECN as shown in Fig 1. The kernel version is.4.3-0. On the source host the Kernel clock was increased from the default value of 100Hz to 1000Hz to improve measurement accuracy. The AQM used was GREEN [6], which is similar to REM in the sense that it produces just enough congestion notification to control the flow so that queuing is reduced, resulting in very low latencies. To emulate a range of RTT values, packets entering the AQM queue are time stamped, and not released until the delay time D has expired. Hence, RTT = D + ( propagation + queueing + processing delays) (7) The actual RTT was measured using the PING program executed at one-second intervals with over a hundred samples for each value of D. Fig. shows the relationship between D and RTT. Because the AQM queue service rate is the bottleneck in the system, D is a good estimator of the actual RTT. However, at small values of D, RTT does not go below 10 ms due to the scheduling latencies of both computers. Figure 1: AQM and Delay Test network By (4) X increases at an increasing rate as packet loss increases. The efficiency E is given by 1/X which, by (4), is E = 1 p. (5) TCP source host TCP AQM with variable delay 0.5 Mbps 100 Mbps TCP destination Host By combining (3) and (5), the efficiency E as a function of the RTT R is given by E = 1. (6) T R Note that the efficiency is highly sensitive to the RTT R. The fundamental result is that efficiency cannot be maintained as the latency is reduced for non-ecn sources (congestion notification by packet dropping). This is a fundamental problem in operating AQMs that aim to reduce queueing in an environment that includes non-ecn sources. This result gives a new fundamental reason for ECN. ECN will enable the operation of an efficient low latency TCP/IP network. Fig. 3 shows how the loss rate increases drastically as RTT is decreased to 0. Operating with no enforced delay creates a congestion notification drop rate of 0.175 which results in about 79% throughput. This is a significant 1% loss of capacity. In reality, the situation is even more grave, as a drop rate of 0.175 results in frequent TCP timeouts which pause transmission altogether. Indeed, with such a high drop rate, it was found that additional TCP sessions would sometimes not even start, as the handshaking packets were lost. Therefore, widespread deployment of low latency AQMs results in efficiency collapse for non-ecn sources.

Round Trip Time (ms) 700 600 500 400 300 00 100 0 Figure : Round Trip Time (RTT) Vs AQM induced delay D 0 00 400 600 Induced Delay D (ms) the RTT R. That is, the complete congestion feedback signal, or price, communicated to the source, is a function of the RTT R as well as of the congestion notification rate p. By () this price is: Pr ice = f ( R, p) = R p (9) The decrease in RTT R caused by the use of the REM like AQMs means that p must be increased to achieve the same price, leading to efficiency collapse when R is low. The throughput, interpreted as a utility function, should be: T = (10) Price Notice that this complete picture has the entire network feedback signal in the price (k and B are independent of the network). 4. SOLUTION TO EFFICIENCY COLLAPSE PROBLEM Packet drop rate (due to congestion notification only ) Figure 3: Congestion Notification Packet Dropping Rate Vs RTT 0. 0.18 0.16 0.14 0.1 0.1 0.08 0.06 0.04 0.0 0 0 00 400 600 Round Trip Time (ms) In literature [7] [8] [9], congestion control analysis has been performed on the basis that the round trip time is a constant. Unfortunately this hides the low latency efficiency collapse. We have demonstrated that it is not enough to only considered the packet-marking rate or dropping rate as a signal of congestion to TCP. In fact, the complete congestion signal that affects TCP also includes In a real network, low latencies will exist in a number of situations: (a) With complete deployment of REM or GREEN across each link on the route. (b) If the network contains other AQMs (RED, Droptail etc) and it is not bottlenecked at the RED or Droptail links. In all of these situations the efficiency collapse problem will manifest itself because the total RTT is low. In previous sections, we discussed the relationship between RTT and the required packet dropping for non-ecn sources and showed that an increased RTT results in a lower packet-dropping rate. Therefore, one solution to reducing the packet dropping is to induce a fixed packet delay to increase the RTT. Since by (7), the minimum RTT time is bounded by the induced delay D, by (3), the loss is bounded to some desired maximum loss l d.. In this case, the D value required to satisfy the maximum bound of l d, for a TCP flow with throughput T bps can be obtained from () to approximated by: D =. (11) T l d Since the source and destination are not known in advance, it is not possible to distribute this enforced delay across the links of the network so that each source and destination receives the minimum possible delay. In this case, a

conservative policy is to enforce the minimum delay at each router. As the minimum delay necessary for reasonable loss figures is in the order of 30 ms for the tested case, this is a good compromise. Figure 3: AQM with controlled delay TS unit: Time stamp Pkt stamp = NOW AQM Figure 4: Notification Signal Splitter ECN? Y N AQM AQM and delay C e C d If NOW < (stamp + D) Requeue Pkt at head Else Transmit Pkt Ironically, the pre-rem like AQM algorithms such as Droptail and RED do not have to introduce the induced latency D because of their own poor queueing performance (large backlog) [6]. This backlog always ensures that the minimal required latency is satisfied. The REM and GREEN like algorithms can deliver lower latency then a path with Droptail or RED as only the minimum delay is induced. The enforced delay D at each queue can be implemented by setting a time-stamp (of value stamp) on the arriving packet with stamp = current time, and not dequeuing the packet until the current time passes D + stamp, as shown in Fig 3. The unfortunate consequence of enforcing the minimum delay D is that ECN enabled flows, which can successfully operate with low latencies, must also suffer the enforced delay. The ideal case, would be to have all sources ECN enabled, and not require the enforced delay D at all. However, the Internet will continue to have a mixture of ECN and non-ecn TCP sources. If the REM/GREEN like AQMs were deployed without the enforced delay, the low latencies and resulting high congestion notification rate would cause a grossly unfair allocation of bandwidth to C ECN enabled flows, as non-ecn flows would suffer many timeouts and retransmission inefficiencies. To solve this problem a new component is introduced, the Notification Signal Splitter, Fig 4. This element maintains a separate queue for ECN enabled packets and non-ecn enabled packets. When the packet arrives, the packet is sent to the appropriate queue based on the ECN Capable Transport (ECT) bit in the IP header. Each queue is controlled by a separate instance of the AQM algorithm. The non-ecn queue induces the minimum delay D on the packets routed through it, and the ECN queue does not have any induced delay. Let us assume that the capacity of the link is C, and the capacities of the ECN and the non-ecn queues are C e and C d, respectively. Then the following must be satisfied: C = C e + C d. (1) If one wishes to apportion the capacity fairly between ECN and non-ecn flows, C e and C d are set proportionally to the number of ECN N e and non-ecn N d sources, respectively, as follow: Ne Ce = C (13) Nd + Ne and Nd Cd = C. (14) N + N d Determining the number of ECN and non-ecn sources requires per-flow information. However, this does not need to be done in real-time, nor does it have to be very accurate. A passive monitoring program can estimate the proportion of ECN and non-ecn sources on, say a monthly basis, and determine the required split. As the Internet migrates to ECN, C e will increase. It is recommended that in fact, ECN sources be given more than their fair share of capacity, to provide extra incentive to migrate, in addition to the low latency. The technique can be further extended by combining the two queues into one, and inducing delay only on non-ecn TCP packets. All other packets, such as ECN TCP, UDP and ICMP packets are forwarded with no induced latency. For each type of packet, ECN TCP, non-ecn TCP, UDP etc, the packet congestion-marking rate can be scaled by a factor which determines that packet s share of the service rate in a fully loaded system. In this way, if ECN TCP packets don t use their share of capacity, other packets can use this capacity. Packet identification is performed by reading the IP header. This extension is a topic of future work. e

5. CONCLUSION This paper has uncovered a significant performance issue in migrating to a low latency Internet. The techniques presented prevent the efficiency collapse that would occur if latencies were brought near the propagation delay. A framework for the transition from packet dropping to ECN has been presented which makes possible the benefits of low latency AQMs without the efficiency collapse problem. ACKNOWLEDGEMENTS This work was financially supported by Agilent Technologies. REFERENCES [1] F. Kelly, "Models for a self-managed Internet", Philosophical Transactions of the Royal Society A358, 000, pp. 335-348. [] S. Floyd and K. Fall. Router mechanisms to support end-to-end congestion control, February 1997, LBL Technical Report. [3] S. Floyd and V. Jacobson Random early detection gateways for congestion avoidance IEEE/ACM Transactions on Networking, vol. 1, number 4, August 1993, pp. 397-413. [4] S. Athuraliya and S.H. Low. "Optimization Flow Control, II: Random Exponential Marking", Submitted for publication, http://www.ee.mu.oz.au/staff/slow/research/, May 000. [5] W. Feng, D. Kandlur, D. Saha and K. Shin, "Stochastic Fair Blue: A Queue Management Algorithm for Enforcing Fairness", in Proc. of INFOCOM 001, April 001. [6] B. Wydrowski and M. Zukerman, GREEN: An Active Queue Management Algorithm, 001, (submitted for publication, available: http://www.ee.mu.oz.au/pgrad/bpw). [7] T.V. Lakshman and Upmanyu Madhow, The performance of TCP/IP for networks with high bandwidth-delay products and random loss, IEEE/ACM Transactions of Networking, vol. 5, number 3, pp. 336-350, June 1997. [8] Matthew Mathis, Jeffrey Semke, Jamshid Mahdavi, and Teunis Ott The macroscopic behaviour of the TCP congestion avoidance algorithm, ACM Computer Communications Review, vol. 7, number 3, July 1997. [9] S. H. Low, A Duality Model of TCP and Queue Management Algorithms, Proceedings of ITC Specialist Seminar on IP Traffic Measurement, Modeling and Management, September 18-0, 000, Monterey, CA (USA)