UC Santa Cruz UC Santa Cruz Electronic Theses and Dissertations

Transcription

1 UC Santa Cruz UC Santa Cruz Electronic Theses and Dissertations Title Re-Thinking the TCP Congestion Control Design in Network Simulators Permalink Author Hume, Allison Publication Date Peer reviewed Thesis/dissertation escholarship.org Powered by the California Digital Library University of California

2 UNIVERSITY OF CALIFORNIA SANTA CRUZ RE-THINKING THE TCP CONGESTION CONTROL DESIGN IN NETWORK SIMULATORS A thesis submitted in partial satisfaction of the requirements for the degree of MASTER OF SCIENCE in COMPUTER SCIENCE by Allison Hume June 2016 The Thesis of Allison Hume is approved: Professor Katia Obraczka, Chair Professor Luca de Alfaro Professor Carlos Maltzahn Tyrus Miller Vice Provost and Dean of Graduate Studies

3 Copyright c by Allison Hume 2016

4 Table of Contents List of Figures List of Tables Abstract Acknowledgments iv vi vii viii 1 Introduction Motivation Background TCP Congestion Control Survey of TCP Congestion Control Algorithms History of TCP Design in ns Results Motivating Results: TCP Inigo Validating ns-3 Version Proposed Design Design Validation TCP New Reno TCP Westwood Implementing a New Algorithm: TCP Inigo Discussion Related Work Conclusions and Future Work Bibliography 45 iii

5 List of Figures 1.1 A 600 second trace of the congestion window (in bytes) of TCP Westwood. Experiment from Gangadhar et al[10] run on ns-3 release version 3.24 (left) and version 3.25 (right) Block diagram showing the implementation of TCP congestion control for ns-3 version Block diagram showing the implementation of TCP congestion control for ns-3 version Log of congestion window and slow start threshold (both in Segments) for TCP Inigo, run on Mininet. Plot courtesy of Andrew Shewmaker Log of congestion window and slow start threshold (both in Segments) for TCP Reno, run on Mininet. Plot courtesy of Andrew Shewmaker Log of congestion window and slow start threshold (both in Segments) for TCP Westwood, run on Mininet. Plot courtesy of Andrew Shewmaker Bottleneck queue length (in packets) for TCP Inigo, Reno, and Westwood, run on Mininet. Plot courtesy of Andrew Shewmaker Congestion window and slow start threshold (both in Segments) for TCP Inigo, NewReno, and Westwood, run on NS Bottleneck queue length (in packets) for TCP Inigo, NewReno, and Westwood, run on Mininet Congestion window (in Segments) for TCP Inigo and NewReno, run on NS3, with RTT of 2ms, 4ms, and 8ms, clockwise from top left Congestion window and slow start threshold for New Reno in ns (top row) and 3.25 (bottom row), with packet loss rates set to 10 3 and with fast recovery enabled (left column) and disabled (right column). 40 second trace taken from a 600 second experiment Congestion window and slow start threshold for New Reno in ns (top row) and 3.25 (bottom row), with packet loss rates set to 0 (left column) and 10 3 (right column). Full trace from a 1000 second experiment Congestion window and slow start threshold for New Reno in ns (top row) and 3.25 (bottom row), with packet loss rates set to 0 (left column) and 10 3 (right column). Full trace from a 100 second experiment iv

6 2.11 Proposed design for a modular and extensible TCP congestion control implementation Comparison between simulation results obtained with our design and implementation of New Reno and ns s New Reno. Left-side graph can be compared with Figure 2.8 s lower left graph, and right-side graph can be compared with Figure 2.10 s lower left graph (Duplicate of Figure 1.1): A 600 second trace of the congestion window (in bytes) of TCP Westwood. Experiment from Gangadhar et al[10] run on ns-3 release version 3.24 (left) and version 3.25 (right) A 600 second trace of the congestion window of TCP Westwood. Experiment from Gangadhar et al [10] run on ns after design implementation (left) and after artificially increasing the bandwidth estimate by a factor of 40 (right) Proposed design for a modular and extensible TCP congestion control implementation, applied to TCP Inigo, a new experimental TCP congestion control algorithm v

7 List of Tables 2.1 Summary of experiment parameters Summary of Design Changes vi

8 Abstract Re-Thinking the TCP Congestion Control Design in Network Simulators by Allison Hume In contrast to live hardware testbeds, which are evaluated largely on performance, network simulators are primarily used to test and validate new protocols and algorithms. As such, a network simulator should be designed to be modular, extensible, re-usable, and robust. This work examines the design of TCP congestion control in the ns-3 network simulator and presents a new design, not specific to ns-3, that will be more modular and extensible. We also demonstrate that the implementation in the current ns-3 release (version 3.25) is not able to duplicate results presented in previous literature. These findings, along with the improved design, are the main contributions of this paper. To validate our design, we show how its modular approach can be used to implement existing TCP congestion control variants, e.g., TCP New Reno and TCP Westwood, as well as new ones, e.g., TCP Inigo. vii

9 Acknowledgments Thank you to Andrew Shewmaker for the initial research direction, information about TCP refactoring in NS3, and help with interpreting results. viii

10 Chapter 1 Introduction 1.1 Motivation Network simulators are an essential component of the network protocol and application development cycle. They complement live hardware testbeds as they enable experimentation under a wide variety of networking environments and conditions. They also allow the reproduction of experiments, which is a fundamental scientific methodology principle. Because network simulators are used primarily for testing and validation, the primary goal of simulators does not need to be performance, as it is in hardware. Instead, the main goal should be to provide a robust environment that can be easily modified and extended so that new protocols can be added to the existing code base with minimal effort. As such, we believe that modularity, extensibility, re-usability, and robustness should be the main focus of network simulator code design. One active area of research in the networking community is introducing and comparing different TCP congestion control algorithms. Network simulation platforms have 1

11 Figure 1.1: A 600 second trace of the congestion window (in bytes) of TCP Westwood. Experiment from Gangadhar et al[10] run on ns-3 release version 3.24 (left) and version 3.25 (right). played a crucial role in advancing the TCP congestion control state-of-the-art and ns-3, and its previous incarnation, ns-2, have a long track record as essential tools for TCP congestion control research [1], [11]. ns-3 has been evolving, however, and there has not been enough of an effort to continue to validate and understand the simulator itself. The work in this paper was motivated by our experience in working on a new variant of TCP congestion control, TCP Inigo [27]. We were planning to use ns-3 as the simulation platform to validate TCP Inigo and to complement experiments run on a test-bed. It became clear, however, that the existing TCP implementation in ns-3, while potentially sufficient for comparing aggregate metrics such as throughput, or for supporting higher level protocols, was not robust enough to support the detailed analysis that is necessary to understand and compare congestion control algorithms. To illustrate the previous claim, Figure 1.1 shows a trace of the congestion window in bytes (see Section II for a review of congestion control in TCP) for TCP Westwood [16] using an experiment found in Gangadhar et al[10]. The left plot shows the experiment run 2

12 on ns-3 release 3.24 while the plot on the right shows the same experiment run on ns-3 release These results show that the implementation of TCP Westwood has changed dramatically between the two implementations. How can researchers trust simulators to compare new versions of TCP when established algorithms are not consistent between releases? The work we present in this paper has been motivated by these findings, and by the difficulties encountered in adding a new congestion control algorithm to ns-3. The main contribution of this work is a new design of the TCP congestion control implementation for network simulators that targets modularity, extensibility, re-usability, and robustness. The design will be presented after a brief background on TCP congestion control and the history of TCP code design in the ns-3 simulator. This work also presents results that question the existing TCP implementation in ns-3, lending weight to the argument that a comprehensive design evaluation, as well as a formal validation process, is very much needed. 1.2 Background TCP Congestion Control The Transmission Control Protocol (TCP) is probably the most widely used endto-end Internet protocol, supporting Internet services such as the Web, , and file transfer. TCP s main goal is to provide to applications reliable, in-order data delivery atop of IP s unreliable, best-effort service. To this end, TCP provides error control, flow control, and congestion control which are implemented based on mechanisms such as acknowledgments (ACKs), re-transmissions, sequence numbers, and timers. TCP segments 3

13 include sequence numbers, which are included in ACKs sent by the receiver to signify to the sender that data was received. Re-transmission occurs when an ACK for a certain range of sequence numbers has not been received within a specified time period. Because TCP s control functions use the same mechanisms, their operation is tightly coupled. However, in this paper, we focus our attention primarily on TCP congestion control. In its early days, TCP congestion control was defined by two algorithms: slow start and congestion avoidance. Slow start is invoked after a TCP connection has been established or after a timeout event. It increases the TCP sender s congestion window cwnd from one maximum segment size (MSS) exponentially until it reaches the slow start threshold ssth. The congestion avoidance algorithm increases cwnd linearly until a timeout occurs. At that point, slow start is invoked. TCP congestion control has continuously evolved over the years. Fast retransmit and fast recovery were among the early modifications to the original TCP congestion control mechanism. The fast retransmit algorithm is used to detect and repair loss to try to prevent timeouts. It employs incoming duplicate ACKs as an indication of loss and responds by re-transmitting the potentially lost segment right away. Fast recovery is used to adjust cwnd until a non-duplicate ACK arrives. RFC 5681 [3] specifies the basic rules defining these four algorithms, but many variations have been defined and used over the years. TCP variants such as TCP Reno [15], TCP New Reno [14], and TCP Westwood [16] refer to specific implementations of those four algorithms. They potentially include additional components such as Selective Acknowledgments (SACK), which make it possible for the receiver to acknowledge scattered blocks of data, not just a continuous block. Newer 4

14 variants often build upon older variants. For example, the TCP Westwood [16] congestion control algorithm is a modification of TCP Reno s congestion control that changes only how cwnd and ssth are modified after a congestion event, which is triggered by n duplicate ACKs or a timeout. While TCP Reno halves the congestion window after three duplicate ACKs, Westwood uses a bandwidth estimator to try to adjust the congestion window based on the currently available bandwidth [16]. Both Reno s and Westwood s congestion control algorithms, however, have the same implementation of slow start and congestion avoidance Survey of TCP Congestion Control Algorithms This section gives a summary of some of the more well known variants of TCP Congestion Control to show the wide range of algorithms that a simulator may have to support. It should be noted how many of these algorithms are minor modifications of an existing algorithm. This observation gives weight to the argument that a TCP congestion control implementation in a simulator should prioritize modularity and extensibility to best support the research goals of the community. Early Algorithms The earliest versions of congestion control, TCP Tahoe and TCP Reno were named after the BSD releases in which they appeared: 4.3BSD-Tahoe and 4.3BSD-Reno. The main improvements from Tahoe to Reno are described in Jacobson [15]. Both algorithms specify that slow start begins after a timeout, at the beginning of slow start the congestion window is one Maximum Segment Size (MSS), and during slow start the congestion window increases by one for each ACK. Tahoe and Reno behave differently after three duplicate 5

15 ACKs, however. After three duplicate ACKs, Tahoe sets the slow start threshold to half the current congestion window, and begins slow start, essentially treating three duplicate ACKs in the same way as a timeout. Reno, instead includes separate algorithm for how to behave after three duplicate ACKs. This algorithm is called Fast Recovery, and causes both the slow start threshold and congestion window to be set to half the current congestion window and all re-transmitted data to be acknowledged before the congestion window is increased again. Both Tahoe and Reno have a Fast Retransmit algorithm which re-transmits all unacknowledged data after three duplicate ACKs, as opposed to waiting for a timeout. Variations of TCP Reno Reno uses an acknowledgment number field that contains a cumulative acknowledgment, indicating the TCP receiver has received all of the data up to the byte indicated in the field [8]. This achieves poor performance when multiple packets are dropped from a window of data, because Reno waits for a re-transmission timer expiration before initiating re-transmission. TCP NewReno [14] and SACK TCP [17] were both developed to attempt to improve this performance issue encountered in TCP Reno. Selective Acknowledgement (SACK) [17] is an option in TCP that specifies that specific segments be acknowledged, as opposed to cumulative acknowledgments, with which a sender can only learn about a single lost segment per round trip time. Selective acknowledgments can be either positive or negative, meaning that the receiver can acknowledge segments that were or were not received. This prevents the receiver from having to choose between preemptively re-transmitting segments that may have been lost, which can waste bandwidth, or waiting to learn of lost segments, which can cause increased latency. TCP 6

16 with SACK is not necessarily tied with any specific congestion control algorithm, although it was originally presented as a modification of TCP Reno, but is often used on wireless and satellite links, which tend to have higher error rates, to prevent packet loss from being mistakenly interpreted as congestion. TCP NewReno attempts to solve the same performance issue seen in Reno without using SACK. TCP NewReno [14] only differs from TCP Reno in the Fast Recovery specification. In Reno, partial ACKs end Fast Recovery. Instead, NewReno attempts to keep the transmission window full by sending a new packet for each ACK, or partial ACK that is at least one MSS, received during Fast Recovery. NewReno remains in Fast Recovery until all of the data that was outstanding at the beginning of Fast Recovery has been acknowledged. Therefore, when multiple packets are lost from a single window, NewReno re-transmits one lost packet per RTT instead of waiting for a re-transmission timeout. Comparisons of TCP NewReno and TCP with SACK found that NewReno does not perform as well as TCP with SACK when a large number of packets are dropped from a window of data. It is, however, comparable for smaller loss rates [17]. TCP NewReno is one of the most common forms of TCP in use today. TCP Vegas [5] is also very similar to TCP Reno, but is not as widely used as TCP NewReno. Vegas extends Reno s re-transmission mechanism by using the system clock to record the RTT for each segment. Vegas then uses the RTT estimate to re-transmit based on only one duplicate ACK if the RTT for that segment is larger than the timeout value. Vegas also preemptively re-transmits the first or second segment after a re-transmission if the RTT is larger than the timeout value, to avoid waiting for a duplicate ACK. Although 7

17 Vegas is not widely used, similar RTT estimates have been used in many later algorithms, and Vegas has been seen to be a very smooth version of TCP [5], [11]. High Latency or High BDP Links The original TCP congestion control algorithms do not perform well for networks with links with very high propagation delays. The growth of the congestion window in all of those algorithms happens as a response to ACK arrivals, the frequency of which is determined by the RTT, and therefore the propagation delay. Long RTTs reduce the congestion window growth rate and cause throughput degradation, as well as unfair sharing of bandwidth resources in situations with links of mixed propagation delay [6]. TCP Hybla [6] is a version of TCP designed to solve this challenge, referred to as the RTT disparity problem. Hybla requires a similar RTT estimate to the estimate introduced in Vegas, but the estimate is performed for both the high latency link, and a lower latency link with which the high latency link should be achieving equivalent performance. Congestion window calculations are then scaled by the ratio of those two estimates, so that the congestion window in the high latency link is no longer penalized by the long RTT. A similar challenge is presented by high bandwidth-delay product (BDP) links. The BDP of a link describes the number of packets in flight when the bandwidth is fully utilized. This number is ideally also the size of the congestion window. Again, because traditional congestion control algorithms grow the congestion window at a rate of one per RTT, it can take too long for the congestion window to reach its maximum size if the maximum size is very large. This is an especially large problem for short-lived flows. Many congestion control algorithms have been developed in response to this prob- 8

18 lem, because of how important high BDP links are in modern networking. Some of the most well known algorithms are BIC-TCP [33], Cubic [13], and Compound [28]. BIC-TCP [33] uses a binary search algorithm to adjust the congestion window size between the max window size that has been seen, and the last window size where there was not loss for one RTT period (the perceived min window size). This allows the congestion window to change quickly when it is far from the true capacity of the network, and change slowly when it is close to the available capacity. BIC-TCP has been seen to work well with High BDP links and to be very stable, and has therefore been used in the Linux Kernel [13]. Cubic [13] is a slight modification of BIC-TCP that uses a cubic function to simplify the window adjust algorithm presented in BIC-TCP. The window growth in Cubic depends only on the time between congestion events, and is not dependent on RTT. This allows better RTT-fairness, similar to what is provided by TCP Hybla. Compound TCP [28] is a modification of TCP Vegas in which the congestion avoidance phase tracks a second moving window, the delay-based window. The sending window is then taken as the minimum of the congestion window and delay-based window. The delay-based window is scalable and can rapidly increase the sending rate when the network is under-utilized. Like BIC-TCP and Cubic, Compound TCP offers both bandwidth scalability and RTT fairness. Comparisons of Cubic and Compound find that Cubic outperforms Compound on wired links with reverse traffic, but that their performance is comparable on wireless links [1]. Although TCP Westwood [16] was originally developed as a wireless congestion control algorithm, it has also been regarded as a successful algorithm for links with high 9

19 BDP [13]. Westwood relies on a bandwidth estimator to set the congestion window and slow start threshold to the correct values for the network, rather than simply reducing them at every congestion event. The bandwidth estimation is maintained with every received ACK, and used to set the congestion window and slow start thresholds after a congestion event. TCP Westwood+ [12] is a slight modification of Westwood designed to improve the correctness of the bandwidth estimation algorithm in the presence of ACK compression. Comparisons of Westwood(+), NewReno, and Vegas show that Westwood(+) improves utilization of low-congestion wireless links as well as fairness [11], [16]. Data Center Algorithms Lastly, data centers are one of the most important networking scenarios. Data centers often have to accommodate diverse applications with low latency and high throughput requirements. Traditional TCP protocols, when used in data centers, cause flows with high bandwidth requirements to build up queues and harm the performance of latency sensitive flows [2]. Data Center TCP (DCTCP) [2] was developed to address this issue. DCTPC uses Explicit Congestion Notification (ECN), which is an option that allows switches in data centers to set certain bits in packet headers when congestion is observed by the switch. Congestion in this sense is measured by buffer occupancy exceeding a fixed small threshold. The DCTCP source reduces the congestion window according to the fraction of packets that are marked as having encountered congestion in the network. This allows DCTCP to achieve high burst tolerance, low latency, and high throughput without requiring switches to contain large buffers [2]. 10

20 Deadline Aware Datacenter TCP (D 2 TCP) [29] was developed to support a specific, important class of data center applications, called Online Data Intensive applications. These applications are real-time, which imposes a 300ms latency constraint. In addition, these applications often use tree-based algorithms, which creates bursts of children to parent traffic. DCTCP is not aware of deadlines, causing it to not support this application very well. D 2 TCP uses ECN feedback in conjunction with deadlines to scale the congestion window. This allows D 2 TCP to reduce the fraction of missed deadlines relative to DCTCP by 75%. D 3 [31] is a full control protocol designed to support data center applications. Like D 2 TCP, D 3 was also designed for the purpose of supporting deadline-sensitive applications. D 3, however, is not compatible with legacy TCP, making it unlikely to become a widely adopted protocol. Incast Congestion Control for TCP (ICTCP) [32] is also a specialized TCP congestion control algorithm, but supports applications such as MapReduce and Search that have a many-to-one traffic pattern. Traditional versions of TCP do not work well for that many-to-one traffic pattern on high-bandwidth, low latency networks, such as data center networks. When many synchronized servers simultaneously send data to one receiver, a type of congestion can occur called incast congestion. The performance of this type of workload is determined by the slowest TCP connection. Previous approaches to mitigating incast congestion have attempted to reduce timeout values, but ICTCP adjusts the TCP receive window proactively before packet loss occurs. ICTCP is able to achieve almost zero timeouts and high goodput for TCP incast. 11

21 Multipath TCP (MPTCP) [25] has also been proposed for use in data centers. MPTCP establishes multiple sub-flows on different paths between the same pair of endpoints for a single TCP connection. This provides many benefits for a data center network such as improved bandwidth utilization and fairness. The MPTCP congestion control algorithm moves traffic off more congested paths to less congested ones, which benefits not just the MPTCP flow, but any competing flows as well. TCP Inigo [27] was developed to achieve results similar to those achieved by DCTCP, without requiring ECN. ECN is not always available in networking hardware and it can therefore be problematic to rely on a protocol that assumes that ECN (or any specific hardware support) exists. TCP Inigo is a protocol that does not require any specific hardware support or switch configuration. Instead, Inigo uses RTT measurements in the sender to calculate the congestion ratio, and adjust the congestion window accordingly. Preliminary results for TCP Inigo are promising in terms of both network utilization and fairness [27] History of TCP Design in ns-3 The previous discussion helps to illuminate some of the challenges that arise when attempting to implement TCP congestion control in a way that is both efficient and extensible. In particular, extensibility in a simulation platform is quite critical as new versions of TCP congestion control have been frequently proposed and added to network simulators for validation and testing. If the TCP implementation in Linux favors efficiency over extensibility that may be acceptable since new TCP variants are not added to the Linux kernel very often. In a network simulator, however, lack of extensibility goes directly against the goals 12

22 tcp-westwood(+) tcp-tahoe tcp-newreno Defines all TCP functionality tcp-socket-base Minimal Shared Code Base Figure 1.2: Block diagram showing the implementation of TCP congestion control for ns-3 version of the software. This section discusses the recent history of TCP congestion control design choices in the ns-3 network simulator [20], which motivate our new design as described in Section 2.3. We should point out that even though our work was motivated by ns-3 s TCP congestion control implementation, our design is applicable to network simulation platforms in general. As shown in Figure 1.2, the second most recent ns-3 release, version 3.24, is organized as follows: (1) it has a module called tcp-socket-base that contains the code base shared by all versions of TCP implemented in the simulator, and (2) there is one main module for each TCP variant containing most of the code required for that specific congestion control algorithm [21]. The tcp-socket-base module contains very little code. It has a function named ReceivedAck, called upon the receipt of a new acknowledgment, which serves mostly to call the NewAck and DupAck functions, which are defined for each TCP variant in its respective module. NewAck and DupAck do not just contain congestion 13

23 tcp-congestion-ops: Defines basic congestion control functionality: PktsAcked() GetSsThresh() IncreaseWindow() tcp-new-version tcp-hybla tcp-westwood(+) Also includes TcpNewReno Refactored Shared Code Base Includes more common functionality: All ACKs Error/Flow Control Fast Retransmit and Fast Recovery Figure 1.3: Block diagram showing the implementation of TCP congestion control for ns-3 version control functionality, but include all TCP functionality that should be invoked upon receipt of a new or duplicate acknowledgment, respectively. Consequently, although the code is organized by congestion control algorithm, a significant portion of code in each module is in no way specific to the corresponding congestion control algorithm. The main downsides of this design are the significant code duplication and the high overhead required to maintain and extend it, e.g., add a new congestion control algorithm. Recently, ns-3 s TCP implementation has undergone a refactoring effort. As shown in Figure 1.3, in the refactored code the majority of the TCP code is shared and only specific congestion control functions need to be implemented to define a new version of congestion control [24]. The result of this effort is the recently released ns-3 version 3.25 [22]. In this version, the ReceivedAck function in tcp-socket-base contains much more shared functionality and calls more specific, well-defined congestion control functions such as PktsAcked, GetSsThresh, and IncreaseWindow. These functions are defined in the 14

24 congestion-ops module. TCP congestion control variants are each implemented in a separate module using the congestion-ops module as their base class. The implementation for TCP New Reno, however, exists in congestion-ops because New Reno is the default TCP congestion control algorithm used by ns-3. The refactored design in ns-3 version 3.25 is more extensible than the previous design, although great care must be taken to ensure that all shared code is truly generic. As will be discussed further in Section 2.2, that is not currently the case. Another criticism of this design is that, since TCP New Reno is used as the default implementation, the implementation of other congestion control algorithms is not always completely clear. It is not obvious, for example, that the TCP Westwood implementation actually uses the SlowStart function defined for New Reno, since SlowStart is not one of the three functions specified in the base class. We argue that, although the current design is moving in the correct direction, additional improvements are needed. 15

25 Chapter 2 Results 2.1 Motivating Results: TCP Inigo The original goal of this research was to analyze a new implementation of TCP congestion control, TCP Inigo [27]. The starting point for this work was to validate previous results, achieved using the Mininet simulator, with the more powerful NS3 simulator. It was hoped that, after previous results were validated, more interesting experiments would be run using the additional functionality provided by NS3. However, the validation step exposed significant issues with the NS3 simulator, changing the direction of the research to be focused on understanding, correcting, and validating the simulator itself. This section summarizes progress made in simulating TCP Inigo and shows the results that caused the research direction to shift. Figures 2.1, 2.2, 2.3, and 2.4 show the results obtained on Mininet, against which we hoped to validate our implementation of TCP Inigo on NS3. Figures 2.1, 2.2, and 2.3 show the size of the congestion window and slow start threshold in segments for TCP 16

26 Figure 2.1: Log of congestion window and slow start threshold (both in Segments) for TCP Inigo, run on Mininet. Plot courtesy of Andrew Shewmaker. Inigo, Reno, and Westwood, respectively. It should be noted that, because Mininet uses the Linux TCP implementation, the TCP Reno implementation is equivalent to the NS3 TCP NewReno implementation. Figure 2.4 shows the size of the bottleneck queue for all three TCP versions. These results were obtained on the Mininet simulator by Andrew Shewmaker, the primary author of TCP Inigo. The experiment from which these graphs were produced involved one source and one sink node connected by a gateway. The access link bandwidth was 500Mbps, bottleneck bandwidth was 100Mbps, and the bottleneck link 17

27 Figure 2.2: Log of congestion window and slow start threshold (both in Segments) for TCP Reno, run on Mininet. Plot courtesy of Andrew Shewmaker. had a queue size of 100 packets. The total Round Trip Time (RTT) was 4ms, the Maximum Transmission Unit (MTU) was 1500 Bytes, and the experiment was run for 100s. It is clear from examining these graphs that the congestion window computations for TCP Reno and Westwood are not realistic in Mininet. The experiment parameters should cause congestion in the network more frequently than the one congestion window drop that is seen. From these plots it is clear that the bottleneck bandwidth is not being enforced. Since Mininet is such a simple network simulator, it is not altogether surprising 18

28 Figure 2.3: Log of congestion window and slow start threshold (both in Segments) for TCP Westwood, run on Mininet. Plot courtesy of Andrew Shewmaker. that there would be approximations and inaccuracies. However, it is clear that comparing Inigo to Reno and Westwood on Mininet would not be a productive comparison. It was hoped that NS3 would be more robust, due to its complexity, age, and heavy usage in the networking community. Figure 2.5 shows the congestion window and slow start threshold in segments for TCP Inigo, Reno, and Westwood for the same experiment run on NS3. The version of NS3 used was the development repository between releases 3.24 and 3.25, with TCP refactoring 19

29 Figure 2.4: Bottleneck queue length (in packets) for TCP Inigo, Reno, and Westwood, run on Mininet. Plot courtesy of Andrew Shewmaker. patches applied. This code base is very similar to NS3 release These plots can be compared to Figures 2.1, 2.2, and 2.3, respectively (N.B. that the plots for NS3 show segment count while the plots for Mininet show log of segment count), and it is clear that the two simulators produce extraordinarily different results. Similarly, Figure 2.6 shows the queue lengths for the first two seconds of the experiment for all three TCP versions on NS3. The first two seconds are shown for clarity, and the same pattern repeats for the rest of the experiment. Comparing these plots to Figure 2.4 also demonstrates extreme differences between the output of Mininet and NS3. While attempting to adjust the Mininet and NS3 experiments to try to isolate the sources of the differences, it was noted that the NS3 results were surprisingly reactive 20

30 Figure 2.5: Congestion window and slow start threshold (both in Segments) for TCP Inigo, NewReno, and Westwood, run on NS3. to certain parameter settings. Figure 2.7 shows three plots of congestion window for TCP Inigo and TCP NewReno. These plots show the same experiment described previously, only with three different RTT values: 2ms, 4ms, and 8ms. At 8ms it is clear that something causes the experiment to no longer create congestion, although the traffic was setup to fill the network, independent of RTT. A very similar phenomenon was also observed when changing the MTU from 1500 Byes to 64 kilobytes. These results brought us to the conclusion that it was impossible to validate our implementation of TCP Inigo in NS3, or to use NS3 to make any comparisons between TCP 21

31 Figure 2.6: Bottleneck queue length (in packets) for TCP Inigo, NewReno, and Westwood, run on Mininet. Inigo and existing TCP congestion control algorithms, with the behavior of the simulator as it was. We therefore began a process of attempting to validate and correct the NS3 TCP implementation. We used previously published results containing experiments run on hardware, in NS2, or in earlier versions of NS3, as baselines to understand the behavior of the current NS3 release. The results of this validation attempt, as well as our proposals for an improved TCP congestion control implementation for simulators, are contained in the remainder of this work. 22

32 Figure 2.7: Congestion window (in Segments) for TCP Inigo and NewReno, run on NS3, with RTT of 2ms, 4ms, and 8ms, clockwise from top left. 2.2 Validating ns-3 Version 3.25 Before presenting our proposed design, we further motivate the need for a new design by presenting experiments we conducted to validate the current state of the simulator. To this end, we tried to reproduce results reported in the literature. In particular, the experiments presented here are taken from previous work by Gangadhar et al[10], Casoni et al[7], and Grieco et al[11]. The simulation parameters and the values we used for these experiments are summarized in Table 2.1, which also includes the reference to the work where the experiment was originally carried out (Original Source column) and the network 23

33 simulator used to run the original experiments (Simulator Used column). A? in that column indicates that the simulator version was not specified in the original paper, so we used the publication date and the simulator s version release schedule to infer the version used in the original experiments. The tcp-variants-comparison row in Table 2.1 refers to a version of the experiment developed by Gangadhar et al that is provided as part of ns-3 release (in the examples/tcp directory). All experiments involve a source and a sink connected by a gateway with pointto-point links, and traffic sent from the source to the sink using the BulkSendHelper traffic generator, which continuously sends traffic to fill the network. The link with the lower bandwidth is referred to as Bottleneck Link, and the other as Access Link. Their bandwidth and propagation delay are shown in the fourth and fifth columns of Table 2.1. Where applicable, packet losses are simulated using a uniform distribution defined by the rate shown in the Loss Rate column in Table 2.1. Different experiments show different degrees of variation between the current ns release and original results obtained from either previous versions of ns-3, ns-2, or hardware. Figure 2.8 shows an experiment originally from Casoni et al [7], in which results from ns-3 version 3.22 s TCP New Reno were compared to Linux. Their results showed cwnd and ssth from ns-3 both very close in scale and pattern to the values obtained from Linux; however, cwnd showed spikes that were attributed to fast retransmit. The top left plot in Figure 2.8 shows the same experiment repeated on ns , with results very similar to those in the original publication. We found, however, that the spikes are due to fast recovery: the increase of cwnd by one segment for every duplicate ACK after the three 24

34 Figure 2.8: Congestion window and slow start threshold for New Reno in ns (top row) and 3.25 (bottom row), with packet loss rates set to 10 3 and with fast recovery enabled (left column) and disabled (right column). 40 second trace taken from a 600 second experiment. duplicate ACKs that initiate fast recovery (section 3.2 of RFC 2582[9]). The top right plot in Figure 2.8 shows the same experiment with that portion of fast recovery commented out. This plot more closely matches the Linux results presented in Casoni et al [7]. For New Reno, this difference is not necessarily an error. The ns-3 code very closely follows the RFC, however the Linux TCP implementation handles fast recovery differently. The Linux implementation does not follow RFC 2582 [9] exactly and does not adjust cwnd for every ACK during fast recovery [7] [26]. 25

35 The important point here is that, in order to disable fast recovery, its code had to be tracked down and commented out within the shared code base. The current design of TCP in ns-3 allows no option for changing or disabling the fast retransmit and fast recovery algorithms. In different scenarios you may want different implementations of the algorithm, or, as in this case, simply to turn one off. Commenting out sections of code is a very error prone method of achieving that result. For TCP Westwood, however, this design does result in an error. The Westwood algorithm specifies that the bandwidth estimation should be applied not just after a timeout, but after n duplicate ACKs as well [16]. The New Reno fast recovery algorithm specifies that after three duplicate ACKs the congestion window is halved. Since this New Reno code exists in the shared code base, after three duplicate ACKs the congestion window is halved for TCP Westwood as well, and the bandwidth estimation does not occur as it should, according to the Westwood specification. Having one fast retransmit and fast recovery implementation in the shared code base is therefore not a viable design for an extensible TCP implementation as it is not guaranteed that any two congestion control algorithms will require the same behavior for those components. The bottom two plots in Figure 2.8 show the same set of experiments, but run on ns , with the refactored TCP. It is important to reiterate that, although the intention of this refactoring was to make the TCP congestion control design extensible, it was still necessary to manually find and comment out the relevant lines of code corresponding to fast recovery in the shared code base. Although there is not a drastic difference between the results for the two versions of ns-3, the new version does respond differently to the removal 26

36 Figure 2.9: Congestion window and slow start threshold for New Reno in ns (top row) and 3.25 (bottom row), with packet loss rates set to 0 (left column) and 10 3 (right column). Full trace from a 1000 second experiment. of the fast recovery algorithm. Looking closely at the bottom right plot in Figure 2.8 it is clear that slow start is initiated much more frequently than in the other plots. In fact, it is quite concerning to see slow start initiated almost 20 times in a span of 40 seconds. This issue could be due to a number of factors but the current design of the TCP congestion control implementation makes it difficult to isolate the source of the change. Because of the extent to which the shared code has been increased, it is hard to track the relationship between the modular component and the shared code base. In this case, it is easy to 27

37 Figure 2.10: Congestion window and slow start threshold for New Reno in ns (top row) and 3.25 (bottom row), with packet loss rates set to 0 (left column) and 10 3 (right column). Full trace from a 100 second experiment. validate the implementation of the SlowStart function, but difficult to see exactly under what conditions SlowStart will be executed. This example further motivates the need for a comprehensive design evaluation. Figure 2.9 shows the results of another experiment comparing ns-3 versions 3.24 and This experiment was first presented in Grieco et al [11] and seen also in Abdeljaouad et al [1]. This experiment originally included one forward flow with 10 back-flows turning on and off but, due to the results discovered while attempting duplication, the 28

38 experiment is presented here without the back-flows. Again, the top row shows results for version 3.24 and the bottom row for version For this experiment, the left column is performed with a packet loss rate of 0 and the right column with a rate of The differences in the plots with an error rate of zero (left column) are quite significant. The top left plot very closely matches the results previously obtained by Grieco et al [11] and Abdeljaouad et al [1], which used the ns-2 network simulator [19]. It appears, looking at the bottom left plot, that in ns-3 version 3.25, without packet loss, congestion events only occur once the congestion window has grown to a much larger size than in ns-3 version Since the experiments have the exact same setup, it is clear that some parameter is not enforced properly in version This is not visible in the equivalent experiments with a non-zero packet loss rate (right column) because even a very small loss rate creates enough perceived congestion events to mask this issue. The difference in these plots speaks to an insufficient validation effort between release versions of ns-3. If such simple experiments are not repeatable, how can the implementation be used to support further research? Figure 2.10 shows behavior similar to the plots in Figure 2.9. The plots in this figure were generated using the tcp-variants-comparison experiment that exists in the examples/tcp directory in every ns-3 release (based on the experiment presented in Gangadhar et al [10]). Again, the top row shows the experiments run on release 3.24, the bottom on release 3.25, the left column shows packet loss rates of 0, and the right column of Again, the difference in the shape of cwnd for 0 loss rate shows a lack of repeatability even between an example provided in each ns-3 release. 29

39 2.3 Proposed Design The examples presented in the previous section motivate the need for re-designing a TCP congestion control implementation for network simulators in order to increase modularity, and consequently robustness and extensibility. Our design is based on the following goals: Extensibility: The primary goal of network simulators is to allow researchers to test and validate new protocols or variants/extensions to existing ones. TCP is a prime example: as one of the most widely used Internet protocols, TCP has attracted considerable attention from the networking research and practitioner communities. In particular, the TCP congestion control algorithms have evolved significantly over the years, with new versions often appearing as minor modifications of existing algorithms. Consequently, it must be straightforward to add new TCP congestion control variants to network simulators existing TCP code base in order to facilitate testing and validation. Modularity: TCP congestion control has evolved significantly over time and currently has many variants, each of which consists of a set of algorithms. Among those algorithms, some may be unique and some shared with other variants. As such, modular design and implementation are critical, especially in network simulation platforms. Modularity will allow different congestion control variants to share implementations of common functions that have been well tested. It will also facilitate experimenting and testing new congestion control versions since modules that implement different functions can be easily enabled/disabled. In addition, it will greatly reduce the overhead necessary for implementing new congestion control mechanisms, as they are often small modifications to one 30

40 tcp-congestion-ops: Defines basic congestion control functionality: PktsAcked() GetSsThresh() IncreaseWindow() Add: FastRetransmit(), FastRecovery() Refactored Shared Code Base Includes common functionality: All ACKs, Error/Flow Control Will not include: Fast Retransmit and Fast Recovery tcp-getssthresh tcp-pktsacked tcp-increasewindow tcp-fastretransmit tcp-fastrecovery tcp-new-version tcp-hybla tcp-westwood(+) Figure 2.11: Proposed design for a modular and extensible TCP congestion control implementation. component of existing algorithms. While our design can be generalized to any network simulator, we used ns-3 to showcase and test our implementation. As such, we will continue to use ns-3 s TCP congestion control implementation as the basis for our design changes. The main modification to the shared code found in tcp-socket-base is that it no longer includes fast retransmit and fast recovery algorithms. The tcp-congestion-ops module remains the base class for all congestion control implementations with the following changes: (1) it no longer contains the implementation of New Reno, which now is implemented in a separate, dedicated module, and (2) it now includes FastRetransmit and FastRecovery function signatures. Each function in the base class will have a corresponding module in which all its variant implementations will reside. Lastly, each specific congestion control algorithm will have a module that defines the selection of its components. 31

41 Figure 2.11 summarizes the proposed design. We believe that this design achieves modularity and extensibility, ensuring implementations are clear and easy to understand. Specifically, having a module for each algorithm, not just each congestion control variant, allows developers to easily find and follow implementations that already exist for one specific piece of functionality, without having to root through all congestion control modules and the shared code base. Table 2.2 summarizes the changes made between ns-3 releases 3.24 and 3.25, as well as our design changes. Each column in the table corresponds to a specific module and the rows describe the state of that module for the three designs. The table clearly shows the proposed design s increased modularity. The extensibility of the proposed design can be inferred from the ease with which a new algorithm module can be created, given an existing set of component modules. Next, Section 2.4 validates our design against ns and Section 2.5 showcases the extensibility of the design by using it as a basis to implement a new TCP congestion control variant. 2.4 Design Validation TCP New Reno Ideally we would like to fully validate our proposed design. Unfortunately, as demonstrated previously, the starting point for our design is not entirely correct. In this section we show that we are able to obtain the same results for TCP New Reno using our design and implementation when compared to the results obtained with ns The 32

42 Figure 2.12: Comparison between simulation results obtained with our design and implementation of New Reno and ns s New Reno. Left-side graph can be compared with Figure 2.8 s lower left graph, and right-side graph can be compared with Figure 2.10 s lower left graph. second part of this section shows the same results for TCP Westwood and contains a deep dive into an example illustrating our attempts at achieving consistent results between ns and The goal is to show how high a barrier poor validation between releases can be for researchers trying to validate their results. Finally, in Section 2.5, we return to our original motivation, i.e., implementing a new TCP congestion control algorithm in ns-3, to show how our proposed design simplifies that effort. The plot on the left side of Figure 2.12 shows results obtained by running the experiment by Casoni et al[7] using our implementation of TCP New Reno. We observe that it matches quite well Figure 2.8 s lower left graph, which shows results from the same experiment run using ns The plot on the right side of Figure 2.12 shows results from the tcp-variants-comparison experiment available as part as the ns release. It matches quite well Figure 2.10 s lower left graph which was obtained by running the same experiment using ns s TCP NewReno. 33

43 Figure 2.13: (Duplicate of Figure 1.1): A 600 second trace of the congestion window (in bytes) of TCP Westwood. Experiment from Gangadhar et al[10] run on ns-3 release version 3.24 (left) and version 3.25 (right). Figure 2.14: A 600 second trace of the congestion window of TCP Westwood. Experiment from Gangadhar et al [10] run on ns after design implementation (left) and after artificially increasing the bandwidth estimate by a factor of 40 (right) TCP Westwood For TCP Westwood, we used the experiment from Gangadhar et al[10]. Figure 2.13 (a duplication of Figure 1.1, presented again for clarity) shows the original outcome of this experiment, in which we observe considerable differences between ns-3 releases 3.24 and As discussed in Section 2.2, one clear bug with the TCP Westwood implementation 34