Report on Transport Protocols over Mismatched-rate Layer-1 Circuits with 802.3x Flow Control

Transcription

1 Report on Transport Protocols over Mismatched-rate Layer-1 Circuits with 82.3x Flow Control Helali Bhuiyan, Mark McGinley, Tao Li, Malathi Veeraraghavan University of Virginia {helali, mem5qf, taoli, Observation point 1Gbps Buffer 155Mbps Sender Receiver Fig. 2: Scenario Setup Fig. 1: Generic Gigabit Ethernet LIM I. INTRODUCTION This report describes our work detailing the operation and performance of TCP-based transport protocols (Reno, BIC, and CTCP) on Layer-1 circuits, in which a Gigabit Ethernet port on an end host has been directly mapped to a SONET circuit of lower rate (e.g., a 155 Mbps OC3). The switch uses IEEE 82.3x flow control to prevent the sender from overwhelming the switch s buffer. Current equipment such as Multiservice Provisioning Platforms (MSPPs) can act as virtual-circuit/circuit gateways (G/Ws) that offers both Ethernet and SONET line interface modules (LIMs) with a SONET-based switch fabric. An Ethernet LIM has Ethernet ports on the front end (the left edge of Stage One in Fig. 1) and SONET (virtual) ports on the back end (the right edge of Stage Three in Fig. 1) through which the card is connected to the backplane. The backplane thus brings SONET signals from all interface cards, both Ethernet and SONET, to the switch fabric card where the signals are crossconnected according to programmed configuration information. Generic Framing Procedure (GFP) and virtual concatenation (VCAT) [1] technologies are implemented in the Ethernet interface cards to map Ethernet frames into SONET frames. For a matched-rate circuit, a 1 Gbps (GbE) port would be cross-connected to a 7-OC3c SONET signal. Configuration of a mismatched-rate circuit is allowed, e.g., a GbE port can be mapped to an OC3c SONET signal. Regardless of whether the circuit is a matched- or mismatched-rate circuit, since the whole port is mapped to a SONET signal, these type of circuits are referred to as port-mapped or Layer-1 circuits. In a mismatched-rate circuit, Ethernet frames may arrive at an Ethernet port at a rate faster than the rate of the SONET circuit to which this port is crossconnected. We consider the case when Pause frames are generated to stop the sender, in accordance with 82.3x flow control. Using VCAT, we create a virtual-concatenation group (VCG) on the back-end of the Ethernet LIM. This groups multiple timeslots to form a VCG (on the LIMs back end), onto which front-end Ethernet traffic from a port is mapped. In Fig. 1, the VCGs are realized in Stage Two. For the SN16, these VCGs have a 2MB buffer. The Pause ON Threshold is configured to 1 MB and the Pause OFF Threshold is configured to 2 KB. The rate mismatch between a GbE port and an OC3 circuit causes this VCG buffer to fill. By using Pause frames, the switch prevents this buffer from completely filling, which would cause packet loss. We will discuss the impact of Pause frames on TCP in the following section. II. TCP BEHAVIOR IN CIRCUITS In Fig. 2, we have a sender directly connected to a switch through GbE interface. We set up an OC3 (155 Mbps) circuit from the switch to the receiver host. The round-trip time () between the sender and the receiver is 8 ms. We have the following parameters in this setup, Standard packet size of 15 bytes Time to emit a single 15 byte packet by a 1GbE NIC is 12us Time to forward a single packet at OC3 rate is 8us The BDP of the link is 1 packets TCP ssthresh is set to 64KB or 42 packets When a TCP connection is set up between the sender and the receiver, the TCP sender starts with the slow-start phase, where congestion window () is initially set to 1 (or 2) packet(s). During slow start, the is increased by one packet after receiving each ACK back from the receiver. Hence, increases in an exponential rate. Fig. 3 explains the slow-start phase. The top line of Fig. 3 shows the packets transmitted by the sender over time. The

2 Serving: 1 Queued: None Serving: 2 Queued: 3 Queued: None Serving: 5 Queued: 6, 7 Serving: 11 Queued: 12, 13, 14, Data frame sent ACK seen by Sender 1 12us trigger ACK Two packets outstanding Four packets outstanding 8 us ACK2 ACK3 ACK4 ACK5 ACK6 ACK7 Time (a) Time = 4ms (b) Time = 6ms Fig. 3: Explanation of Slow Start 5+1/5 bottom line shows the ACK packets received by the sender over time. Each packet transmitted by the sender is numbered is the the order they transmitted. We number the ACK packets as ACKn, where n is the data packet being ACKed. Initially, is set to 1 and packet 1 is sent at time. ACK1 arrives at the sender after one. This causes the to increase to 2, at which point the TCP sender sends packets 2 and 3 back-to-back. Since the transmission time for a 15 byte packet is 8us at the OC3 rate, packet 2 and 3 arrives at the receiver 8us apart. Therefore, ACK2 and ACK3 are also received 8us apart at the sender. After receiving ACK2, is increased to 3 and TCP sender sends two new packets, 4 and 5, back-to-back (as packet 3 is still outstanding). While packet 4 is being forwarded by the switch, packet 5 will be queued inside the switch buffer. Similarly, when ACK3 arrives after 8us, grows to 4 and packets 6 and 7 are sent. As packet 5 is being served by the switch at this time, packet 6 and 7 will be queued. A similar analysis can be done for packet 8 through 15. This behavior, that x if the size of the congestion window is x, 2 packets will be queued inside the switch, is explained by Savage et. al in [2]. During slow start, every successfully acknowledged packet increases the windowsize by one packet. Thus, the sender transmits two packets for every new acknowledgment. Since the acknowledgments are generated at the bottleneck rate, this implies that the sender is bursting data at twice the bottleneck rate, leading to the formation of a queue at the bottleneck link [3]. When the sender window is W 2 packets, the sender gets W 2 acknowledgements with a spacing of one time unit between consecutive acknowledgments. In response, the sender transmits two packets for each acknowledgment or a total of W packets in W 2 time units. Since the bottleneck can forward only W 2 packets in W 2 time units, the other half of the packets are queued. Therefore, during slow start, a sender window size of W builds up a queue of size W 2 at the bottleneck router. Note that, during one period, if the congestion window is less than or equal to the BDP in packets, then packets will not build up in the buffer. That is, at the start of a new period, all the previous packets will have been transmitted out of the buffer. However, when the congestion window is greater than the BDP in packets, the amount of packets in the buffer builds up according to the above description / (c) Time = 8ms (d) Time = 8ms + 8us 1 99 (e) Time = 12ms Fig. 4: Congestion Avoidance. < BDP Since the ssthresh is set to 42 packets, TCP will enter into the congestion avoidance phase once grows past 42. During congestion avoidance, increases by 1/ packets as each ACK packet is received. In other words, is increased by one packet after each. To understand TCP s behavior in congestion avoidance phase, we use Figures 4 and 5. In Fig. 4, we explain the case when is below the BDP of the link. Fig. 5 explains the case when reaches the BDP and exceeds the BDP. In both figures, packets are numbered in the order they are sent. At time, packet 1 is sent. Then we show states of the system at specific times, shown below each figure. Larger packets above the line are data packets sent by the sender, and shorter packets below the line are ACK packets. ACK packets are numbered to the packet which they acknowledge. In Fig. 4, we assume that the is set to 5 packets, which is exactly half of the BDP, and all packets and ACKs from previous s are processed. After 4ms, packet 1 arrives at the receiver (Fig. 4a). Since the number of outstanding packets in Fig. 4b is 5, the sender sits idle. ACK1 for the first packet arrives at the sender after 8ms (Fig. 4c), which allows the sender to send packet. Similarly, the following ACKs permit the sender to send more packets. Once ACK5 is received, is increased to exactly. This allows the sender to send packet 1 and 11 back-to-back. Packet 11 gets queued inside the switch buffer while packet 1 is being forwarded. Hence, the buffer grows to 1 momentarily, but drops to again once packet 11 is forwarded. Therefore, the switch buffer momentarily queues one packet only when grows to the next integer. This behavior continues until

3 (a) Time = 8ms (b) Time = 12ms (c) Time = 16ms (d) Time = 16ms + 8us (e) Time = 24ms Fig. 5: Congestion Avoidance. = BDP grows to BDP of the link. In Fig. 5, we explain what happens when the is equal to BDP, which is 1 packets. When is equal or larger than the BDP, TCP sends packets at a streaming rate, or continuously without idle periods waiting for ACKs. We assume that packet 1 is the first packet when is 1 and sent at time. Figures 5a and 5b show the system state after 8 and 12ms. Once ACK1 arrives after 8ms (Fig. 5c), the grows to the next integer 11. Hence, the sender can send packets 199 and 2 back-to-back, but packet 2 will be queued. Similarly, ACK11 allows the sender to send packet 21, which is queued as packet 2 is being forwarded by the switch (Fig 5d). Likewise, each incoming ACK will allow the sender, being in a streaming state, to send one more packet, which will be queued. So, there will be always one packet queued inside the switch buffer until grows to 12, at which time there will be always 2 packets queued inside the switch buffer (Fig. 5e). In other words, during congestion avoidance phase, there will be always ( BDP ) number of packets queued inside the switch buffer. As ACK packets are being received by the sender, the congestion window keeps growing. Eventually, for long flows, there will be packet loss once congestion window grows beyond the capacity of the end-to-end path between the end hosts. The capacity of an end-to-end path consists of two components, BDP of the path Buffer space inside the bottleneck switch TABLE I: Packet Loss Switch Buffer = 1MB, Circuit Rate = 155 Mbps Switch Circuit BDP Buffer Rate for loss (Pkts) (Mbps) (ms) (Pkts) (Pkts) The BDP of a path is simply the product of the round-trip time and the circuit rate C, or RT T C. The buffer space B inside a switch is not easy to find. In most cases, this value is proprietary. If B is known, the value at which packet loss will occur can be found by the following equation, loss > BDP + B or, loss RT T C > B (1) Based on Equation 1, Table I shows the congestion window values at which packet loss will occur for a 1MB switch buffer and typical wide-area round-trip times. Having switch buffer space B smaller than the BDP of the end-to-end path does not impact the achievable maximum throughput, as long as packet losses do not occur. A switch buffer will cause low throughput when it is lower than the difference between the congestion window and BDP of the end-to-end path (Equation 1). Following a packet loss, TCP reduces the congestion window to half. A reduced congestion window that drops below the BDP of the path reduces the throughput. In order to prevent packet loss from buffer overflow, Ethernet switches and NICs use IEEE 82.3x Ethernet Flow Control or Link Level Flow Control (LLFC). A switch sends Pause frames back to the sender (NIC or another switch) when the buffer occupancy reaches a threshold. Each switch buffer maintains two thresholds, a Pause ON threshold and a Pause OFF threshold. A switch buffer sends Pause frames if the Pause On threshold is crossed. Within each Pause frame, a time field specifies the time period during which the sender should not transmit any packets. At the end of the specified time period, or after receiving a Pause frame with zero in the time field, the sender resumes normal transmission of packets. A switch sends the Pause frame with zero in the time field when the buffer size falls below the Pause OFF threshold. Let us look at the impact of Pause frames on TCP. An application writes the data into the TCP send buffer by calling the write() system call. The TCP layer assembles a packet when data is available in the send buffer. Each packet is pushed down to the IP layer for transmission. The IP layer enqueues each packet in an output queue (qdisc) associated with the appropriate NIC. The size of the qdisc can be modified by assigning a value to the txqueuelen variable associated with each NIC card. If the output queue is full, the attempt to enqueue a packet generates a local-congestion event, which is propagated upward to the TCP layer. The TCP congestion-control algorithm then enters into the Congestion

4 Window Reduced (CWR) state, and reduces the congestion window by one every other ACK (known as rate halving). The only scenario in which TCP will continue to enqueue packets while the NIC does not transmit them is if Pause frames have caused the NIC to stop transmitting. Therefore, the relationship between the at which the CWR state is reached, the txqueuelen, and the Pause ON threshold in packets (P auseon) is given by the following equation, CW R > RT T C + P auseon + txqueuelen (2). Therefore, using TCP over circuits can have detrimental effects on the achievable throughput if the congestion window grows in an uncontrolled way. However, if TCP buffers are properly tuned, losses can be avoided. The maximum number of outstanding packets in TCP is the minimum of the congestion window and the receiver s advertised window, or min(, rwnd). Hence, if the TCP receive buffer is set so that the number of outstanding packets does not cross the capacity of the end-to-end path, packet losses will not occur. On the other hand, to achieve maximum throughput, TCP buffers should be at least equal to the BDP of the end-to-end path. However, setting exactly equal to the BDP of the path may not result in maximum throughput. When the congestion window is exactly equal to the BDP, TCP is in streaming state. Every incoming ACK packet releases a new packet. Therefore, to keep the end-to-end path full of packets at all time during this streaming state, a TCP sender has to respond to an ACK immediately. Due to multitasking on end host, it is advisable to set TCP buffers higher than the BDP of the end-to-end path. A larger congestion window will cause packets to be queued inside the switch buffer. While the end host does not respond immediately to an ACK, queued packets inside the switch will keep flowing, fully using the circuit bandwidth. III. CTCP Starting with Linux , congestion control algorithms can be dynamically loaded in as kernel modules. We have developed a TCP congestion control module called CTCP. In CTCP, a configurable parameter sets the congestion window value, which remains unchanged for the duration of the connection, allowing the window-based flow control dictate the data-transfer pattern. If the idle period is larger than one retransmission timeout, TCP resets the congestion window to the restart window value, which in practice is 1 segment [4]. CTCP eliminates this feature. Further, by specifying the congestion window value, we cap the number of unacknowledged packets allowable, enforcing a strict upper limit on the resources (buffer space, bandwidth, host memory, etc.) a CTCP flow will use. A. Experimental Setup IV. EXPERIMENTS Fig. 6 shows the connection configuration for a Layer-1 circuit. We connected two end hosts, Zelda1 and Wuneng, to Gigabit Ethernet (GigE) interfaces of Sycamore SN16 Zelda1 Number of PAUSE Frames GigE SN16 Reno - OC3 BIC - OC3 Reno - OC12 BIC - OC12 SONET Interface = 8.85ms SN16 GigE SONET Interface Fig. 6: Layer-1 Circuit Setup PAUSE Frame Count Wuneng Transfer Size (MB) Fig. 7: Pause Frame Count switches. We set up a SONET circuit between the two Sycamore switches. The Ethernet ports are then mapped to this SONET circuit. We did memory-to-memory data transfers of variable sizes from Zelda1 to Wuneng using Reno, BIC, or CTCP as the transport protocol. We also varied the SONET circuit rate (OC3 and OC12). For each of these transfers, we record the instantaneous throughput every 1 ms, and the congestion window () and round-trip time () every 5 ms. We also captured the entire transmission using tcpdump. The roundtrip time between these two hosts is 8.85 ms. The txqueuelen is set to 1 packets. The SN16 switches have a Pause ON threshold of 1 MB and a Pause OFF threshold of 2 KB. B. Results In Fig. 7, we plot the number of Pause frames received by the sender against the transfer size for Reno and BIC. For the OC3 circuit rate, the number of Pause frames for both Reno and BIC increase at a steady rate. In case of the OC12 rate, the number of Pause frames seen by BIC increases at a lower rate, whereas Reno sees little or no Pause frame. In all circuit rates, CTCP sees no Pause frame. We analyzed the steady increase rate of the number of Pause frames by looking at the lengths of the Pause ON periods, Pause OFF periods, and the number of Pause frames seen in each of these Pause ON periods (Fig. 8). During the Pause ON periods, the sender NIC sits idle, while the switch drains data from its buffer at the SONET circuit rate. For the OC3 rate (Fig. 8a), the average length of the Pause ON period is.27967s, during which time the switch drains 529 KB of

5 A B C.27967s.4965s Pause ON Pause OFF Pause ON (a) Circuit Rate = OC3 (155 Mbps) 22 Throughput Throughput over OC3 - Reno A B C.698s.1167s 2 18 Pause ON Pause OFF Pause ON (b) Circuit Rate = OC12 (622 Mbps) Fig. 8: Pause Frame data. This causes the switch buffer to fall below the Pause OFF threshold. As the sender NIC resumes transmission after the Pause ON period, it transmits data at a 1 Gbps rate during the following Pause OFF period. This causes data to be queued in the switch buffer at (1 155) Mbps rate, resulting in 2 KB of data queued in.4965 seconds. Hence, the switch crosses the Pause ON Threshold again, and enters into the Pause ON period. The number of Pause frames generated during a Pause ON period is 16 for OC3 rate, or for every ( ) = seconds 16 Pause frames are generated. In other words, for each 623 KB of data forwarded by the switch, 16 Pause frames are generated. This gives an estimated 2629 Pause frames for a 1 MB transfer, where the actual number from Fig. 7 is A similar analysis can be done for the OC12 rate (Fig. 8b). Since the difference between the NIC rate and the OC12 circuit rate is smaller, the switch takes less time to fall below the Pause OFF threshold. Hence, the number of Pause frames is lower for an OC12 circuit than the OC3 circuit. At such high circuit rate, the sender sometimes fails to transmit data at the available bandwidth due to multitasking. Therefore, in some instances, BIC incurs less (or no) Pause frames. Being less aggressive than BIC, Reno incurs only occasional Pause frames at this rate. In Figures 9, 1, and 11, we plot throughput, congestion window (), and round-trip time () for a 1 GB transfer over an OC3 circuit using Reno, BIC, and CTCP. Due to regular Pause ON and OFF periods, the throughput graph for Reno (Fig. 9a) shows drops at regular interval. The average throughput of the entire 1GB transmission is 142 Mbps for all transport protocols. As explained in Sec. II, the congestion window plot (Fig. 9b) for Reno after the initial slow-start period, and at regular intervals in BIC (Fig. 1b), drops after crossing 175 as TCP enters CWR state. In our experiment, the NIC transmit-queue length (txqueuelen) is set to 1 packets, the switch buffer can hold 7 packets (1 MB Pause ON Threshold) before entering into Pause ON period, and there are approximately 5 packets on the wire, a local congestion event is received from the NIC buffer when the congestion window crosses 175 packets. Hence, both Reno and BIC drop their congestion windows to half once reached above 175 packets. Following the drop, Reno enters the congestion avoidance state, and increases the congestion window by only one packet Throughput (Mbps) (ms) (a) Throughput Reno over OC (b) CWND Reno over OC (c) Fig. 9: Reno over OC3-1MB Transfer

6 24 Throughput Throughput over OC3 - BIC 22 Throughput Throughput over OC3 - CTCP Throughput (Mbps) Throughput (Mbps) (a) Throughput (a) Throughput 18 BIC over OC CTCP over OC (b) CWND (b) CWND 16 BIC over OC3 19 CTCP over OC (ms) 8 (ms) (c) (c) Fig. 1: BIC over OC3-1MB Transfer Fig. 11: CTCP over OC3-1MB Transfer

7 in each. Being more aggressive than Reno, BIC quickly recovers from this drop. If the bandwidth-delay product (BDP) of a path is large, this slow increase of the congestion window will hurt the overall throughput [5]. The number of queued packets in the NIC buffer and the switch buffer causes the to buildup. When these buffers are full, each new packet will be queued behind a maximum 17 packets. For an OC3 circuit, the service time for a standard 15B packet is 77us. Hence, a packet may experience a maximum queueing delay of 17 77us or 131ms. Therefore, the maximum seen in Reno (Fig. 9c) and BIC (Fig. 1c) is as large as ( ) 14ms. V. CONCLUSIONS We can draw two primary conclusions from this work involving port-mapped circuits, where a port s traffic is mapped to a single, dedicated Layer-1 circuit. First, using Layer-1 circuits allows us to achieve high throughput but, since all the traffic entering a port is treated identically, port-mapped circuits are suitable only if two end hosts are directly connected to the circuit switch. Second, we can see the benefits of CTCP: lossless transfer while maintaining a steady and achieving the full throughput. With BIC and RENO, unless we use the send or receive buffers to prohibit the number of outstanding packets from exceeding the ncap, there will be an increase in the and the flow will experience losses, even if the maximum throughput is still achieved. REFERENCES [1] T. Armstrong and S. S. Gorshe, Eds., IEEE Commun. Mag., Special issue on Generic Framing Procedure (GFP) and Data over SONET/SDH and OTN, vol. 4, no. 5, May 22. [2] A. Aggarwal, S. Savage, and T. Anderson, Understanding the performance of TCP pacing, in Proceedings of the 2 IEEE INFOCOM Conference, Tel-Aviv, Israel, Mar 2. [3] C. Partridge, ACK spacing for high bandwidth-delay paths with insufficient buffering, Work in Progress, IETF Internet Draft draft-rfced-infopartridge-1.txt, [4] M. Allman, V. Paxson, and W. Stevens, TCP Congestion Control, IETF RFC 2581, Apr [5] S. Floyd, HighSpeed TCP for large congestion windows, Feb. 23. [Online]. Available: