Westwood SCTP: load balancing over multipaths using bandwidth-aware source scheduling

Transcription

1 Westwood SCTP: load balancing over multipaths using bandwidth-aware source scheduling C. Casetti and W. Gaiotto CERCOM - Dipartimento di Elettronica Politecnico di Torino Torino, Italy Abstract While it is more and more usual to buy computer equipment with multiple network interfaces, and the proliferation of ISPs for both wired and wireless users provides different solutions to access the network core, upper layers across the protocol spectrum seem unprepared to accept the challenge posed by multiple interfaces and, consequently, by multiple paths between the same source-destimation pair. This paper tries to partially answer this challenge by proposing an enhancement of the SCTP protocol that aims at balancing load across multiple connections on disjoint paths in an accurate fashion, through the knowledge of bandwidth estimation on each path. Some results confirm the viability of the approach and invite further research. I. MULTIHOMING AND LOAD BALANCING Nowadays, it is not uncommon for laptops and palmtops to have more than one network interface: beside the now obiquitous Ethernet ports and 82.11b cards, other technologies such as GPRS, UMTS, bluetooth or satellite links offer additional opportunities to access network services. Commonly, only one access method at a time is selected, depending on the available network coverage. However, even in case of overlapping coverages by different access technologies, the TCP/IP protocol stack now implemented in nearly every PC does not provide any alternative. Several researchers have addressed the issue of multiple interfaces and have highlighted the advantages of multihoming, either in the case of one host having a single interface and multiple network-layer addresses, or in the case of one host having multiple interfaces and an equal number of unique addresses, one per interface. It is usually argued that multihoming provides enhanced reliability and fault tolerance, allowing the use of one network interface during normal operation and the migration of traffic to another interface should the first connection fail (hopefully, without the connection being reset). Multihoming support is also desirable in case of mobility handover, i.e., when the a mobile host moves to a foreign network and is in the process of acquiring a new address. If multiple connections can be used at the same time, the advantages comes connection persistence, as well as from bandwidth aggregation and load balancing. The layer at which multihoming support is implemented is of course debatable. If the support is limited to the IP layer, then additional mechanisms external to the host, such This work was supported by the Italian Ministry of University and Research through the PRIMO project as those envisioned by Mobile IP, are needed to provide seamless handover. If the support is located at the transport layer, i.e., the transport layer handles the splitting of data onto several IP addresses corresponding to local physical (or virtual) interfaces, then we have a self-contained solution: data are sent to one interface, leaving the other as backup; or, data can be evenly split among all active interfaces; finally, one could also provide for the traffic load to be split according to more sophisticated policies that depend on metrics, such as congestion level, estimated RTT, path reliability or monetary cost, associated to each active interface. The IETF has recently standardized a transport-layer protocol, called SCTP [1], whose design partially addresses some of the issues outlined above. Recently, SCTP has attracted considerable attention from the networking community [2]. Beside implementing the same congestion control algorithms as TCP-SACK [3], SCTP supports multihoming and trasmission over multiple paths through the concept of association. An association is a set of transport-layer connections from the same couple of endpoints, each connection theoretically routed over one of the available multiple paths. Upper-layer protocols use a single SCTP endpoint at the transport layer to exchange data, while SCTP handles the transmission of such data over multiple connections, each operating over a single network-layer interface. The IETF standard provides for SCTP to send data over one connection at a time for every association, leaving the others for redundancy purpose (i.e., to send retransmitted packets). The activity level of each backup connection is constantly monitored through probe packets called heartbeats. Sending data over multiple paths using SCTP is theoretically feasible, although it must be carefully planned out, to avoid delays caused by out-of-sequence delivery 1. A blind splitting of SCTP packets, i.e. in an equal-share, round-robin fashion, would be useless, if not harmful, in case of paths with uneven bandwidths, or with different RTTs. Indeed, the transfer time of a single file would be dominated by the transfer time on the slowest path. II. BANDWIDTH-AWARE SOURCE SCHEDULING OVER SCTP Our contribution is twofold. Firstly, we have devised and simulated a simple algorithm which, using RTT and band- 1 SCTP provides in-order delivery to upper layers /4/$2. 24 IEEE 325

2 width estimation, performs a source scheduling at the SCTP layer that tries to maximize the chance of in-order delivery over multiple paths. RTT estimation is already part of the SCTP standard, being used to set retransmission timeouts. Bandwidth estimation is performed a-la-westwood [4], i.e., by low-pass filtering the rate of returning SCTP acks. The TCP Westwood (TCPW) flavor we chose to implement was the so-called Westwood+ [5]. Secondly, the standard congestion control algorithms in SCTP have been integrated with TCPW functionalities, in order to fully exploit the advantages of bandwidth estimation. Coherently, we have dubbed the new protocol Westwood SCTP, or W-SCTP. We summarize the scheduling algorithm by way of a simple example: suppose a multi-link path Π i is available; the path is characterized by a pair [B i,p i ], i.e., available bandwidth on the bottleneck link and the one-way flight time (roughly, half the RTT). In the following, we assume that the transmission time of packets on the bottleneck link and their one-way propagation time are the dominant delays on each path. It is well known that, in the absence of cross traffic, two backto-back packets of size D sent on a multi-link path will be delivered with a time separation (dispersion) equal to D/B i. That is to say, that the time needed for a train of n such packets to be delivered is equal to p i +(n 1) D/B i. The first term depends on the bandwidths of each link and on the physical propagation delays, but it only affects the first packet, since all subsequent packets will be delivered separated by the same dispersion. If we have two paths Π i and Π j that can be used by the same W-SCTP association, and we assume that the delivery delays p i and p j on each path are comparable, then, to have in-sequence delivery of a single file at the same W-SCTP destination, we need to accurately schedule each packet of the file at the W-SCTP source by sending n i packets on path Π i and n j packets on path Π j. To account for the lag introduced by packet dispersion, we must verify that n i D B i = n j D B j i.e., n i /n j = B i /B j. A weighted Round Robin scheduling with n i and n j as weights can be used, with the two values relatively prime to each other, to ensure that the number of out-of-sequence deliveries is as small as possible. This strategy, while preferable to a blind Round Robin, still would not ensure in-order delivery of single packets, but of only single rounds of packets. We now outline our scheduling enhancement. Let us suppose that, at each endpoint, a wall-time clock C is available, and that path Π i is associated with a delivery clock C i, which indicates the earliest time when the opposite endpoint becomes idle, after completing the reception of the last bit of data in flight on Π i. Whenever a new packet needs to be sent, the scheduler picks the Π i which guarantees the fastest delivery by observing that, for a packet of size D to be sent on Π i,the new C i can be estimated as follows: if the local-end of Π i is still busy sending a packet, C i = C i + D/B i, otherwise, C i = C + D/B i + p i. We will refer to this scheduling algorithm as Bandwidthaware scheduling: we choose the path where we want to send a packet by predicting which path will be fastest in delivering it to the destination. III. ARCHITECTURAL CHANGES REQUIRED TO SCTP As explained in [1], SCTP is designed to establish robust communication association between two endpoints, each of which may be reachable by more than one transport address [...] thus ideally one may need a separate set of congestion control parameters for each of the paths. In this section, we highlight some differences in SCTP architecture and in the handling of congestion control, that were necessary in order to implement bandwidth-aware scheduling in W-SCTP. Firstly, we introduced an architecture called multibuffer multihoming, that derives from and enhances the multihoming support of the original SCTP. As we have seen, in our approach, when multiple paths are available for the association, all paths are used at the same time for the transmission of packets, called chunks in SCTP lingo. Instead, standard SCTP only uses the primary path for transmission and the rest for retransmissions or back-up. Furthermore, the management of each connection within the same association was made independent. Each W-SCTP connection has its own send buffer (there is a single, shared send buffer in SCTP), its own congestion window (CWND), Slow Start threshold (SSTHRESH), and Retransmission Timeout (RTO) timers. There is, however, a single receiver buffer in W-SCTP, like in Standard SCTP. All the chunks from each connection in the association are collected by this receiver buffer and SACKs are generated as in the original SCTP. Different Bandwidth Estimation processes are concurrently run by each connection, using the rate of returning SACKs (SACKs return on the channel from which the last packet was received). The presence of multiple send buffers led to some changes in the SACK management at the source. SACKs return on the channel from which the last packet was received, but it is possible that a SACK is reporting information about other on-going connections of the same association (as is the case for delayed acknowledgments). Thus, when a SACK arrives at the source, the information is processed on each interface and, consequently, all send buffers are refreshed. As a result, the bandwidth estimation for one connection can be still be updated even if that connection did not receive the SACK. Since the beginning of the association, each single connection proceeds without interference from other connections, handling only those packets present in its send buffer. When a new packet needs to be transmitted, it is inserted in the send buffer of the connection (on a specific Π i ) indicated by scheduling algorithm (Round-Robin or Bandwidth-aware). From then on, it is the connection responsibility to see that it is delivered: the packet remains in the send buffer until acknowledged; it causes Head-of-the-Line blocking to its own connection, but not to the other connections. Congestion control is run independently in each connection of the same association; the loss of a packet in a connection /4/$2. 24 IEEE 326

3 does not affect other connections CWND and SSTHRESH, but only those of the connection the packet was entrusted to. The behavior of the retransmission remains the same as in SCTP: lost packets are retransmitted over one of the other channel (although the choice of the channel is made through the use of the bandwidth estimator, to pick the one with the largest available bandwidth). The packet, though, physically remains in the original send buffer. The presence of multiple send buffers led to some changes in the SACK management at the source. SACKs return on the channel from which the last packet was received, but it is possible that SACK is reports information about other on-going connections of the same association. Thus, when a SACK arrives at the source, the information is processed on each interface and, consequently, all send buffers are refreshed. Finally, when too many consecutive timeouts occur, a connection failure is declared, as in SCTP. In this case, all chunks queued in the send buffer of the failed channel are shifted into the send buffer of the path chosen for the retransmission. The procedure of reopening a connection after the failure is the same as in SCTP. IV. SIMULATION RESULTS We have implemented W-SCTP on the ns-2 simulator, and we have run some tests on simple topologies, comparing its performance with that of simple Round Robin scheduling at the SCTP interfaces (RR-SCTP). Generally speaking, we considered a W-SCTP association spanning L paths, with L =2, 3, 4; onm =1, 2, 3 out of L paths, a TCP Westwood (TCPW) connection shares the same path with the W-SCTP connection. Each path includes two links toward the destination, the second being the bottleneck link. The RTT is fixed at 12 ms on each path. To allow a meaningful comparison, the aggregate capacity of the bottleneck links is the same in each topology, regardless of the number of parallel paths, and is equal to Mb/s, with the slowest bottleneck link always assigned a capacity of 128 kb/s, while the remaining link(s) have a capacity of 1/(L 1) Mb/s. In the 2-path topology, this choice nominally yields a 1 : 8 bandwidth ratio, although the presence of background traffic on the fastest bottleneck link(s) results in a smaller available bandwidth. Background traffic, represented by TCPW connections, never shares the slowest bottleneck link with W-SCTP. Experiments consist in the transfer of files of increasing size, ranging from 1 KB to 2 MB. We first studied a light-traffic scenario, with few background TCPW connections, then simulated heavy traffic, loading the channel with more background TCPW. Figure 1 shows the transmission time needed by W-SCTP and RR-SCTP to complete the file transfers, on different topologies; for the sake of readability, Figure 2 reports the gain of W-SCTP over RR-SCTP in transfer completion time (a value of 2 on the y-axis means that W- SCTP is twice as fast as RR-SCTP in delivering all bytes to the other end). As can be observed, W-SCTP fully exploits the knowledge of the available bandwidth; the benefits of the proposed approach are clearly visible, and lead to transmission time savings of up to 3% for 2 MB files. The gains are Average Transmission Time (s) path W-SCTP + 1TCPW 3-path W-SCTP + 2TCPW 4-path W-SCTP + 2TCPW 4-path W-SCTP + 3TCPW 2-path RR-SCTP + 1TCPW 3-path RR-SCTP + 2TCPW 4-path RR-SCTP + 2TCPW 4-path RR-SCTP + 3TCPW Fig. 1. Completion times of different-size file transfers using SCTP with equal share round-robin (RR-SCTP), compared to W-SCTP. Light background TCPW traffic. Transmission Time Gain path W/RR-SCTP + 1TCPW 3-path W/RR-SCTP + 2TCPW 4-path W/RR-SCTP + 2TCPW 4-path W/RR-SCTP + 3TCPW Fig. 2. Completion time gains of W-SCTP over RR-SCTP, for different-size file transfers. Light background TCPW traffic. most prominent when only two paths are available, which is conceivably the most common case. Figure 3 reports the time for which the receiving end had an out-of-order sequence of received chunks in its buffer. Out of order chunks cannot be pushed to the application layer, unless the application has explicitly waived the in-order delivery. Such delay can conceivably affect the QoS perceived by end users. Similarly yo transmission times,we also report the gain in Figure 4. The latter figure helps us underscore that packet reordering is especially frequent on two-path topologies. Indeed, when one path fails, or is congested and becomes backlogged, there is a chance that a sizable part of packets are delivered before their counterpart on the congested connection has been rescheduled onto one the available connections. This chance, and the amount of rescheduled packets, is smaller when more than one backup path is available. It is however interesting to observe that 25% of the time is spent by RR-SCTP in packet reordering. On the contrary, packet reordering is scarcely a factor when W-SCTP is used. The behavior under heavy traffic (Figures 5 and 6) confirms the previous findings. A final remark comes from the effect of paths with different RTTs. We show the CDF of the reordering delay for the case of /4/$2. 24 IEEE 327

4 Average Reordering Delay (s) path W-SCTP + 1TCPW 3-path W-SCTP + 2TCPW 4-path W-SCTP + 2TCPW 4-path W-SCTP + 3TCPW 2-path RR-SCTP + 1TCPW 3-path RR-SCTP + 2TCPW 4-path RR-SCTP + 2TCPW 4-path RR-SCTP + 3TCPW Fig. 3. Reordering delay measured over different-size file transfers using SCTP with equal share round-robin (RR-SCTP), compared to W-SCTP. Light background TCPW traffic. Average Transmission Time (s) path W-SCTP + 4TCPW 3-path W-SCTP + 7TCPW 4-path W-SCTP + 1TCPW 2-path RR-SCTP + 4TCPW 3-path RR-SCTP + 7TCPW 4-path RR-SCTP + 1TCPW Fig. 5. Completion times of different-size file transfers using SCTP with equal share round-robin (RR-SCTP), compared to W-SCTP. Heavy Background TCPW traffic. Average Reordering Delay Gain path W/RR-SCTP + 1TCPW 3-path W/RR-SCTP + 2TCPW 4-path W/RR-SCTP + 2TCPW 4-path W/RR-SCTP + 3TCPW Fig. 4. Reordeing delay gains of W-SCTP over RR-SCTP, for different-size file transfers. Light background TCPW traffic. Average Reordering Delay (s) path W-SCTP + 4TCPW 3-path W-SCTP + 7TCPW 4-path W-SCTP + 1TCPW 2-path RR-SCTP + 4TCPW 3-path RR-SCTP + 7TCPW 4-path RR-SCTP + 1TCPW Fig. 6. Reordering delay measured over different-size file transfers using SCTP with equal share round-robin (RR-SCTP), compared to W-SCTP. Heavy background TCPW traffic. all connections with the same RTT (12 ms) and with different RTTs (1 ms, 4 ms, 2 ms). Figure 7 and 8 show the CDF for the two cases. While the reordering delay remains within the acceptable limits of less than two seconds, the introduction of the further uncertainty on the RTT negatively affects the performance, although in a minimal way, confirming the robustness of the proposed algorithm. V. CONCLUSIONS The issue of multipath load balancing is likely to become a hot topic as soon as consumers have the opportunity to multihomed network devices, capable of accessing several network infrastructures at the same time. Performing a correct load balancing is quite challenging and bandwidth/rtt estimation can help establish the correct ratio of traffic on each path, especially if they are unevenly provisioned. Westwood SCTP (W-SCTP) is a promising new protocol that combines the advantages of bandwidth estimation for congestion control purposes with the chance to perform an accurate transportlayer load balancing. REFERENCES [1] R. Stewart et al., RFC 296: Stream Control Transport Protocol, IETF, October 2 [2] S. Fu and M. Atiquzzaman, SCTP: State of the Art in Research, Products, and Technical Challenges, IEEE Communication Magazine, April 24 [3] M. Mathis, J. Mahdavi, S. Floyd, A. Romanow, TCP Selective Acknowledgement Options, RFC 218, April 1996 [4] C. Casetti, M. Gerla, S. Mascolo, M. Sanadidi, R. Wang, TCP Westwood: Bandwidth Estimation for Enhanced Transport over Wireless Links, ACM Wireless Networks, No.8, pp , 22 [5] S. Mascolo, L. A. Grieco, R. Ferorelli, P. Camarda, G. Piscitelli Performance evaluation of Westwood+ TCP congestion control, Performance Evaluation, 55 (24), pp , Elsevier, North-Holland, January /4/$2. 24 IEEE 328

5 1.8 CDF WSCTP + TCPW WSCTP + 2 TCPW WSCTP + 7 TCPW Reordering Delay (s) Fig. 7. CDF of the reordering delay for W-SCTP associations transfering 2-MByte files, with increasing background TCPW traffic. Identical RTT on all paths 1.8 CDF WSCTP + TCPW WSCTP + 2 TCPW WSCTP + 7 TCPW Reordering Delay (s) Fig. 8. CDF of the reordering delay for W-SCTP associations transfering 2-MByte files, with increasing background TCPW traffic. Different RTT on all paths /4/$2. 24 IEEE 329