Design Decisions for Multipath TCP

Transcription

1 Design Decisions for Multipath TCP Costin Raiciu, Mark Handley July 21, Introduction Multipath TCP is a proposed set of extensions to TCP that allow the simultaneous use of multiple paths through the Internet for a single TCP connection. There are a number of goals, including improved ability to utilize multiple network interfaces (for example, on smart phones), multiple provider links (for example, multihomed servers), increased robustness to path failure, and potentially improved ability to cope with mobility. The ability of a protocol to satisfy these goals depends on detailed design choices. The purpose of this document is to discuss how the set of requirements for multipath TCP constrain these design choices, and leads towards a particular set of design choices. 1.1 Requirements Multipath TCP should be able to satisfy at least the following high-level requirements: Application Compatibility: for existing TCP applications, there should be no change to the operating system API or to the reliable in-order byte-stream service model provided. It should be possible to enable multipath extensions to TCP and for existing applications to continue to function acceptably well without requiring any modification. Performance: Multipath TCP should be able to use the network paths at least as well as regular TCP; in other words the worst case behavior of multipath TCP should be similar to regular TCP. Similarly, regular TCP flows should not suffer significantly more from sharing a path with a multipath TCP flow than they would from sharing the same path with a regular TCP flow. Network Compatibility: whenever a regular TCP flow would successfully transfer data, a multipath TCP flow should also succeed in transferring the same data. In particular, this means that multipath TCP should work in the presence of currently deployed firewalls, NATs, transparent proxies and other middleboxes. 1.2 TCP Overview TCP functionality includes in-order delivery, reliability, congestion control and flow control. packet header is as shown below. The key mechanisms used by TCP are: The TCP 1

2 Source Port Destination Port Sequence Number Acknowledgment Number Data U A P R S F Offset Reserved R C S S Y I Window G K H T N N Checksum Urgent Pointer Options Padding data Session State Machine: the SYN, FIN and RST flags are used to drive the TCP connection state machine, to establish and tear down TCP sessions. Sequence Numbers: included in every segment, they allow the receiver to order data and prevent duplicates and holes before passing the data to the application. Cumulative Acknowledgments: included in every segment, they acknowledge reception of data segments. The cumulative acknowledgment succinctly summarizes the state of a connection. A TCP sender uses duplicate acknowledgments and a retransmit timer to detect lost segments. To ensure reliable delivery, lost segments are retransmitted. Selective Acknowledgments, carried in TCP options, can be used to optimize loss detection and avoid unnecessary retransmissions. Receive Window: carried in each TCP segment, the receive window tells the sender how many bytes the receiver has space to accept and buffer, relative to the cumulative ACK. This provides flow control, allowing a slow receiver to pace a fast sender, and avoids the need to discard data at the receiver. Congestion Window: maintained by each data sender, it tells a data sender how many segments it can safely inject into the network without waiting for an acknowledgment. The use of the urgent pointer and Push flag are not expected to be modified by multipath TCP, so we will not discuss them in this document. 1.3 Multipath TCP Concept The basic idea of multipath TCP is to establish multiple subflows between the same pair of TCP endpoints. Any form of path identifier could be used to distinguish these subflows, but for the purpose of this document we assume they are distinguished by using different sets of IP addresses and/or TCP ports. 2

3 Once more than one subflow has been established for a connection, a single flow of bytes sent by the application is striped across the subflows. The precise striping algorithm is outside of the scope of this note, but it can be assumed that a striping algorithm will, as far as reasonable, attempt to fill complete TCP segments where possible, and will attempt to make the receiver s reassembly task reasonably simple. At the receiver, it is assumed that packets will often arrive out-of-order because the different paths have different latencies. The receiver must be able to reconstitute the original bytestream and feed it in-order to the receiving application. To satisfy the robustness requirement, it should be possible for a multipath TCP connection to survive the unexpected failure of any of its subflows, so long as one working subflow remains. In practice this requires that a segment transmitted on one subflow must be able to be retransmitted on a different subflow if the first subflow is no longer able to deliver it. Finally, it is assumed that multipath TCP should be negotiated as required between capable end-hosts without resort to any external signaling mechanism. Thus signaling should occur in-band, and should cause fall-back to regular TCP behavior should either end-system wish it. 2 Basic Mechanisms To complete a design, we need to map all of the basic TCP mechanisms into their multipath equivalents and show how they handle both normal and anomalous behavior. 2.1 State Machine To define multipath TCP, we need to detail any changes to the TCP state machine. In particular, this entails connection establishment and teardown, and equivalent mechanisms for additional multipath TCP subflows Connection Establishment For multipath TCP to be backwards compatible with regular TCP, it must leverage the TCP SYN handshake to establish the connection. This is most easily performed by including a TCP option in the SYN packet indicating that the endpoint is multipath-tcp capable and wishes to enable multipath TCP on this connection. In the SYN/ACK, a passive listener that wishes to enable multipath TCP also sends a multipath-tcp-capable option. Thus at the end of the handshake each endpoint knows the other endpoint is TCP-capable. The lack of such an option in either SYN or SYN/ACK causes the connection to be treated as a regular TCP connection. Care must be taken to support TCP simultaneous open. It is strongly desirable (indeed necessary, depending on other design choices made) that the connection establishment should reach a well-defined state, where either both endpoint or neither endpoint believes multipath TCP is enabled by the end of the three-way TCP handshake. This made slightly more problematic by the presence of middleboxes. We postulate two types of middlebox behavior that might interfere with connection establishment: Dropping SYN packets or SYN/ACK with the multipath-capable option. Removing the multipath-capable option from SYN or SYN/ACK packets. 3

4 Dropped SYN packets will be retransmitted. To provide eventual fallback to regular TCP behavior, a host should retransmit a SYN without the multipath-capable option after a set number of retrys. A similar procedure should be followed for SYN/ACK packets. When the multipath-capable option is removed from a SYN packet, this causes immediate fallback to regular TCP behavior, so causes no problem. When the multipath-capable option is removed from a SYN/ACK packet by a middlebox, this might lead to an inconsistent state, where the active opener believes the connection is not multipath-enabled, but the passive opener believes it is multipath-enabled. To avoid this inconsistent state, it is necessary for the ACK that completes the TCP handshake to provide indication as to whether the active opener believes the connection successfully enabled multipath TCP. Design Principle 1: SYN and SYN/ACK packets must carry multipath-capable options indicating their sender wishes to enable multipath TCP. Design Principle 2: the third packet of the handshake must carry some form of ACK of the multipathcapable option in the SYN/ACK Subflow Establishment When additional subflows are set up, they need to be established in a way that resembles as closely as possible a new TCP session. This is to allow firewalls and NATs to establish the correct state. Design Principle 3: new subflows are established by sending an additional SYN packet which contains an appropriate TCP option that allows the recipient to tell that this is a new subflow for an existing connection. The same design choices regarding SYN/ACK and the third packet of the handshake apply to subflow establishment RST semantics With regular TCP, receipt of a RST will reset the connection. For multipath TCP, it must only reset the subflow, otherwise a failure on one path would cause the whole connection to fail FIN semantics A FIN could either signal that a subflow is being closed, or that the whole connection is being closed. Consider first the case where a FIN indicates the whole connection is being closed. If data is being sent on more than one subflow when the FIN is sent, the receiver needs to know whether to wait for outstanding data on the subflows on which the FIN was not sent. If such a subflow dies, it then becomes necessary to resend data on the subflow on which the FIN has already been sent, which violates the semantics middleboxes expect to see. Even to know there is outstanding data, there needs to be an unambiguous binding between the FIN and the data sequence space, not just the subflow sequence space. Thus the semantics of the FIN flag are closely bound to the semantics of sequence numbers. 4

5 From the point of view of middleboxes, it is strongly desirable for a FIN to be sent on all subflows. However, if this were required but a single FIN also indicates the end of data, it now either becomes necessary to receive a FIN on all subflows (which is not possible if one fails in the last RTT), or an explicit signaling of end of data is needed. Thus multipath TCP requires an explicit signaling of end of data, no matter what choice is made regarding the semantics of the FIN flag in the TCP header. Given this, it is then more flexible for the FIN flag to indicate the end of data on that subflow only. This also fits more closely the expected semantics that middleboxes expect to observe. Design Principle 4: The FIN flag in the TCP header indicates the closure of that subflow only. Design Principle 5: An explicit way to indicate end of data across all subflows is needed. This end of data indication must be transmitted reliably and acknowledged to allow the sender to safely free its send buffers. 2.2 Sequence Numbering Multipath TCP takes the application byte-stream and splits it across multiple subflows to the destination. The question then is whether the TCP sequence numbers in the TCP packet header indicate the data sequence space for the whole connection, or only indicate data sent on the subflow in question? If a single sequence space is used, no change to the TCP on-the-wire protocol is needed; a packet would have the same sequence number no matter which subflow it was sent on. The sender then keeps track of which packets were sent on which subflow so as to disambiguate acknowledgments. There are three problems with this approach: If a packet is first sent on one subflow, then resent on another subflow, it is not possible to correctly infer loss. Middleboxes such as traffic normalizers, some firewalls and transparent proxies cannot relay such a subflow correctly. Some firewalls such as pf rewrite sequence numbers to improve the randomness of initial sequence numbers. The presence of such a firewall could cause a subflow to fail, or under rare circumstances to reassemble data in the incorrect order. Thus it is clear that the sequence numbers in the TCP packet header must refer only to the subflow, not to the whole multipath connection. However subflow-local sequence numbers are not sufficient to allow the receiver to reassemble the byte-stream in-order. To achieve this, an additional set of sequence numbers must be included, giving the data s sequence number within the whole connection. This provides the proper separation of functionality: Design Principle 6: TCP sequence numbers are used as subflow sequence numbers to detect lost packets for the purpose of congestion control, and also to enable fast retransmission. They also present a consistent view to middleboxes. A separate sequence space is maintained per subflow. Design Principle 7: Additional data sequence numbers are used to allow the receiver to reassemble the application data stream into its original order at the receiver. 5

6 We will return to the issue of how to encode and carry data sequence numbers in Section 3. Given that data sequence numbers are added, the next question is how should acknowledgments be used? The TCP packet header contains a cumulative acknowledgment field, and design principle 6 states that this is used for lost packet detection for congestion control and for triggering fast retransmission. Beyond this, the regular TCP acknowledgment field serves two additional purposes: It is used to free buffers at the sender when data has been received. It is used together with the receive window field to implement flow control. We will discuss flow control first, then return to the issue of freeing buffers at the sender Flow Control With regular TCP, flow control is implemented via the combination of the receive window field and the acknowledgment field in the TCP packet header. The receive window indicates the number of bytes beyond the sequence number from the acknowledgment field that the receiver can buffer. The sender is not permitted to send more than this amount of additional data. Multipath TCP also needs to implement flow control, although packets now arrive over multiple subflows. To determine the correct behavior we must examine how a receiving host handles packets arriving from the subflows. If the receiving host performed per-subflow buffering, and only put data back into the correct order at the point when it needed to pass data to the application, then it might make sense for the receive window to refer to the per-subflow buffers. However, this would lead to a deadlock scenario: suppose there are two subflows, and the next packet that needs to be passed to the application was sent on subflow 1, but was lost. In the meantime subflow 2 fills its receive buffer. Subflow 1 now fails. We would wish the missing data to be resent on subflow 2, but it cannot be, because there is no available buffering. The connection is now deadlocked. The conclusion is that the buffering at the receiver must be treated as a single pool of buffering that is shared between the subflows. Design Principle 8: There must be a single pool of receiver buffering shared between subflows. For flow control to work correctly, the receive window must refer to this per-connection buffer pool, and not to per-subflow buffers. A single receive window must refer to the data sequence space, rather than the subflow sequence space. In other words, it must indicate to the sender the maximum data sequence number that can be sent. However, in accordance with design principle 6, the sequence number and acknowledgment number in the TCP header refer to the subflow sequence space, not the data sequence space. In data packets we add an additional data sequence number, so the receiver can determine how to reassemble the data flow. This leads to the question of whether we need to add an additional cumulative data acknowledgment field, analogous to the data sequence number field. The receive window field would then be encoded relative to the value in the cumulative data acknowledgment, and could serve its flow control role properly. The alternative to adding a data acknowledgment field is to infer cumulative data acknowledgments from the subflow cumulative acknowledgments. The receive window would then be encoded relative to this inferred cumulative data acknowledgment. 6

7 (a) Drops due to incorrect inference (b) Stalls due to incorrect inference Figure 1: Problems with Inferring the Cumulative Data ACK from Subflow ACKs To infer cumulative data acknowledgments the sender would keep a scoreboard of which data sequence numbers were sent in which subflow sequence numbers. In fact it needs to do this anyway to perform fast retransmission on receipt of duplicate acknowledgments. When an acknowledgment is received, the sender simply looks up in the scoreboard to infer which data sequence numbers are being acknowledged. For fast retransmission, this works fine, but unfortunately it is not so reliable for flow control in the presence of reordering. Consider the scenario in the left hand side of Figure 1. For simplicity of exposition, sequence numbers in this example indicate packets, but the same principle applies with byte sequence numbers. The receiver has sufficient buffering for two packets. In accordance with the receive window, the sender sends two packets; data segment 1 is sent on subflow 1 with subflow sequence number 10, and data segment 2 is sent on subflow 2 with subflow sequence number 20. The receiver acknowledges the packets using subflow sequence numbers only; the sender will infer which data is being acknowledged. At the start of the example, the inferred cumulative ack is 0. In the Ack for 10, the receiver knows it is acknowledging data 1 in order, but the receiving application has not yet read the data, so relative to 1, the receive window is closed to 1 packet. In the Ack for 20, the receiver knows it is acknowledging data 2 in order, and the application still has not read, so relative to 2, the receive window is now zero. Unfortunately the acks are reordered simply because the RTT on path 2 is shorter than that on path 1, a common event. The sender receives the Ack for 20, infers that 2 has been received but 1 has not. The data cumulative ack is therefore still 0. 1 When the ack for 10 arrives, the receiver infers that 1 and 2 have been received, so the data cumulative ack is now 2. The receive window indicated is 1 packet, relative to 2, and so the sender can send packet 3. However, the receiver cannot buffer 3 and must drop it. The sender might note that something strange happened on receipt of the ack for 20, but it is unclear what the sender can do about this. If it takes this to indicate the receiver dropped data after sequence number zero, then it must resend this data. However, this is inconsistent with the data having been acked. In general, the problem is that although it is possible to infer a data cumulative ack from the subflow acks, as the subflow acks arrive in a different order from that in which they were sent, the inferred data cumulative 1 The sender also notes that the receive window has been reduced to zero relative to sequence number 0, which is somewhat strange as packet 2 has been acked. 7

8 ack and the actual data cumulative ack as intended by the receiver step differently with each ack message. If the receiver encodes the receive window relative to the actual data cumulative ack it maintains and the sender decodes it relative to the inferred data cumulative ack, the sender will infer the wrong maximum data sequence number that is permissible. This is not a corner case; it will occur whenever RTTs differ so as to cause the acks to arrive in a different order from that in which they were sent. The right hand side of Fig. 1 shows another possible consequence of this mismatch in calculating the allowed data receive window. The scenario is the same as before, except that the receiving application reads data between the receipt of data packet 1 and data packet 2. In this case the receive window does advance, but the sender does not know it until later, and a sending opportunity is missed. The examples given are for very small windows and only two subflows. Generally the problems become worse as windows increase, and become harder to analyze with more subflows. These problems will occur whenever flow control kicks in and RTTs vary. In fact, due to the implementation of dynamic receive buffer sizing in modern operating systems, the receive window often tracks only just above the level actually required. It is likely that receive window problems can occur in such operating systems even when flow control should not kick in. Our conclusion is that to restore the full flow control functionality of regular TCP, we need to add an explicit data acknowledgment field in addition to the subflow acknowledgment field in the TCP header. Receive window can then be unambiguously encoded relative to this explicit data acknowledgment value. Design Principle 9: Multipath TCP must include an explicit cumulative data acknowledgment field indicating the highest data sequence number that has been received in-order. Design Principle 10: For flow control to function correctly, the receive window field in the regular TCP header must encode the maximum offset that the sender is allowed to send relative to the cumulative data acknowledgment field. These principles do not necessarily require each TCP ACK has to be accompanied by a data ACK. Because the data ACK is only used for flow control, it does need to be included when it stays constant. This is different from regular subflow ACKs that are also used to trigger fast retransmission Freeing Sender Buffers Given the presence of an explicit cumulative data acknowledgment, it makes sense for this to be used to free buffers at the TCP sender, rather than inferring receipt from the subflow acknowledgments. In the absence of middleboxes though, either behavior would be acceptable. However, there is one corner case where it makes a difference which mechanism is used. Consider a connection with two subflows; subflow 1 is direct, but subflow 2 traverses a middlebox that pro-actively acknowledges TCP segments once it has received them in order, even though they have not yet reached the TCP receiver. Such middleboxes might include performance enhancing proxies for wireless networks, transparent proxies, or some forms of firewall. In the presence of such a middlebox, consider what happens when connectivity to the receiver is lost via subflow 2, as might happen if it moved out of coverage of a wireless basestation. The middlebox acknowledges a segment, but only then discovers it can no longer reach the receiver. The sender receives the subflow acknowledgment for this segment. If it uses the subflow acknowledgment to free the data at the sender, then this segment cannot now be resent on subflow 1 which is still working. As a result the connection fails. 8

9 If instead the sender waited for the explicit cumulative data acknowledgment before freeing the data, it would not suffer from this problem. Design Principle 10: Data must only be freed from the sender s buffer on receipt of the data cumulative ack, not on inference of reception based on receipt of a subflow ack. 3 Encoding To complete a design for multipath TCP, we must determine not only the information that must be sent (Section 2) but also how it should be encoded within TCP packets. For connection management (e.g. multipath TCP negotiation, and subflow setup) we must use TCP options, TCP s traditional extension mechanisms. TCP options will be ignored on receipt by a host that does not understand them, and so this permits fall-back to regular TCP behavior in these cases. The other additional data that needs to be sent is data sequence numbers, data acknowledgments and the data fin. There are two main ways in which these could be carried: Convey the additional data using using new TCP options. Convey the additional data within the TCP payload part of the packet, using a chunked TLV encoding or an escape mechanism to allow a multipath TCP receiver to separate control data from application payload data. For data sequence numbers, from the point of view of the end systems there seems to be little difference between using a data sequence number option or embedding the data sequence numbers in control data in the payload. When TCP segmentation offload hardware is used, there is a slight advantage to payload encoding of data sequence numbers. This is because while the OS sends just a single large data packet, this can be split into multiple segments by the NIC. This presents no problem for payload encoding, but for option encoding it means there is no longer a strict binding of TCP option to segment. In fact this is a general problem with using TCP options to convey the data sequence number mapping; it can also occur with middleboxes and with the receiving NIC. It is however, not a significant limitation; the data sequence number option merely needs to specify a binding between an absolute data sequence number and a specific byte position within the subflow, together with a length for which the binding is valid. Thus the cost is not correctness, but rather a few extra bytes of encoding. To summarize: A hypothetical payload encoding would divide the payload bytestream of each subflow into chunks, where each chunk has a TLV header. A chunk containing application payload data would be preceded by a chunk providing the data sequence number of that application payload data. A hypothetical option encoding of data sequence number would include the data sequence number itself, and also the subflow bytestream position corresponding to that data sequence number and the length of the binding. A receiver cannot assume the option will arrive in the same packet as the payload it describes. For the encoding of cumulative data acknowledgments, the issues are rather different. 9

10 Figure 2: Flow Control on the path from B to A stops the data flow from A to B Encoding data acks in TCP options poses no additional problems. However, encoding data acks in the payload can be problematic. For the sake of concreteness, let us assume that a hypothetical payload encoding uses the chunked TLV structure above, and that a data ack is contained in its own chunk 2. The problem arises because although TCP pure acks are not congestion or flow controlled, data acks embedded in the payload must be. Once the OS has embedded a data ack, it must be treated as being data: It must be consistently retransmitted if it is lost so that middleboxes such as firewalls can track subflow sequence number state correctly. Due to TCP segmentation offload and middleboxes, it cannot be assumed that a payload-encoded data ack will arrive in a single packet, nor that one will not be split between two packets. It must be subject to TCP flow control, because the receiver must buffer the data before the TLV can be decoded. Also a preceding application data chunk may be half-way through being transmitted when the receive window closes. Such a half-sent chunk must be finish being sent before a subsequent data ack can be sent. It is difficult to not subject the data ack to congestion control, unless there is no data flowing from the receiver to the sender. Even then, middleboxes may apply congestion control to pure data acks. Of these, the problems show up most clearly when we consider flow control. Figure 2 provides an example. In this scenario, A s receiver buffer is full because the application has not read the data, but A s application wishes to send data to B whose receive buffer is empty. This might occur for example when B is pipelining requests to A, and A now needs to send the response to an earlier request to B before handling the next request. A sends segment 10, B stores it locally, and wants to send the data ACK, but can t do so: flow control imposed by A s receive window stops him. Because no data acks are received from B, A cannot free its send buffer, so this fills up and blocks the sending application on A. The connection is now deadlocked. A s application will only read when it has finished sending data to B, but it cannot do so because his send buffer is full. The send buffer can only empty when A receives an data ack from B, but B cannot send a data ack until A s application reads. This is a classic deadlock cycle. 2 The same basic issue arises with other payload encodings such as using escape codes. 10

11 A will eventually time out the data it sent to B, because no data ack is received and retransmit it. Eventually it times out the connection. Design Principle 11: Data acknowledgments must not be subject to flow control. In practice this means that cannot be encoded in the payload, and option encoding must be used. 4 More Deployment Considerations Our discussion thus far only covered the basic mechanisms needed to support multipath TCP. There are many other deployment considerations that must be taken into account for the protocol to work correctly through the Internet. These considerations are due to deployed middleboxes that optimize various aspects of TCP communication, and that could potentially interfere with multipath TCP mechanisms. Irrespective of whether payload or option encoding is used for data sequence number mapping, there is a basic assumption that the payload length is not modified in transit: however, there are certain application level proxies that do just that. Some FTP and SIP aware NATs will rewrite IP addresses within the payload, and due to ASCII encoding this may potentially changing the length of the payload. To give TCP the illusion that the sent data was received, these proxies will fix the ACK stream to match the modified data stream. With payload TLV encoding of the data sequence number, this causes a loss of TLV synchronization, causing the connection to fail. With option encoding of the data sequence number mapping, this results in overlapped mapping or a gap in the mapping. An overlapped mapping would cause payload corruption and a gap in the mapping would cause the subflow to fail. To deal with this issue, multipath TCP needs to detect it. The only reliable way is to checksum the sent data. A checksum should be included with each segment, and checked at the receiver. If the payload length is modified, the checksum will fail. As this is a condition that Multipath TCP cannot handle, it should simply close the affected subflow. If it has other good subflows (i.e. ones with no checksum failure) it will keep using those. If the problem occurs on the only remaining subflow, it should setup a new subflow on the same path, specifying an infinite mapping. This reverts multipath TCP to single path TCP for the remainder of the connection. Design Principle 12: Data sequence number mappings should include a checksum of the data for which the mapping applies. Failure of the checksum should cause the subflow to be terminated, or a fallback to regular TCP. Such checksumming can incur performance penalties, especially for flows on local area networks. expect checksumming to be disabled for flows on LANs and within data centers. There are a few more subtleties in coping with middleboxes that alter the TCP connection. A multipath TCP subflow will only be established on a path that does not block the multipath capable option on TCP SYN and SYN/ACK packets, so there is a basic expectation that the path can then also transfer multipath options on TCP data packets. However, routes can change, and a multipath TCP connection may find itself on a new path which then either drops packets with unknown options or removes unknown options from packets. A multipath TCP implementation needs to detect these conditions. The normal response should then would be to drop the subflow, or if it is the only subflow, then to fall back to regular TCP behavior. Design Principle 13: When faced with a failure to transfer data on the only remaining subflow, multipath TCP should attempt to fall back to regular TCP behavior. We 11

12 Finally, it is possible for a middlebox that pro-actively acks packets to also modify the receive window to reflect its own buffering capability. Multipath TCP must be robust to such middleboxes. Design Principle 14: For calculating the receive window, multipath TCP should ignore all acknowledgments that do not advance the right hand edge of the receive window. Acknowledgments The design principles in this note have evolved over a considerable period of time, and with the aid of many people from the EU FP7 Trilogy project and the IETF Multipath TCP working group. In particular credit to due to Alan Ford, Phil Eardley, Lars Eggert, Marcelo Bagnulo Braun, Sebastien Barre, Bryan Ford, Michael Scharf, Janardhan Iyengar, Yoshifumi Nishida, Iljitsch van Beijnum, Randall Stewart, Tim Shepard, Murari Sridharan, Scott Brim, Andrew McGregor and Joe Touch. 12