Design and Evaluation of a Kernel-Level SCTP Implementation

Transcription

1 Design and Evaluation of a Kernel-Level SCTP Implementation Albert Meixner, Peng Yin, Donald Onyango, and Amin Vahdat Department of Computer Science Duke University falbert,py,dco,vahdatg@cs.duke.edu Abstract In this paper, we describe a kernel level implementation of the Stream Control Transmission Protocol (SCTP), standardized in RFC 2960 in October SCTP is a reliable transport layer protocol. It was initially developed as a signaling protocol to overcome some of TCP s limitations. It respects packet boundaries, and supports multiple independent streams of data within one connection. Unlike the fixed packet format of TCP, SCTP uses chunked packets, which is much more flexible. Being a relatively young member in the transport protocol family, SCTP has not been widely adopted. It is not supported by any of the widespread operating systems. Only a user level reference implementation and some experimental kernel implementations are available. To our knowledge, the SCTP implementation presented in this paper is the first kernel implementation for FreeBSD. We conducted some preliminary experiments with our kernel level implementation of SCTP. The results obtained confirmed some of the expected advantages of SCTP over TCP. 1 Introduction SCTP (Stream Control Transmission Protocol) [9] is a reliable connection-oriented transport protocol operating on top of a potentially unreliable connectionless packet service, such as IP [1]. It is designed by the Signaling Transport (SIGTRAN) group of the Internet Engineering Task Force to serve as a reliable signaling and control transport protocol for telecommunications traffic to run on IP networks via a number of proposed adaptation layers. However, SCTP also has potential for traditional data applications as a result of a number of desirable features, including message framing,multi-homed connection association, disassociation of reliable transfer from in-order delivery, and multiple logical streams multiplexed over a single physical connection. The expedient and reliable exchange of control information is vital for the success of modern telecommunication networks, and influences the user perceived quality of service substantially. The evolution of modern heterogeneous networks further requires that signaling transport is provided via pure IP networks

2 with satisfactory reliability and performance. Currently, the exchange of signaling messages in an IP-based networks is typically performed by either UDP [14] or TCP [2]. However, both of these protocols have a number of limitations when used for signaling and control. For example, UDP only provides unreliable datagram service. The burden of error checking and congestion control is thus placed on the end applications. TCP, on the other hand, TCP provides reliable data transfer. However, the abstraction of a single reliable and in-order byte stream is not always a perfect match for telecommunication and certain classes of data applications. First, the-byte oriented nature of TCP requires explicit delineation of messages by the end application. The end application is further required to signal the end of a message via the PUSH flag to enforce immediate transfer of the message. Second, many applications that do require reliability do not require strict ordering of the underlying data. TCP s coupling of reliable and in-order delivery can cause a head of line blocking problem. Here, application data is buffered in the kernel waiting for the delivery of earlier packets. This can cause unnecessary delays for applications that are able to move forward with processing of later packets. Delivering the later (out-of-order) packets to the applications would also allow more ACK s to be injected in the network, perhaps aiding ACK-paced congestion control mechanisms [11]. A third limitation of TCP for certain applications is that TCP connections are strictly identified by a pair of transport addresses (IP address + port number). This prevents transparent support of multi-homed hosts (though connection handoff mechanisms have been proposed for fault tolerance [17]). Next, TCP is not robust against certain denial of service attacks, for example, SYN flooding. Finally, TCP protocol extensions are somewhat limited by a pre-defined maximum size. SCTP differs from UDP by providing a reliable transfer of datagrams. At the same time, it overcomes some of the limitations of TCP enumerated above. In this paper, we describe a kernel-based implementation of SCTP. We believe that our implementation (proceeding in parallel with a number of related efforts) is among the first true kernel-level implementations of SCTP. We further present a performance evaluation of our prototype to validate some of the potential benefits of SCTP, with a particular focus on traditional data applications. Our source code is publicly available at vahdat/sctp. tar.

3 The rest of the paper is organized as follows. Section 2 gives an overview of SCTP transport protocol. Section 3 presents our kernel implementation of SCTP. Section 4 proposes some possible extensions and modifications for current SCTP standards. Section 5 details our performance evaluation results. Finally, Section 6 presents our conclusions. 2 SCTP Overview 2.1 Basic Properties of SCTP SCTP is designed as a reliable transport layer protocol that operates on top of the potentially unreliable network service, such as that provided by IP. Relative to TCP, SCTP has the following properties. It guarantees error-free non-duplicated transfer of user data. This is supported by the checksum and the verification number. It also uses a selective retransmission mechanism to correct data loss and errors. Unlike byte-oriented TCP, SCTP transfers data in chunks. SCTP performs path-mtu discovery to optimize the data fragmentation size, and segments large messages into chunks that fit into an IP packet. SCTP also supports optional bundling of multiple user messages into a single IP packet. To reduce the head-of-line blocking problem, SCTP uses the concept of a stream, an ordered flow of messages to be delivered to upper level protocols or the application program. The message sequence is only maintained within streams, while no order is enforced on messages in different streams. Thus, while one stream may be blocked because, for example, the packet containing the next in-sequence message was dropped, other streams within an SCTP association can proceed. This mechanism reduces the head-of-line blocking. The number of streams within each association is specified at the association startup time. In addition, a bypassing mechanism is used to deliver an SCTP message to the users upon the complete arrival of a message. SCTP tolerates partial network failures by supporting multi-homed nodes at both ends of an SCTP association. A list of multiple IP addresses associated with a single SCTP port number is provided by one side of an association to the other side. Consequently, several paths between the multiple-homed nodes are monitored and selectively utilized by SCTP, enhancing the robustness of the protocol against partial network

4 failures. Further, SCTP is designed to coexist with TCP [7] by employing TCP-friendly congestion control mechanisms. Finally, SCTP possesses some security features that make it robust against various denial of service attacks, as described in Section SCTP Packet Format 32 Bit Source Port Verification Tag Destination Port SCTP Common Header Checksum Type Flags Length Value Chunk 1 Type Flags Length Value Chunk 2 More Chunks Figure 1: SCTP packet format As shown in Figure 1, a typical SCTP packet is composed of a common header followed by several chunks. Each chunk contains either control information or user data. With the exception of INIT, INIT ACK, and SHUTDOWN COMPLETE chunks, multiple chunks can be bundled into one SCTP packet, up to the limit of the MTU size. Table 1 summarizes the chunk types in SCTP. The common SCTP header is 12 bytes long. It is composed of a 16 bit source port number, a 16 bit destination port number, a 32 bit verification tag and a 32 bit checksum field. The verification tag is used by the receiver of the packet to validate the sender of the packet, as discussed in section The checksum field performs error checking using the Adler-32 algorithm, which is stronger than the 16-bit checksum used by TCP and UDP. An invalid checksum causes the packet to be silently discarded. Each SCTP chunk is composed of a type field, a flags field, a length field and a value field. The type field identifies different types of control chunks and data chunks. The length field is required because the payload contained in the value field has variable length.

5 Chunk Type DATA INIT INIT ACK COOKIE ECHO COOKIE ACK ABORT SHUTDOWN SHUTDOWN ACK SHUTDOWN COMPLETE SACK HEARTBEAT HEARTBEAT ACK ERROR Description Contains payload user data Initiates an association Acknowledges receipt of INIT Echoes cookie back to its generator Acknowledges receipt of COOKIE ECHO Closes an association Initiates graceful close of association Acknowledges receipt of SHUTDOWN Acknowledges receipt of SHUTDOWN ACK Reports TSN gaps in received DATA Probes the reachability of peer point Response to HEARTBEAT Notifies peer endpoint about errors Table 1: SCTP Common Packets 2.3 SCTP Association and Shutdown State Diagrams State of Server State of Client (2) rcv INIT generate Cookie snd INIT ACK CLOSED [ASSOCIATE] (1) create TCB, snd INIT rcv valid COOKIE ECHO (4) create TCB, snd COOKIE ACK COOKIE WAIT rcv INIT ACK (3) snd COOKIE ECHO, COOKIE ECHOED rcv COOKIE ACK (5) ESTABLISHED Figure 2: SCTP association state diagram Figure 2 shows the association state diagram of SCTP. It is adapted from RFC 2960 [9]. The SCTP end point that initiates the association setup is called the client, while the other end point is called the server. SCTP uses a unique four-way handshake and a cookie mechanism to establish associations (analogous to connections) while preventing potential denial of service attacks such as SYN flooding. The mechanism works as follows:

6 1. Initially, the server and the client are in the CLOSED state. The client sends an ASSOCIATE request to the server to set up the association. The client generates a transmission control block (TCB), which contains all the necessary data for the maintenance and management of the association. A randomly generated Tag C is contained in the INIT packet. The client then enters the COOKIE-WAIT state. 2. The server processes the INIT request, and generates a temporary TCB. From this TCB, a minimal subset of information that is needed to recreate the TCB can be extracted. A Message Authentication Code (MAC) is also produced from this subset of TCB information and a secret key using a hash function. The TCB and the MAC are put into a Cookie, which is further packaged up and sent back to the client in an INIT ACK packet. The packet is marked with Tag C, together with a randomly generated Tag S. After this, the server deletes all information associated with the temporary TCB and goes back to the CLOSED state. 3. After receiving the INIT-ACK, the client responds by echoing the cookie back in the COOKIE-ECHO packet, which is marked by Tag S. The client then enters the COOKIE-ECHOED state. 4. When the server receives the cookie from the client, it validates its TCB with the MAC to confirm the authenticity of the cookie. Then, the TCB is recreated from the information contained in the cookie. Only at this time, the server actually assigns resources to the association and enters ESTABLISHED state. 5. A COOKIE-ACK packet is sent back to the client, and the client also enters the ESTABLISHED state. Figure 3 shows the shutdown state diagram of SCTP, as adapted from RFC 2960 [9]. SCTP does not support the half-connected state present in TCP. Once shutdown is initiated by one host, the other host can only finish sending outstanding packets and has to continue the shutdown afterwards. The shutdown process is completed after both the initial shutdown request and the response have been acknowledged by their respective receivers. For brevity, we omit a detailed description of the shutdown process summarized in Figure 3.

7 ESTABLISHED [SHUTDOWN] (1) check outstanding data chunks (2) No more outstanding snd SHUTDOWN SHUTDOWN SENT rcv SHUTDOWN ACK (5a) snd SHUTDOWN COMPLETE delete TCB rcv SHUTDOWN (5b) snd SHUTDOWN ACK move to SHUTDOWN ACK SENT SHUTDOWN PENDING CLOSED rcv SHUTDOWN check outstanding data chunks (3) SHUTDOWN RECEIVED No more outstanding (4) snd SHUTDOWN ACK SHUTDOWN ACK SENT rcv SHUTDOWN COMPLETE (6a) delete TCB rcv SHUTDOWN ACK (6b) snd SHUTDOWN COMPLETE delete TCB Figure 3: SCTP shutdown state diagram 2.4 Security Issues of SCTP SCTP sets its security objectives as The availability of reliable and timely data transport services and integrity of the user-to-user information carried by SCTP [9]. To achieve this goal, SCTP supports the following mechanisms: four-way handshake, the cookie mechanism, and the verification tag. As described in Section 2.3.1, SCTP uses a four-way handshake instead of the conventional three-way handshake to establish an association. This four-way handshake plus the cookie mechanism makes SCTP more robust against the blind denial of service attacks where malicious nodes send setup request with a fake source IP address [9]. TCP is vulnerable to SYN flooding because each time a SYN is sent to the server, the server responds by setting up a TCB without authenticating the SYN packet. A malicious sender thus can easily flood the server by sending lots of SYNs. SCTP is made more robust against this kind of attack by delaying TCB setup in the four-way handshake. Malicious blind attackers could also try to insert extraneous content into an SCTP flow without the knowledge of the SCTP packets in the flow. The verification tag is yet another security mechanism guarding against such blind attackers. The verification tag is generated randomly during the association setup, and is included in every SCTP packet. The verification tag must be verified at the endpoint for the packet to be admitted. Without intercepting the packet, the blind attackers are prevented from faking the 32-bit verification tag. SCTP packets with invalid verification tags are discarded silently.

8 2.5 Related Work A number of efforts are investigating various SCTP prototypes. A user level reference implementation has been developed in a cooperation of Siemens and the University of Essen [15]. It is published under the GNU Public License and runs under Linux, FreeBSD and MacOS. We used this implementation as a reference and for compatibility testing. Kernel level implementation are available for Linux [8] and Sun Solaris [12]. The Linux version was developed by Motorola as a kernel reference implementation. It is published as Open Source. The Solaris implementation is a prototype done by the Sun Microsystems Lab and is only available as a binary package without source code. 3 SCTP Kernel Implementation We used FreeBSD 3.4 for our implementation. The major guideline for the implementation is RFC 2960 [9]. Where this document is imprecise, we refer to the reference (user-level) implementation and other sources. Though they are not actually separated in the code, it makes sense to describe the sender and the receiver separately. 3.1 Receiver Data Chunk FreeBSD IP Layer TSN Received TSN Bitfield Reorder and Reassembly Buffer SCTP Packet Chunk Queue Chunk Bundling SACK Application Figure 4: Receiver Block Diagram Just as in TCP, the design of the receiver in SCTP is simpler than that of the sender. The major duties

9 of the receiver are to reorder and reassemble incoming packets and to send acknowledgments to the sender. Both functions are different from their TCP counterparts for various reasons. While reassembly is not necessary in TCP, packet reordering has to take multiple streams into account now. Selective acknowledgments are mandatory in SCTP, so the receiver has to track exactly the arrived packets instead of simply storing the last sequence number received. Figure 4 shows the basic design of the receiver. Whenever a packet is received, its checksum and verification tag are first checked. It is then processed chunk by chunk. The chunk handlers modify the state of the connection and write reply chunks to the output queue. When the entire packet is processed, the chunks in the output queue are sent. To reorder and reassemble the incoming packets, SCTP first places them into a queue, which is sorted by the transmission sequence number. A single pass over the queue is then performed to find all packets that are ready for delivery to the application and to reassemble complete packets. The queue is checked every time a new packet arrives. If an in order chunk is found, it is moved to the socket s receive queue. To track incoming packets, each connection maintains an array of bits indicating received TSNs. This structure is implemented as a circular buffer. The buffer is moved when an in order packet arrives. SACK blocks can be directly derived from this structure by doing an RLE representation of the set bits. SACK packets are currently sent for each incoming packet - not for each DATA chunk. 3.2 Sender Application Data Chunk Data Queue Fragmentation Retransmission Chunk Queue Chunk Bundling SCTP Packet SACK FreeBSD IP Layer Figure 5: Sender Block Diagram

10 The sender has to guarantee reliable delivery, fragmentation and congestion control. Again these mechanisms are implemented quite differently from TCP. Figure 5 shows the basic design. When a packet is passed to the kernel from the user level, it is stored in the socket send queue. It is immediately fragmented and receives a queue header, which contains the data chunk header and information required for packet retransmissions. Performing fragmentation at this point is not ideal because, for example, the MTU may change while the packet is in the queue. However, performing fragmentation later would require significant changes to the sender implementation because it currently treats packet fragments exactly like other packets and relies on them to be smaller than the MTU. The packet remains in this queue until it is acknowledged by the receiver. Whenever it is necessary to send data, such as after timeouts, after the user sends requests, or after changes in the congestion window, this queue is checked for packets ready to be sent. The routine first identifies the packets that need to be retransmitted because of timeouts or fast retransmits. The sender then checks for new packets that fit into the congestion and receiver window, and completes the data chunk header. Finally it puts a copy of all those packets into the out queue. The purpose of the out queue is to bundle multiple chunks into single packets. All chunks - not only DATA chunks - are put into this queue before they are sent. 3.3 API RFC 2960 suggests an API that should be used by the SCTP implementation. This API is designed to support the special capabilities of SCTP and is also used by the reference implementation. However, it is not compliant with the Berkeley socket API, which is widely adopted to access network functions and which is integrated in most operating systems. We decide to use the Berkeley socket API for our SCTP implementation because it better integrates SCTP with other protocols, makes conversion of existing programs easier, and does not require programmers to learn a new set of commands. Special features of SCTP are made available through the set/getsockopt function calls and a modified address structure. The RFC2960 API could be added to our implementation either as a set wrapper function provided in a library or as a further extension of the kernel that provides another interface to the lower level network functions. We are investigating both options for subsequent versions of our implementation.

11 3.4 Streams The major difference of SCTP compared with other protocols is the support for multiple streams within a single connection. However, the socket interface does not have any commands or parameters to control multiple streams. We find two solutions to address this problem. The first solution requires the modification of the sockaddr in structure, which is passed to sendto and recvfrom calls. This structure contains a number of zero bytes, so we could modify it without affecting the length. It now contains two new fields (stream and ppi) to control the respective fields in the data packets. This solution is appealing because it does not require any additional function calls. However, recvfrom and sendto are not normally used for connection-oriented protocols, and a sockaddr struct has to be set up before every call. The second solution uses setsockopt to set the active stream. This active stream will be used to send until it is modified by another call to setsockopt. Accordingly, directly before a recv call, getsockopt can be used to retrieve the stream on which the packet is received. The same works for ppi. This approach does not modify existing data structures and fits into the existing socket framework better. All that it requires is an additional function call after each recv and a function call to change the sending channel. We recommend the usage of these functions. Sample code depicting the use of this API for both a sender and receiver are presented in Figures 6 and 7. Before the connection is invoked, the application can specify the number of streams it wishes to use within that specific connection. This is also done with setsockopt calls. The getsockopt can then be used to query the number of actually available streams. 3.5 Implementation Status The current implementation provides basic SCTP functionality, but does not cover all the features available in SCTP. Table 2 summarizes features and their implementation status. RFC conformance was tested by both running our implementation with the reference implementation and by performing the tests described in [13]. However, we have not completed all of the tests in [13] yet.

12 int s, outstreams = 1, instreams = 1, sendstream = 0; char *str = "Hello World!"; struct sockaddr_in sin; s = socket( AF_INET, SOCK_STREAM, IPPROTO_SCTP ); sin.sin_family = AF_INET; sin.sin_len = sizeof( sin ); sin.sin_port = htons( ); sin.sin_addr.s_addr = inet_addr( " " ); setsockopt( s, IPPROTO_SCTP, SCTP_INSTREAMS, &instreams, sizeof( int ) ); setsockopt( s, IPPROTO_SCTP, SCTP_OUTSTREAMS, &outstreams, sizeof( int ) ); connect( s, (struct sockaddr *)&sin, sin.sin_len ); setsockopt( s, IPPROTO_SCTP, SCTP_SENDSTREAM, &sendstream, sizeof( int ) ); send( s, str, strlen( str ) + 1, 0 ); close( s ); Figure 6: Sender Sample int s, s2, len, instreams, outstreams, recvstream; char buffer[512]; struct sockaddr_in sin; fd_set fd; s = socket( AF_INET, SOCKET_STREAM, IPPROTO_SCTP ); sin.sin_family = AF_INET; sin.sin_len = sizeof( sin ); sin.sin_port = htons( ); sin.sin_addr.s_addr = INADDR_ANY; bind( s, (struct sockaddr *)sin, sin.sin_len ); listen( s, 1 ); s2 = accept( s, &sin, &len ); getsockopt( s2, IPPROTO_SCTP, SCTP_INSTREAMS, &instreams, &len ); getsockopt( s2, IPPROTO_SCTP, SCTP_OUTSTREAMS, &outstreams, &len ); FD_ZERO( &fd ); FD_SET( s2, &fd ); select( FD_SETSIZE, &fd, NULL, NULL, NULL ); getsockopt( s2, IPPROTO_SCTP, SCTP_RECVSTREAM, &recvstream, &len ); len = recv( s2, buffer, sizeof( buffer ), 0 ); close( s ); close( s2 ); Figure 7: Receiver Sample

13 Impl. RFC 2960 Handshake Yes Mandatory Shutdown Yes Mandatory Heartbeat Yes Mandatory Multi Homing Limited Mandatory Delayed Ack Yes Optional Chunk Bundling Limited Optional Streams 32 Mandatory Unordered Packets Yes Mandatory Packet Reassembly Yes Mandatory Packet Fragmentation Limited Optional SACK Yes Mandatory Flow Control Yes Mandatory Congestion Control Yes Mandatory MTU discovery No Optional Error Messages Yes Mandatory RFC conformance Limited Mandatory Table 2: SCTP implementation status 3.6 Future Implementation Work A number of features are still missing from our SCTP implementation. The most important step is to assure compliancy with RFC 2960, largely improved error handling and multi-homing. However, as motivated by our evaluation, the missing features currently missing are not absolutely necessary to gain advantages from using SCTP over TCP for certain classes of applications. Once the implementation is finished, SCTP performance should be re-evaluated and modifications to the implementation or even the protocol should be considered. The chunked structure of packets should make simple modifications to the protocol rather easy to deploy. Finally, the API could be improved by allowing the application to receive packets only from certain streams. This would remove the need to query the source of each packet after receiving it. This could be extended to allow multiple sockets on a per-stream basis.

14 4 Proposed Extensions and Modifications 4.1 Transport Layer The short-comings of SCTP will only be apparent after it has been used under various circumstances and after some practical experience has been gained. Currently, the protocol seems flexible enough to deal with most application cases. However, we identify some possible modifications that might improve the current standard. Stream Priority Currently all streams within an association are created equally. Previous work in related fields shows that FIFO queues do not offer ideal performance, therefore we propose to assign a priority value to each stream to schedule data transmissions. These priorities should be accessible to the application through the API. SCTP Vegas The current mechanism for congestion control defined in RFC 2960 is comparable to those used by the TCP Reno implementation. It may be desirable to use the congestion avoidance mechanism proposed for TCP Vegas [5, 6]. Association Parameter Control A well defined way to dynamically modify parameters of established connections would be helpful. This includes the number of streams, buffer sizes and the senders IP addresses. We propose to define an OPTION chunk, which contains the modifications as variable length parameters. All changes can be optional and have to be acknowledged by the receiver, maintaining backward compatibility with older implementations. 4.2 Application Layer Protocols To efficiently use the stream capabilities of SCTP, it is necessary to make slight modifications to higher level protocols and applications. These changes are generally minor and easy to implement. To illustrate that point, we propose enhancements to run FTP and HTTP efficiently over SCTP. The File Transfer Protocol [16] uses two separate connections, one for control information and one for data. The control connection is the first one to be initiated. It remains active until the user logs out. The

15 data connection is opened whenever a data transfer command is received on the control connection. Once the data transfer is finished, the data connection is closed. This also signals the end of the transfered data. It is easy to see that in SCTP data and control can be combined in one association by assigning them different streams. We propose to set up an association with two in and out streams. Stream 0 is used to issue commands and receive responses, while stream 1 is used for data. The end of the data transmission can be signaled by sending a special value in the ppi, which marks the last packet. Alternatively the chunk transfer encoding scheme can be used. This matches very well with SCTP since each chunk can be sent in a separate SCTP packet. The rest of the protocol would not require any changes, except for the commands for setting up the data connection (PORT, PASV) that become obsolete. Among other benefits, such a structure would eliminate the need to establish a new connection for each new data transfer. For HTTP, we will only consider HTTP version 1.1 [10] because of its support for persistent connections. The main disadvantage of this scheme is that it suffers from head-of-line blocking described earlier in this document and requires objects to be transferred in order over the TCP connection. This effect can be minimized by spreading the transfers across multiple streams. When a client connects to the server, it creates an arbitrary number of streams corresponding to the number of objects it would like to retrieve. When it requests data, it specifies the stream on which the data should be sent by adding another parameter in the request header. An example of this approach follows: GET /index.html HTTP/1.1 Host: Connection: keep-alive Sendstream: 0 Requests can be sent on any stream. If the sendstream field is missing or has the special value current, the response is sent on the stream on which the data is received. The end of the data transmission can again be marked by a special value in the ppi field or by using standard content-length or chunked transfer schemes. The modifications described above are only to show that existing protocols can be enhanced to take advantage of the streams. This is only for illustrative purpose. We do not expect them to be standardized as described here.

16 5 Evaluation All measurements were done using two Pentium II 400 Mhz running FreeBSD 3.4 connected via Ethernet. Different link speeds and latencies were simulated using dummynet. For all the experiments we simulated a 256KBit/s link with 250 ms latency and an 8KByte queue. The tests were done using delayed acknowledgment, with one ack for two packets. For associations with multiple streams, data was split among streams by sending packets on different streams in a round-robin fashion. The default packet size was 1KByte. 5.1 Congestion Control Congestion control in SCTP is similar to the mechanism used in TCP. Some adjustments are necessary to accommodate the packet oriented nature of SCTP. The implementation uses coarse-grained timeouts for retransmissions and incorporates one slight modification from the mechanisms described in the RFC fast retransmits can change the cwnd only once per RTT. Figure 8 shows the congestion window as a function of time Congestion Window Timeout Fast Retransmit Congestion Window Time Figure 8: Congestion Window 5.2 TCP Friendliness The algorithm for SCTP control is based on that of TCP congestion control [3] to ensure that SCTP does not behave more aggressively than TCP. To test the mutual impacts of SCTP and TCP on one other, we

17 performed a number of related experiments. For each experiment, two competing connections (one over TCP and the other over SCTP) each transmit 2000 packets. Each packet is 1488 bytes in length. We then observe the congestion behavior of the two competing connections, in particular to determine whether SCTP is friendly toward TCP traffic. Figure 9 illustrates the case where one TCP connection and one SCTP connection start transmission simultaneously. Here, delivers 13.6 KB/sec and TCP delivers 14.0 KB/sec. In Figure 10, the TCP connection starts data transfer first. During the period of simultaneous transmission, SCTP provides 14.6 KB/sec versus 14.8 KB/sec for TCP. Finally, in Figure 11, the SCTP connection starts to transfer data first and delivers 13.9 KB/sec versus 14.5 KB/sec during the period of simultaneous transfer. In summary, our initial tests demonstrate that SCTP appears to be TCP friendly for our set of measurements. Clearly, more extensive experimentation under a variety of conditions is required to verify this, but we consider the initial results to be encouraging since SCTP employs a congestion control mechanism that is very similar to that of TCP. SCTP manages the congestion control based on the granularity of associations rather than streams, so increasing the number of streams within a single association will not make SCTP associations to behave more aggressively TCP SCTP Bytes/Sec Seconds Figure 9: An SCTP connection and a TCP connection sharing a link, case 1

18 TCP SCTP Bytes/Sec Seconds Figure 10: An SCTP connection and a TCP connection sharing a link, case TCP SCTP Bytes/Sec Seconds Figure 11: An SCTP connection and a TCP connection sharing a link, case Head of Line Blocking TCP performs a strict in-sequence delivery per connection where arriving packets must wait in buffers until all previous in-sequence packets are received. This limitation can create the head-of-line blocking situation described earlier. However, the multi-stream nature of SCTP allows independent streams to be created within one association. Message sequencing is only maintained per stream instead of per connection. This allows shorter data sequences to easily drain through buffers because their streams are independent of larger out-of-sequence data streams. Such streams may have otherwise caused head-of-line blocking. Figure 12 shows the effect of the number of streams on the burstiness of the traffic and hence the

19 Packet Interarrival Time Variance % loss-rate 2.5% loss-rate 5.0% loss-rate Streams Figure 12: Head-of-line blocking - Burstiness effect of the head-of-line blocking in SCTP. We aim to measure the application level burstiness as this is a direct result of head-of-line-blocking at the transport layer. The burstiness is measured in terms of Packet Inter-arrival Variance, a statistical measure of the difference of in-sequence arrival of packets. The higher the variance, the more bursty the traffic, and hence the greater the effect of head-of-line blocking at the end hosts. From Figure 12, we see that one stream per connection, as found in ordinary TCP, produces the highest inter-arrival time variation and hence the worst head-of-line blocking. As the number of streams increases, the burstiness decreases exponentially. Note that these graphs do not include the effect of application layer head-of-line-blocking, as present in HTTP/1.1, which would gain even greater benefits from using multiple streams. 5.4 Transport Layer Fragmentation The transport layer data fragmentation of SCTP, combined with MTU discovery, guarantees that each SCTP packet always fits into a single IP packet. This is desirable for two reasons. Unlike TCP, the data in SCTP cannot be spread across packets arbitrarily, because it has to respect the original data unit boundaries. Using plain IP fragmentation does not work well with reliable delivery because, for each lost fragment, the whole datagram has to be resent. Furthermore, having a small number of large packets negatively effects congestion control because there are not enough packets in the pipe to trigger fast retransmission. Figure 13 shows the average bandwidth for transmitting 2KByte packets across a link with a MTU of 1500.

20 Bandwidth (KBytes/s) IP Fragmentation SCTP fragmentation Loss Rate (%) Figure 13: SCTP vs. IP fragmentation 5.5 Selective Acknowledgments The congestion control mechanism of both SCTP and TCP relies on packet loss being caused almost exclusively by congestion. Non-congestion related loss, which is common in wireless networks, severely degrade the performance of both protocols. However, as shown in [4], using selective acknowledgments can reduce the negative effects of a high loss rate because it allows the sender to retransmit lost packets earlier. In figure 14 we see the benefits gained by SCTP s SACK mechanism. Note that as the loss rate rises above some point, selective acknowledgment does not have any positive influence anymore. At that point, the congestion window is generally so small that the sender cannot take advantage of the additional knowledge gained from selective acknowledgment, because it can have at most 1 or 2 packets outstanding. 6 Conclusions Initially developed for transferring telecommunication signaling data, SCTP has much broader applications thanks to a number of features that are also useful for a broad range of traditional data applications including, the chunk oriented nature of data transfer, the multistream mechanism, the transparent support for multihomed hosts, and robustness against blind denial of service attacks. Our work is among the first true kernel level implementations of SCTP protocol. We have implemented most of the key features specified in RFC Our initial experiments with our SCTP kernel implementation demonstrate some of SCTP s potential

21 Bandwidth (KBytes/s) SCTP TCP Loss Rate (%) Figure 14: Loss rate vs. bandwidth benefits for a broad range of telecommunications and data applications. References [1] RFC Internet Protocol-DARPA Internet Program Protocol Specification. IETF, Network Working Group, September [2] RFC Transmission Control Protocol - DARPA Internet Program Protocol Specification. IETF, Network Working Group, September [3] M. Allman, V. Paxson, and W. Stevens. RFC TCP Congestion Control. IETF, Network Working Group, April [4] H. Balakrishnan, V. N. Padmanabhan, S. Seshan, and R. Katz. A Comparison of Mechanisms for Improving TCP Performance over Wireless Links. Proc. ACM SIGCOMM Conference, Stanford, CA, USA, Aug 1996, URL [5] L. Brakmo, S. O Malley, and L. Peterson. TCP Vegas: New techniques for congestion detection and avoidance. Proceedings of the SIGCOMM 94 Symposium, pages 24 35, [6] L. Brakmo and L. Peterson. TCP Vegas: End to End Congestion Avoidance on a Global Internet. IEEE Journal on Selected Areas in Communication, 13: , [7] Jungmaier A. et al. Performance evaluation of the stream control transmission protocol. Joint IEEE ATM Workshop 2000(ATM 2000 Conference, Heidelberg, Germany, pages 26 29, [8] Karl Olaf Knutson et al. sctp-kref may 2001 bakeoff 3 release. piggy/sctp refcode/, May,2001. [9] R. Stewart; Q. Xie et al. RFC Stream Control Transmission Protocal. IETF, Network Working Group, October [10] J. Fielding and J. Gettys et al. RFC Hypertext Transfer Protocol HTTP/1.1, January IETF, Network Working Group. [11] Van Jacobson. Congestion avoidance and control. In Proceedings of ACM SIGCOMM 88, pages , August [12] Sun Microsystems Laboratories. SCTP Prototype Software for the Solaris Operating Environment Version 1.1, May

22 [13] S. Mahajan and A. Singh et al. Conformance Test Specification for SCTP. Internet Draft, Febrary [14] J. Postel. RFC User Datagram Protocol. ISI, August [15] J. Postel. SCTP Reference Implementation sctplib pre9. Siemens and Computer Networking Technology Group of the University of Essen, [16] J. Postel and J. Reynolds. RFC File Transfer Protocol. ISI, October [17] Alex C. Snoeren, David G. Andersen, and Hari Balakrishnan. Fine-Grained Failover Using Connection Migration. In Proc. 3rd USENIX Symposium on Internet Technologies and Systems, March 2001.