Application Layer Network Window Management in the SSH Protocol Chris Rapier Michael A. Stevens

Transcription

1 Application Layer Network Window Management in the SSH Protocol Chris Rapier Michael A. Stevens Abstract Multiplexed network applications, such as SSH, require the use of an internal flow control mechanism, usually implemented similar to TCP receive windows. The result of this is that the effect TCP receive window for that application is the minimum of the application buffer and the TCP receive window. In one of the most common SSH implementations this buffer is statically defined to 64KB and acts a significant throttle on bulk data transfers. In our work we dynamically define the size of this buffer and have yielded more than an order of magnitude improvement in throughput performance. Problem statement It s not uncommon for a user on a high performance network to see dramatic throughput rates with measurement tools such as iperf but dismal results with applications like SCP. The conventional wisdom regarding this discrepancy is that the ciphering process consumes so much CPU time that transfers are necessarily throttled. However, a quick experiment will show that, within reasonable limits, the throughput is independent of CPU power. Nor does it explain why short paths have better throughput than long paths. Obviously, the performance bottleneck must lie elsewhere. If we look closely at the SSH code we find that it, like a number of other network Bandwidth Delay Pr oduct = Bandwidth m RTT applications, make use of multiplexed channels inside of a single TCP connection to handle data transfer and/or control information. Because these channels cannot use TCP windows they must perform flow control at the application layer. Often the result is a second window imposed over an existing TCP window that forces the effective receive window size to the minimum of the two[1]. This means that even if the system TCP window is correctly tuned a user may still encounter dismal throughput under SSH. For example, a user transferring a file set from Host A to Host B might reach 30MB/s throughput using FTP but only 1.5MB/s using SCP. Not because of processor load but because of drastically undersized receive windows. This has proven to be a constant source of problems for researchers using high performance network connections. The security environment obviously demands encrypted authentication but the easiest application that does this is unacceptably slow. A researcher trying to transfer a 400GB dataset over a GigE connection is simply not going to be satisfied with a 1.2 MB/sec transfer rate (more than a 3 day wait). Alternatives that meet increased security requirements do exist: Kerberized FTP, Grid Services, and even VPNs. While protocols are much faster the infrastructure investment associated with them often limits their use to larger institutions. Smaller groups and individuals often don t have the time, expertise, or money to make use of these

2 alternatives and are often forced to accept poor performance. The community needs fast, cheap, and secure communications but we ve accepted that we can only choose two of those. Fixing the windowing problem in SSH will allow us to have all three. Removing that infrastructure barrier will help speed adoption, spur demand, and inspire new applications to make use of high speed encrypted data transfers. Approach The fundamental problem, as stated, is that the maximum effective receive window for any SSH connection will be limited to the smaller of the two buffers (internal or TCP). In the most common implementation of SSH, the receive flow control buffer is defined to a maximum of 128KB[2]. However, because of the way in which it is updated the maximum effective buffer is half that, or 64 kilobytes. If we take the Bandwidth Delay Product formula[3]; Bandwidth Delay Product = Bandwidth and solve for bandwidth using the buffer size of 64KB; 64KB Bandwidth m = RTT we see that throughput will always be no more than 64k/RTT. For example, on a path with a 80ms RTT the maximum throughput is 0.8MB/s. In comparison, a path with a 15ms RTT the throughput would be 4.3MB/s. A number of individuals have attempted to resolve this problem by statically increasing the buffer size in the source code. However, this is often a piecemeal solution that might work well for an individual along some paths but not on a more widespread basis. A home user simply doesn't need an 8MB buffer while a network researcher would be stifled by anything significantly smaller. Our approach has been to change the buffer from a statically defined value to one dynamically assigned during execution. By performing a getsockopt() function when the SSH window goes through an adjustment the application window can be set to match the TCP receive window size. By doing this at each window adjustment and not just when the connection is created, it is possible to make full use of kernels that incorporate an autotuning TCP stack[4]. Results to Date Proof of concept tests undertaken between well-connected hosts on high BDP paths have shown throughput rates of over 24.4MB/s using the ARCFOUR cipher. An unmodified SSH implementation only achieved 1.9MB/s using the same (or any other) cipher. All tests indicate that the 24.4MB/s value is strictly CPU limited. Throughput results from this initial test can be seen in Figure 1. Tests undertaken more recently using a more powerful processor have shown peak speeds of 42.6MB/s. These results have consistently ranked in the top 10 non-measurement m RTT

3 throughput speeds on the Internet 2 Weekly NetFlow Reports[5]. Excerpts from these reports can be found in Tables 3, 4, and 5. Continuing work While it might seem that the work has been completed, what has been done to date is only a partial solution. The RTT typically used to determine window size (either in an autotuning kernel or by manual calculation) is the host-to-host time but the internal window should be based on the application to application RTT. Research needs to be performed on where to define the endpoints of the RTT in order to fully maximize network usage. Additionally, enabling out of band communication between the client and server will allow queue length information to be exchanged allowing for even finer tuning of the transfer rate. We are also continuing to develop SSH and SCP to further increase throughput while maintaining protocol compliance and compatibility with the existing installed sshd and ssh client base. For example, currently SCP pipes data to the SSH process on both the sending and receiving side. Eliminating this pipe would dramatically decrease the number of memory copies used. SCP and SSH also have mismatched buffers for STDIN and STDOUT resulting in 4 reads for every write performed. Just matching these buffers can give a 20% to 40% decrease in the amount of time spent in the kernel and user space, respectively. We also plan to introduce mid stream cipher switching so that a user can use encrypted authentication and clear text data transfer which dramatically reduces the processor requirements. Additional enhancements possibly include parallel file transfers using multiplexed connections, streaming data transfers, and transferring multiple files without rekeying. References [1] S. Ubik, P. Cimbal, Achieving Reliable High Performance in LFNs, TNC Zagreb, Croatia, May 19-23, [2] OpenSSH 3.8p1 Source Code (channels.c) [3] W. R. Stevens, TCP/IP Illustrated Volume 1: The Protocols, Reading, MA, Addison Weseley Longman Inc. 1994, pp [4] J. Semke, J. Mahdavi, M. Mathis. Automatic TCP Buffer Tuning. Proceedings of SIGCOMM 98 Conference, Vol 28, No. 4, pp , August 1998 [5] Internet2 Flow Reports [online], Internet 2 Corporation, ref. Oct 20,

4 XPLOT GRAPH TEXT Text for both ssh-norm and ssh-hpn tsg images Time sequence graphs produces by tcptrace and xplot are excellent tools diagnostic tools that provide deep insight into the behaviour of any given tcp connection. In these images we find: The green line, representing the current ACK values received, the yellow line, representing the current advertised receive window (RWIN) size, and the black Xs representing the data segments. The vertical distance between the ACK and RWIN lines is equal to the current available window and the horizontal distance is the round trip time. The position of the segments between the ACK and RWIN line are an indication of how much of the window is being used and hence how efficiently the connection is using the network. Text for ssh-norm.tsg.png In this image we see a typical tcptrace from an scp transfer using a stock version of OpenSSH 3.9p1 with a throughput of 1.9MB/s. Note that the ACK and RWIN lines indicate a fairly large available window. However the segments, which are riding right on top of the ACK line, clearly show that a very small portion of the available window is being used. This is the result of an internal application window that is much too small for this path. Text for ssh-hpn.tsg.hpn In this image we see a tcptrace from an scp transfer using the high performance networking patch for OpenSSH 3.9 with a throughput of more than 30MB/s. Again, we see that the ACK and RWIN lines are again indicating a reasonably large window. In this case though, the position of the segments continuously approach the RWIN line indicating that the receive windows is being increasingly consumed. In short, the internal application window is consistently growing to make more efficient use of the network. It is also worth noting that the lack of congestion scallops in the RWIN line indicates that this connection is processor bound. Text for both ssh-norm and ssh-hpn owin images The outstanding data graphs produce by tcptrace and xplot are helpful in determining the current local congestion window and is a rough guide to the dynamics of the windows. The yellow line is the advertised receive window. The red line is the amount of outstanding data at arbitrary points during the connection. The blue and green line represent average and weighted averages of the outstanding data. Text for ssh-norm.owin.png Here we see that the amount of outstanding data is extremely low in comparison to the available window. In fact the amount of data in transit never exceeds 80KB which is less than 3% of the carrying capacity of this path (PSC <-> NCSA via I2). Text for ssh-hpn.owin.png

5 Here we see two things first that the advertised receive window is adjusting to match the conditions of the application at this point the SSH application is processor bound. However, at more than 30MB/s this should be considered acceptable. Second we see that the application is using almost all of the available window indicating that the application is making extremely efficient use of the network. In fact, its using more than 58% of the available carrying capacity of a moderately trafficked path (PSC <-> NCSA via I2).

6 SIDEBAR TEXT Why are windows important? TCP is a reliable data transmission protocol and, as such, relies on acknowledgements of received data (ACKS) to make sure that each sent packet of data is received. However, its inefficient to force the sender to wait for an ACK for each packet of data it sends. Therefore when an ACK is sent the receiver tells the sender how much data it can send before it must wait for an ACK. This value is the Receive Window (RWIN) and is equal to the remaining space in a receive buffer that holds data until it can be processed by the CPU. This is illustrated in Figure 2. The optimal size of this buffer is equal to the amount of data that can be in transit between the sender and receiver at any given time which is equal to the round trip time between the two hosts multiplied by the slowest link in the path. This is known as the Bandwidth Delay Product (BDP) and is different for each path and network conditions at the time of the transfer. If the RWIN is less than the BDP then the full carrying capacity of the network is not being used and the receiver spends time waiting for the sender to transmit more data. This ends up slowing the transfer rate sometimes considerably. Examples of this can be found in Table 1 and Table 2. The problem is that many systems and many applications were written for a much slower environment than high performance users currently exist in. Since most RWIN sizes are set to a single static system value the size is often compromise and results in many connections being inefficient. Likewise, if an application uses a statically sized flow control buffer (essentially an application RWIN) it may act as a significant drag on performance in many instances. Table 1 This table provides a list of Bandwidth Delay Products (BDP) where BDP = Path sl RTT and Path sl is the slowest link on that path in MB/s. The optimum receive window size for that path is equal to the BDP Path sl (MB/s) RTT (s) Bandwidth Delay Product KB KB KB MB MB

7 Table 2 This table shows the maximum potential throughput for the sample paths in Table 2 if an undersized Receive Window (RWIN) is used where Throughput = RWIN. Efficiency is RTT the potential throughput on each path for that RWIN in comparison to the available bandwidth. Path sl (MB/s) RTT (s) RWIN Throughput (MB/s)_ Efficiency KB % KB % KB % KB 17 14% KB 68 5%

8 BRAGGING TABLES FOLLOW The following tables are the results from recent Internet 2 Weekly Flow Reports. These reports list the fastest flows observed in the Internet 2 core in various categories. The following reports encompass a 3 week period from September 20, 2004 to October 11, The hpn-ssh process ran on TCP port between a Quad Itanium 1.3Ghz host using Linux 2.6 at NCSA and an Intel Pentium 4 2.4Ghz Host using Linux 2.6 Web100 kernel at PSC. Week of 2004/09/20 from Table 3. Fastest Bulk TCP Non-measurement Flows with Unique AS Source and Destination Throughput (b/s) Packet size (bytes) Duration (s) Src AS Dest AS Application type 883.7M PUNET Technology Ltd.,Taiwan [9270] CERN [513] > M NCSA [1224] SDSC-TEST1 [1227] > M PSC-NCNE [5050] NCSA [1224] > M NASA-HPCC-ESS [7847] USAN [194] Hotline 245.0M CSUNET-NE [2153] CERN [513] NFS 237.4M NCSA [1224] PSC-NCNE [5050] > M NASA-ESDIS-NET [22767] USAN [194] Hotline 214.8M CERN [513] SLAC [3671] > M APAN-JP [7660] CARIN-AS-BLOCK [7082] > M SDSC-TEST1 [1227] NCSA [1224] > 5150 Week of 2004/09/27 from Table 4. Fastest Bulk TCP Non-measurement Flows with Unique AS Source and Destination Throughput (b/s) Packet size (bytes) Duration (s) Src AS Dest AS Application type 739.6M ORNL [50] SDSC-TEST1 [1227] > M ORNL [50] NCSA [1224] > M Unknown [32093] ORNL [50] > M APAN-JP [7660] CARIN-AS-BLOCK [7082] > M CARIN-AS-BLOCK [7082] APAN-JP [7660] > M ORNL [50] Unknown [32093] > M PSC-NCNE [5050] NCSA [1224] > M NCSA [1224] PSC-NCNE [5050] > M NASA-ESDIS-NET [22767] USAN [194] Hotline 219.6M NASA-HPCC-ESS [7847] USAN [194] Hotline

9 Week of 2004/10/04 from Table 5. Fastest Bulk TCP Non-measurement Flows with Unique AS Source and Destination Throughput (b/s) Packet size (bytes) Duration (s) Src AS Dest AS Application type 5.322G CERN [513] ABILENE [11537] > M BWI-GIGA-POP [10886] CARIN-AS-BLOCK [7082] Hotline 801.2M ORNL [50] SDSC-TEST1 [1227] > M ORNL [50] NCSA [1224] > M APAN-JP [7660] CARIN-AS-BLOCK [7082] AFS 384.9M ORNL [50] Unknown [32093] > M Purdue [17] ORNL [50] > M PSC-NCNE [5050] NCSA [1224] > M NCSA [1224] PSC-NCNE [5050] > M NCSA [1224] CIXP (Geneva) and TIX (Zurich) [559] > 8086

10 Figure 1: This graph compares the average throughput rate of a standard distribution of SSH (OpenSSH 3.8p1) against SSH with the adjustable window patch. Various ciphers are used. The hosts were a Dual PIII 1Ghz and Quad Itanium 1.3Ghz using linux 2.6. Transfers were from a memory file system to /dev/null.