Process/ Application NFS NTP SMTP UDP TCP. Transport. Internet

Transcription

1 PERFORMANCE CONSIDERATIONS IN FILE TRANSFERS USING FTP OVER WIDE-AREA ATM NETWORKS Luiz A. DaSilva y, Rick Lett z and Victor S. Frost y y Telecommunications & Information Sciences Laboratory The University of Kansas Lawrence, KS z Sprint Corporation Government Systems Division Kansas City, MO Introduction The mid-1990s have witnessed the deployment of the rst wide-area ATM networks, operating at speeds ranging from tens to hundreds of megabits per second. The increasingly widespread availability of such bandwidths enables, among a variety of other services, the transfer of a large volume of data in short periods of time. Applications such as the downloading of les from the World Wide Web and the concept of a data warehouse in corporations result in a pressing need for such high speed le transfers. FTP is a popular and reasonably powerful means of transfering data between two hosts. It provides for the establishment of two separate connections between the source and destination hosts. One of the connections This research is partially supported by DARPA under prime contract DABT63-94-C-0068 to Sprint Corporation. The authors can be reached via at ldasilva@tisl.ukans.edu. is used for the login between the machines, and utilizes the TELNET protocol; the other connection is used for the actual transfer of data [1]. High-bandwidth transfers of large les with strict time limitations present an entire new set of requirements to FTP. For instance, FTP transfers are commonly used to transmit large data les such as simulation data and satellite imagery. These types of applications typically have stringent requirements for le transmission delay. Conventional throughput values of 8 to 9 Mbps cannot meet the needs of emerging applications, and new methods of maximizing throughput must be utilized to meet these requirements. In our experiments, we utilized a commercial widearea ATM network, and actual FTP transfers were employed to duplicate a real-world application. This paper deals with the main limiting factors for FTP in such a scenario and ways to minimize these limitations. The remainder of this paper is organized as follows:

2 Process/ Application Transport Internet Telnet FTP TCP SMTP IP NTP UDP NFS Figure 1: Upper layers of IP protocol stack Section 2 presents the main issues involved in highspeed le transfers in a wide area environment; Section 3 presents the set-up and procedures used to test the performance of FTP in such an environment; Section 4 lists the main results of these experiments. Finally, Section 5 draws our main conclusions. 2 FTP in Wide Area ATM Networks FTP is a mature and widely used protocol. However, its use in wide area high-speed networks (eg. ATM WANs) presents a new set of issues. FTP makes use of Transmission Control Protocol (TCP) services such as reliable transmission. It sits ont top of TCP on the protocol stack, as shown in Figure 1. Like any application which utilizes TCP, the performance of FTP in a wide-area network is aected by TCP's window based ow control. End-to-end reliability is implemented by TCP through a sliding window-based acknowledgment and retransmission scheme. The transmitting host is allowed to transmit no more than a certain number of messages before the rst one is acknowledged. At any given moment, there are no more than W unacknowledged bytes within a TCP connection - we shall refer to W as the TCP window size [1, 2]. The primary limiting factor of throughput for TCP applications over high-speed connections is the TCP window size. The window size (in bytes) needed to ensure continuous transmission in the absence of packet errors can be calculated theoretically as the product of the link speed and the round trip delay. In this fashion, for a link operating at 34 Mbps and a round-trip delay of 60 ms, a window size of 255,000 bytes would be needed. Conversely, in the case of a coast-to-coast connection within the U.S., conventional TCP window sizes (i.e., 32 to 64 kilobytes) limit the throughput to about 9 Mbps. It is possible to extend the maximum TCP window size (up to a few MB in some cases), with varying degrees of diculty for dierent operating systems: Digital UNIX provides it as a system option; kernel patches are available for certain versions of Solaris; a source license is needed for changes in the SunOS kernel. Once a proper window size is used, the next major limiting factor is the disk access speed. Clearly, the use of disks capable of operating in the tens of Mbps range is needed to take full advantage of a high speed connection. The overhead incurred in the use of FTP itself and operating system overhead can also limit maximum throughput. 3 Experimental Set-Up The test network used for our experiments was congured to simulate a distributed national network with a site on each coast and two sites in the interior of the United States. The network is depicted in Figure 2. At the time of this experiment, the sites were connected to the wide-area ATM network at DS-3 rates ( Mbps). Even though some of the hosts are connected to the respective switches at a higher rate, traf- c pacing was used in order to avoid rate mismatches. The maximum achievable throughput, after considering physical and ATM layer overheads, can be calculated as approximately 37 Mbps. A set of Permanent Virtual Circuits (PVCs) was designed to connect the sites using Sprint's commercial ATM network with six bi-directional PVCs connecting the sites in a full mesh grid. These connections use the same backbone facilities that are used by all of Sprint's ATM customers. The test trac was carried by Sprint's production ATM switches and Inter-Machine Trunks (IMTs). Although the hosts shown in Figure 2 were not dedicated to the experiments, they presented minimal additional load. Additional experiments designed to evaluate networking capabilities of these hosts (see [4], for instance) indicate that CPU and memory were not limiting factors for the results reported in this paper. All measurements were repeated several times to account for variations in trac through the backbone, and values reported here represent result averages. 3.1 Modications in FTP As discussed in the previous section, high-speed le transfers require that large window sizes be utilized. In using FTP, the window size is determined by negotiation between the client (the host running the FTP source code) and the server (the host running the FTP daemon 1 ). Therefore, both the source and daemon codes must be modied to allow the use of large windows. It should be emphasized that the maximum window size supported by the operating system must generally be extended, as discussed in Section 2, in order for these modications to take eect. In UNIX systems, the Berkeley socket interface provides functions that support network communication 1 A daemon is a process which executes independently to fulll a specic function within the system.

3 A KANSAS D.C. B D CALIFORNIA SPRINT COMMERCIAL ATM WAN FORE ASX-200 ATM Switch DEC AN-2 ATM LAN Switch C SUN SPARCstation 20 DEC Alpha AXP 3000 MISSISSIPPI Figure 2: Test Network using many possible protocols (TCP/IP being one of them). When an application creates a socket, the operating system allocates a data structure to contain the information needed for communication [3]. In FTP, a socket is created prior to the transfer of data. One of the parameters of the system call to create this socket is the socket buer size, which is directly related to the TCP window size. This system call must be modied to allow a larger window size than the default (typically, 32 or 64 KB). The applicable portion of the FTP code with this modication is shown in Figure 3. Extensions designed to allow the users to set the desired window size every time he/she uses FTP were implemented as well. For our experiments, we have modied the available FTP code running under SunOS 5.4 and Digital UNIX Results In order to evaluate the performance of the le transfers under dierent conditions, the following measurements were taken: data = socket(af_inet, SOCK_STREAM, 0); if (setsockopt(data, SOL_SOCKET, SO_SNDBUF, &window, sizeof(window)) < 0) { fprintf(stderr, "Cannot set SO_SNDBUF for data socket"); if (setsockopt(data, SOL_SOCKET, SO_RCVBUF, &window, sizeof(window)) < 0) { fprintf(stderr, "Cannot set SO_RCVBUF for data socket"); if (getsockopt(data, SOL_SOCKET, SO_RCVBUF, &check, arglength)!= -1) { printf("rcvbuf Window Size: %d\n", check); if (getsockopt(data, SOL_SOCKET, SO_SNDBUF, &check, arglength)!= -1) { printf("sndbuf Window Size: %d\n", check); Figure 3: Portion of the modied FTP code. Round-trip Delay - Round-trip time (RTT) was measured to determine the theoretical TCP window size needed to ensure continuous transmission. The ping command was used to estimate this delay.

4 Source/ RTT Window Size Throughput Dest (msec) (bytes) (Mbps) B - A B - D A - D D - C B - C A - C Table 1: Round-trip time and maximum throughput results. An MTU size of 9180 bytes was used. Source/ File Size Throughput Destination (million bytes) (Mbps) B - C B - A D - B D - A D - C A - D A - B A - C Table 2: FTP throughput results with small windows. An MTU size of 4470 bytes was used. Maximum Throughput - The ttcp tool was used to measure maximum throughput; ttcp is a benchmarking tool for measuring the throughput between two systems using the UDP or TCP protocols. The program was created at the US Army Ballistics Research Lab (BRL) and is in the public domain. File Transfer Throughput - File transfer throughput was obtained directly from the FTP application. File sizes ranging from tens to hundreds of megabytes were used, recognizing the fact that high volumes of data are expected to be transmitted over high-speed networks. Cell Counts - ATM-layer measurements were taken at intermediate switches. These measurements were primarily aimed at identifying possible cell losses and/or retransmissions. 4.1 Round-Trip Delay The round-trip delay in wide-area connections is determined primarily by the circuit distance between the transmitting and receiving sites (i.e., the delay is dominated by the propagation delay of the signal, under normal loads). Coast-to-coast round trip delays are on the order of ms. Round-trip times measured in our experiments ranged from 36 ms to a maximum of 73 ms. Each PVC had consistent round-trip times over a number of tests. These results, summarized in Table 1, correlate very well with the actual physical route miles associated with each connection. 4.2 Maximum Throughput The maximum achievable throughput between two hosts was obtained for dierent TCP window sizes, with the results summarized in Table 1. A window size whose value is below the minimum to ensure continuous transmission, as discussed in Section 2, will limit the throughput to: W indow Size M aximum T hroughput = RT T (1) If we take as an example the rst result reported in Table 1, we see see that the throughput was limited to 4.1 Mbps. This is in close agreement with our measured throughput (3.9 Mbps). On the other hand, as we increase the window size, the theoretical link rate is approached. 4.3 FTP File Transfers The additional overhead of FTP, and specially the number of disk reads and writes, will reduce the eective throughput to values lower than those reported by ttcp for the same connection. A Maximum Transmission Unit (MTU) size of 4470 bytes was utilized in our initial le transfer experiments. This value is of special signicance due to the fact that a very large number of legacy LANs utilize FDDI connections, with transmission units of 4470 bytes. Table 2 summarizes the results for a number of transfers with dierent size les utilizing regular window sizes (32 to 64 KB). These low values of throughput are as expected for high speed data transfers using small TCP windows. We have observed no correlation between throughput and le size. The experiment was then repeated using the modied version of FTP, as discussed in Section 3.1, with the results listed in Table 3. By adapting FTP to use large windows, we were able to increase throughput by one order of magnitude. The ow control exercised by TCP is no longer the limiting factor; we believe that at this point, we have reached the limitations imposed by the disk access times and application and operating system overhead.

5 Source/ MTU Size File Size Throughput Destination (bytes) (million bytes) (Mbps) B - D D - B Table 3: FTP throughput results with large windows. An experiment in which three separate trac transfers were performed simultaneously (A to B, A to C, A to D) was also carried out. Since we are operating significantly below the network capacity, the throughput obtained for each trac stream was not adversely aected by trac produced by the other two. The research of wide-area network performance under competing trac streams is one of the authors' areas of interest, and is currently being further investigated. 4.4 ATM Layer Measurements Cell counts were collected at intermediate switches; these measurements serve the purpose of identifying whether a signicant number of cell retransmissions occurred. As an example, the transmission of 14 million bytes from B to C is the rst case reported in Table 2. Since an ATM cell can carry up to 48 bytes of information, we would expect a minimum of 291,670 cells. The actual number of cells measured at an intermediate switch was 297,375; the experiment was repeated three times with the same result. Considering that some cells may be only partially lled, such a result seems to indicate the absence of signicant cell losses within the network. All the connections behaved in the same fashion. 5 Conclusions When rst utilizing a wide-area ATM network, especially using TCP as the transport protocol, it is not uncommon for the user to experience a gap between expectation and reality: throughput which is one or two orders of magnitude below link speed is often reported. This sometimes leads to the misconception that it is impossible to obtain high throughput with TCP in this type of environment. Our experiments indicate that the high-speed transfer of les over a wide-area ATM network using FTP is achievable under two conditions: the protocol must use TCP window sizes compatible with the bandwidthdelay product for the connection (typically, this will be in the range of hundreds of thousands of bytes); and all hosts involved must be equipped with disk drives whose access rates are at least as high as the desired transmission rate. Even though this paper concentrates on FTP as one of the most popular protocols for le transfer, the main conclusions can be extended to any application which transfers les using TCP. The experiments serve to highlight the importance of window management when TCP is used in an ATM WAN environment. 6 Acknowledgements The experiments reported in this paper would not have been possible without the active contributions of Daniel Bromberg, at the NASA Jet Propulsion Laboratory, Suresh Bhogavilli and Javad Boroumand, at NASA Goddard Space Flight Center, and Kenneth Finnegan, at the Naval Research Laboratory. Sprint's Broadband Operations Center (BBOC) also provided valuable assistance monitoring these tests. The following trademarks are used in this paper: Digital UNIX, DEC Alpha AXP 3000 and DEC AN2 are trademarks of Digital Equipment Corporation; SunOS, Solaris and Sun SPARCstation 20 are trademarks of Sun Microsystems; FORE ASX-200 is a trademark of FORE Systems. UNIX is a registered trademark licensed exclusively by X/Open Company Ltd. References [1] Uyless Black. TCP/IP and Related Protocols. McGraw-Hill, second edition, [2] Smoot Carl-Mitchell and John S. Quaterman. Practical Internetworking with TCP/IP and UNIX. Addison-Wesley, [3] Douglas E. Comer and David L. Stevens. Internetworking with TCP/IP, volume III. Prentice Hall, [4] Joseph B. Evans et al. AAI ATM WAN Performance Tools, Experiments and Results. In DARPA ATM WAN Performance Workshop, June Available as http: // aai/ presentations/ workshop.