Transactions on Information and Communications Technologies vol 9, 1995 WIT Press, ISSN

Transcription

1 Transactions on Information and Communications Technologies vol 9, 995 WIT Press, ISSN Communications over ATM: communication software optimizing the L. Delgrossi*, G. Lo Re* "IBM European Networking Center, Wangerovstr., D-6900 Heidelberg, Germany ^Centro Studi sulle Reti di Elaboratori, Consiglio Nazionale delle Ricerche, Viale delle Scienze, 90 Palermo, Italy Abstract The last few years have seen network data rates skyrocket from a few Mbps to a Gbps or more. However, a lack of integration of the host-network interface, the operating system, and network protocols has resulted in end-applications that are able to exploit only a small fraction of the total bandwidth available for data transfer. This paper investigates the performances of the TCP/IP protocols over ATM networks. TCP/IP is the "de facto" standard for internetworking; the Asynchronous Transfer Mode is a standard networking technology for the broadband ISDN currently under development. The analysis activity included collecting traces with respect to different configurations of the relevant communication parameters involved. Introduction An advanced field in the computing world is represented by emerging multimedia applications comprising remote visualization, medical imaging, tele-conferencing, remote collaboration, and video distribution. Such applications are dependent upon the performance of the communication subsystem, which includes the communication network, network/host interconnection and the host itself. These applications require not only highspeed networks but also carefully engineered end-to-end protocols. These protocols have to be efficiently implemented within the constraints of the various existing operating systems and host architectures. For this kind of applications high-performance workstations are available today. Unfortunately, such machines have been optimized for computational tasks while the large band available with the most recent networking technology poses significant challenges to the communication subsystem. These challenges have in the last few years motivated, among the others, the design and development of new

2 Transactions on Information and Communications Technologies vol 9, 995 WIT Press, ISSN High-Performance Computing in Engineering communication protocols as ST-II [] and RSVP []. However, the world-wide diffusion of the TCP/IP protocols family and the not negligible time required for the complete acceptance and diffusion of the new standards, suggest an analysis of the TCP/IP protocols to study their behaviour within the new applications. Many articles and studies on TCP/IP performance have been published in the past, e.g. [3], [7]. The majority of them presents results of analysis and experimentations over the traditional networks for applications without specific demands on terms of high throughput. The introduction of fiber optics and the use of ATM technology [] resulted in high-speed transmission links and, as a consequence, the fastest paths are moving out of the domain for which TCP was originally engineered. This paper presents the first results of a study aimed to characterize the performance of TCP/IP protocols over an ATM network connecting a cluster of IBM workstations. The main objective of this activity has been to evaluate the performance of TCP/IP in terms of data throughput and end-to-end delay Indications on how to run TCP/IP over ATM can be found in [4] and [6]. A set of extensions to the TCP protocol to broaden the domain of its application to match the increasing network capability has been recently proposed by Jacobson [4]. This paper is organized as follows. Section presents an overview of ATM and TCP/IP implementation in the AIX kernel. Section 3 describes the scenarios considered in the performance study and shows the results of the various experiments and measurements. Finally, Section 4 concludes with somefinalremarks. Asynchronous Transfer Mode Asynchronous Transfer Mode (ATM) is the most compelling of the emerging networking technology due to the benefits it provides in terms of speed, scalability, and isochronous functionality. ATM has no throughput barriers and can support any data type It can scale with CPU performance increases, and is ideally suited for multimedia collaborative applications. Currently, ATM is evolving as standard networking technology for the broadband ISDN. The characteristics and features of ATM networks are different than those found in traditional LANs. In particular: ATM provides a Virtual Connection (VC) switched environment. VC setup may be done on either a Permanent Virtual Connection (PVC) or dynamic Switched Virtual Connection (SVC) basis. Data to be passed by a VC is segmented into 53 octet quantities called cells (5 octets of ATM header and 4 octets of data). The function of mapping user Protocol Data Units (PDUs) into the information field of the ATM cell and vice versa is performed by the ATM Adaptation Layer (AAL) When a VC is created a specific AAL type is associated with the VC There are four different AAL types, which are referred to individually as "AALT, "AAL", "AAL3/4", and "AAL5. The ATM Adaptation Layer used in this study is AAL5 Laubach [4] defines the standard values to be adopted for AAL5: the default MTU size for IP members operating over the ATM network is 90 octets; the LLC/SNAP

3 Transactions on Information and Communications Technologies vol 9, 995 WIT Press, ISSN High-Performance Computing in Engineering 3 header is octets, therefore the default ATM AAL5 protocol data unit size is 9 octets. CPCS PDU Trailer CPCS-PDU PAD CPCS-UTI CPI Length CRC up to ^ Figure : AAL5 CPCS-PDU Format. The TCP/IP AIX Implementation This section describes the TCP/IP sockets data processing at the sender and receiver sides. In AIX, applications that use TCP/IP protocols to communicate across networks use the socket abstraction. In the application layer, a program needing access to network communication has to open a socket and to specify the type of service required: connection-oriented over TCP or connection-less over UDP datagrams. The semantics of opening, reading, and writing to sockets are similar to those for manipulating files. As an application writes to a socket, the data is copied from user space into the socket send buffer in kernel space. The socket send buffer is made up of smaller buffers called mbufs, which are organized into linked lists. Once the data is copied into the socket send buffer, the socket layer calls the TCP layer, passing it a pointer to the linked list of mbufs (mbuf chain). The TCP layer allocates a new mbuf for its header information, and copies the data in the socket send buffer into the TCP header mbuf, if there is room, or into a newly allocated mbuf chain TCP then checksums the data, updates its various state variables, which are used for flow control and other services, and calls the IP layer, with the header mbuf now linked to the new mbuf chain (in the TCP layer, this chain is called a segment). The Internet Protocol (IP) layer determines which interface the mbuf chain should be sent to, updates and checksums the IP part of the header sent from TCP, and passes the chain (a packet to IP) to the Interface (IF) layer. The IF layer prepends the link (ATM) layer header information in the TCP/IP header mbuf, checks the format of the mbufs to make sure they conform to the device driver's input specifications, and then calls the device driver write routine. At the device driver layer, the mbuf chain containing the data is enqueued on the transmit queue and the adapter is signaled to start transmission or direct memory access (DMA) operations.at this point, control returns up the path to the TCP output routine, which continues sending as long as it has more to send. When all data has been sent, control returns to the application, which then runs asynchronously while the adapter transmits data. When the adapter has completed transmission, it interrupts the system and the device interrupt routines are called to adjust the transmit queues and free the mbufs that held the transmitted data. On the receive side, an application opens a socket, establishes a connection when required, and attempts to read data from it. If there is no data in the socket receive buffer, the socket layer causes the application to go to the sleep state (blocking) until data arrives. When packets are received by the adapter, they are transferred, using DMA, from the adapter into chain of mbufs resident in kernel space. The adapter the interrupts the system. The device driver software serves the interrupt starts protocol processing at the IF layer.

4 Transactions on Information and Communications Technologies vol 9, 995 WIT Press, ISSN High-Performance Computing in Engineering The IF layer removes the link header and enqueues the mbuf chain (done with pointers, no copy) on the IP input queue and schedules an off-level interrupt to do the IP input processing. When the off-level interrupt occurs, the IP input routine sequels the mbuf chain, checks the IP header checksum to make sure the header was not corrupted, determines if the packet is for this system, and if so, passes the mbuf chain to the TCP input routine. The TCP input routine checksums the TCP header and data for corruption detection, determines which connection this data is for, removes its header information, links the mbuf chain onto the socket receive buffer associated with this connection, and uses a socket service to wake up the application if it is sleeping as previously described. The socket layer copies the data stored in the socket receive buffer into the application's buffer and then frees the mbuf chain Execution control then passes back to the application. 3 Performance Analysis For our work we used two different measurements tools, namely AIX Trace and ZM4 The AIX operating system trace facility is a powerful system observation tool. The trace facility captures a sequential flow of time-stamped system events, providing a fine level of detail on system activity. Events are shown in time sequence and in the context of other events.the basic idea of ZM4 approach is to look at the execution of the system to be analysed (i.e. the object system) as a stream of "events". Events are significant points in the system, significant in the sense of the investigation. Events are generated by event-statements merged into the object code. The event-statement writes a - byte information to the PS/ parallel port with this one byte being the event id. During monitoring an external monitoring device (a PC-AT card called DPU) is connected to the parallel port catching the event id, adding a time-stamp to it, and storing the resulting "event record" in a FIFO memory on this monitoring card. The PC-AT reads continuously trace portions from that FIFO and writes them on the disk. The testing environment adopted comprised two IBM RISC/6000 workstations equipped with TURBOWAYS 00 ATM Adapters. The physical bit-rate of the ATM adapter card is 00 Mb/s and the complete ATM and AAL cell processing is done in hardware and firmware on board The adapter features a dedicated Intel i960 processor plus a specialized chip set to handle Segmentation And Reassembly (SAR) of ATM cells. A RISC/6000 model 50 workstation has been used as sender and a model 360 as receiver. Since we expected the communication side at the receiver to be critical, we decided to use a more powerful machine at this side. Both workstations run the IBM AIX 3..5 operating system. The connections over ATM have been set so that only Permanent Virtual Connections (PVC) have been used. During the performance evaluation, the network and the workstations were completely dedicated to this task. In the following, we describe the various experiments carried out. 3. Throughput Evaluation The achievable throughput is certainly one of the most interesting factors to be evaluated. The overall throughput depends on several parameters, including the size of the buffers used in the application and at the socket layer, and on the protocol adopted. Therefore, we decided to experiment with data transfer over both TCP and UDP, and to collect data relative to various buffer sizes.

5 Transactions on Information and Communications Technologies vol 9, 995 WIT Press, ISSN High-Performance Computing in Engineering 5 In this experiment we measure the total throughput at the TCP receiver obtained varying the socket buffer size at the sender and receiver. The buffer size used by the application is constant and equal to 64K. Table reports the traffic produced on a single PVC between two RISC/6000 workstations. Table : TCP Throughput (sockets buffer size) Mb/s * 04 45* As it can be observed, the throughput increases with the buffer size until a threshold is reached at 5K. For buffer sizes greater than 5K, the values obtained are unpredictable because of a large variance. The * symbol in the table indicates that the correspondent value is a "peak" rather than an "averase" value ^ In next experiment we measured the throughput as a function of the application buffer size. In light of the values obtained with the previous experiment, the socket buffer size was fixed at 5. The not optimal behaviour for sizes lower than 4 is explainable because the width of the AlXmbufi is 4 For application buffer sizes lower than this value the full mbufs capacity is not exploited, and this determines the performance downgrading. Table : TCP Throughput (application buffer size) Mb/s 7 : 4» * For application buffer sizes ranging into [4,... ], the diverse throughput values tend to be constant. This means that the receiver and the sender are well synchronized with regard to the transmission rate and not frequent changes of the TCP window size occur. In this range we can also observe a logarithmic increasing of the throughput related the to buffer size hor sizes of the application buffer higher than the behaviour of the throughput is floating during the time. The reason can be found in an a too fast data sending rate at the origin resulting in a floating TCP window f J^ ^^G experiment has been replicated using the UDP protocol instead ot KP and the results obtained are showed in Table 3. The UDP protocol a lows to send datagram whose maximum length is ^ bytes How the table clearly shows the best throughput is achievable for values multiple of the PDU size. It is corresponding to the maximum number of mbufs filling the MTU (Maximum Transfer Unit). It is clear from the previous tests that for small block sizes, the software is the limiting factor to the system performance. Table 3: UDP Throughput Mb/s /.4 / (-B)

6 Transactions on Information and Communications Technologies vol 9, 995 WIT Press, ISSN High-Performance Computing in Engineering Small block sizes force the application to make frequent system calls, which force the AIX operating system to context switch frequently. Larger block sizes reduce the per-byte software overhead, since the system calls are amortized over a larger data transfer. As this overhead becomes relatively smaller, the data transfer dominates the performance and, since the software does not participate in actual transfer to and from the device, the hardware becomes the limiting factor. This can be seen by examining the relative performance gain for each doubling in block size. The performance is almost doubled as block size is increased from 64 bytes to, but the increase from to 6 gives less than a 0% gain 3. Delay Evaluation The results presented in this section have been obtained by means of the AIX TRACE facility and the ZM4 tool previously described. The unavailability of source code for AIX 3. allowed us only to monitor the user-level programs and the device driver level routines. We consider the whole execution of the TCP/IP protocols as a black box providing frames for the device driver level. For these reasons we placed the monitoring probes at the entrance and exit of the TCP/IP protocol code, as shown in Figure. Application Send Socket TCP/IP Network Receive Socket TCP/IP Network Figure : Probe Location The following experiment aimed at the measurement of the transmission delay when using the TCP protocol. The experiment was setup as follows: the sender transmits over the network variable sizes frame, as indicated by the table header For sizing exceeding the MTU, the communication protocols partition Table 4: TCP Delay dd-first-seud dd-first-recv dd-sec-send # segments dd-lasl-seud dd-last rec\ end recv

7 Transactions on Information and Communications Technologies vol 9, 995 WIT Press, ISSN High-Performance Computing in Engineering 7 the data into multiple segments. The number of such segments is shown in the table. Times in the table are expressed in ^s. The first row shows the time when the first Protocol Data Unit (PDU) is sent. The second row shows the time when the first PDU is received. Note that these delays are measured on different workstations and the use of the ZM4 tool guarantees the correct synchronization. In the next row; the time of the successive send is reported. Finally, in the last row, the user level end-of-receive time is shown. The generic TCP/IP behaviour in case of multiple segments is schematically represented in Figure 3. Values for the tj variables depend on the TSDU size. The factor that contributes most to the overall delay is the t-tl interval. This interval is due to the acknowledgment reception for thefirstsegment transmission. The reason of user space device driver t t3 4 5 t6 Figure 3: TCP message transmission to start transfer tl dd-first send t dd-second-send t3-t4 other dd-sends t5 dd-last-receive t6 end-transfer this initial delay could be attributed to the scheduling activity of the receiving host The arrival of thefirstsegment causes the wake-up of the blocked process associated with the connection and the consequent context switch []. The experiment explains the relationship between the throughput and the size of the application buffer. The constant overhead due to the first acknowledgment determines better performance for big sizes of the application buffer. This is because the time spent waiting for thefirstacknowledgment is amortized if the successive message transfer is long enough. The same experiments have been replicated using the UDP protocol As expected, the absence of the waiting period for the acknowledgments produces noticeable benefits on the time necessary to send a message using UDP datagrams. [ dd-first- sen d dd-first-recv dd-sec-seiid # segments (Id-last-send dd-last reev end ree\ Table 5: UDP Delay r I j " f! ~ ' 64 (-B) i_ 33 i \ 744.3

8 Transactions on Information and Communications Technologies vol 9, 995 WIT Press, ISSN High-Performance Computing in Engineering The generic UDP/IP behaviour in case of multiple segments is schematically represented in Figure 4. As expected, the t-tl interval is no more critical because UDP makes no use of acknowledgments. to start transfer user space ^ ^ tldd-firstsend t dd-second-send t3-t4 other dd-sends device driver h t5 dd-last-receive to Iltt3 4 t5 t6 Figure 4: UDP message transmission t6 end-transfer 4 Conclusions The performance of the TCP/IP protocols implementation over a simple ATM environment was studied through monitoring of the connected end hosts. Numerical results showed a strong dependency between message size and socket buffer size on one hand and throughput/delay on the other As far as the throughput is concerned the best performance is achievable for big block sizes However, for many sources of traffic, a 64 blocks size may be unrealistic. As soon as the physical network bandwidth is in Mb/s range, the physical network is not the bottleneck for the communication. The network delay is in the jus range, while the protocol processing delay in the ms range From our experience, it grows the necessity of workstations with higher performance access to computational resources and to the memory References. Braden, B & Zhang, L RSVP: A Resource ReSerVation Protocol, Connexions, The Interoperability Report, 994,,-7.. Herrtwich R G & Delgrossi L The ST-II Protocol for Multimedia Communication, Connexions, The Interoperability Report, 994,, Jacobson V, Braden R, Borman D TCP Extensions for High Performance,7#r 7JJ, May Laubach, M. Classical IP and ARP over ATM, RFC 577, January Luckenbach, T. Performance experiments within local ATM networks, pp , fmc6w/w#s o/a7'or <S TV, Geneva, Oechsle, R & Graf, M. The Internet Protocol Family over ATM, IBM European Networking Center Technical Report N , Papadopoulos, C & Parulkar, G.M. Experimental Evaluation of SUNOS IPC and TCP/IP Protocol Implementation, IEEE ACM Transactions on Networking 993,, Stuttgen, H J. Network Evolution and Multimedia Communication, IBM European Networking Center Technical Report N , 994.