Improving PVM Performance Using ATOMIC User-Level Protocol. Hong Xu and Tom W. Fisher. Marina del Rey, CA

Transcription

1 Improving PVM Performance Using ATOMIC User-Level Protocol Hong Xu and Tom W. Fisher Information Sciences Institute University of Southern California Marina del Rey, CA Abstract Parallel virtual machine (PVM) software system provides a programming environment that allows a collection of networked workstations to appear as a single concurrent computational resource. The performance of parallel applications in this environment depends on the performance of reliable data transfers between tasks. In this paper, we improve PVM communication performance over the ATOMIC LAN, a gigabit per-second local area network. This is achieved by separating the PVM data-transmission path from the PVM control-message path and transmitting PVM data messages over a user-level application programming interface (API) provided by the Myrinet ATOMIC interface. The Myrinet-API, although signicantly faster than the TCP/IP kernel stack, does not provide reliable communication. Therefore, a user-level protocol has been developed based on the Myrinet-API to oer reliable sequenced packet delivery. This protocol provides faster data transfers between PVM tasks over the ATOMIC LAN. Performance results obtained at USC/ISI demonstrate that our version of PVM can reach up to 140 Mbps throughput, which is 94% of the achievable network bandwidth over the ATOMIC LAN. 1 Introduction Over the past few years a rapid increase in the performance of microprocessors as well as in the speed of local area networks (LAN) has made workstation clusters a viable platform to solve computation-intensive problems. PVM [1, 2], a message passing based software system, provides a programming environment that allows a collection of networked workstations to appear as a single parallel computational resource. PVM currently provides facilities for process control and data communication based on the TCP/IP network protocol suite [3]. By taking advantage of popular TCP/IP implementations, PVM has become a This work is supported by the Advanced Research Projects Agency through Ft. Huachuaca contract #DABT63-93-C-0062 entitled \Netstation Architecture and Advanced Atomic Network". The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ocial policies, either expressed or implied, of the Department of the Army, the Advanced Research Projects Agency, or the U.S. Government. widely used de facto standard for distributed computing on high-performance workstations interconnected through a variety of high-speed networks. The performance of PVM applications greatly depends on the performance of reliable data transfers between workstations. There are two basic classes of PVM applications: task parallel applications and data parallel applications. For task parallel programs, communication latency of synchronization messages is critical to the overall application performance. In data parallel programs, data are partitioned and distributed to all host computers in the PVM conguration. The same operations are performed on each partition of data by PVM processes running on each host. PVM processes then communicate with one another to exchange values of updated data. In such situations, bulk data transfers frequently occur between the workstations, which can result in performance bottlenecks. Minimizing the communication overhead involved in PVM data transfers for both task parallel and data parallel applications requires an especially high-throughput low-latency form of reliable communication. The ATOMIC [4] LAN is a high-speed network that oers 640 Mbps bandwidth. The ATOMIC workstation cluster at USC/ISI consists of a collection of Sun SPARCstation 20s interconnected through the ATOMIC LAN. At each host computer, a user-level process can access the ATOMIC LAN through either the TCP/IP kernel network protocol stack or the user-level ATOMIC Myrinet-API. The performance of the general-purpose TCP/IP stack is limited due to the overhead of data copies and software layers in the kernel. On the other hand, the Myrinet-API exhibits higher throughput and lower latency, since the Myrinet-API allows any user-level process to directly access the network without going through the kernel. PVM data communication is currently implemented using TCP/IP. Thus, the performance of PVM data transfers over the ATOMIC LAN is limited by the performance of TCP/IP. In this paper, we address how to improve PVM task-to-task data communication performance over the ATOMIC LAN by separating the PVM data-transfer path from the PVM control-message path. The key point behind our work is to replace the critical path of PVM service by a fast user-level protocol based on the high-bandwidth, lowlatency Myrinet-API. This ATOMIC transport-layer

2 protocol (ATP) provides reliable, sequenced packet delivery, point-to-point communication for faster data transfers between PVM tasks on dierent workstations. We observe a signicant increase in the performance of PVM task-to-task data communication using the proposed ATP. The performance results show that our version of PVM can reach up to 140 Mbps throughput, which is 94% of the achievable network bandwidth over the ATOMIC LAN. We further conjecture that performance of future distributed computing applications using our version of PVM over the ATOMIC LAN will also be greatly increased. The remainder of the paper is organized as follows. Section 2 describes the ATOMIC networking architecture. Section 3 decomposes the PVM service in order to reimplement the critical path of PVM task-to-task data communication using the fast user-level protocol over the Myrinet-API. Section 4 presents our implementation, including the ATP development, as well as our scheme that interleaves memory copies with DMA operations. Section 5 gives the performance results obtained from the USC/ISI ATOMIC workstation cluster. Section 6 discusses related work and Section 7 mentions several areas of future research. Section 8 presents the conclusions. 2 ATOMIC Networking Architecture The ATOMIC LAN is a high-speed network that oers 640-Mbps bandwidth at an inexpensive per-host cost. It is a switch-based local area network composed of host interface boards and network switches. A network switch in the ATOMIC LAN is a nonblocking crossbar in which packets are simultaneously relayed from distinct incoming ports to distinct outgoing ports with no internal channel collisions. Packets competing for the same outgoing port are arbitrated in the round-robin fashion at a per-packet level. The ports on both the switch and the host interface board are bi-directional and are connected to each other by a twisted-pair link with two independent uni-directional channels. Switch ports and host interface ports can be connected to one another in arbitrary topologies including trees, meshes, etc. Packet transmission over the ATOMIC LAN has a very low error rate. Packet transfer tests of over 1,000 Terabits ( bits) over the prototype ATOMIC LAN [4] at USC/ISI resulted in no bit errors and no lost packets. The ATOMIC LAN employs a source-routed, cutthrough packet switching technique. In cut-through routing, the header bytes in the packet determine the route, and the remaining data bytes follow in a pipeline fashion. At each intermediate switch, the packet can be advanced into the required outgoing channel as soon as the header is received and decoded. If the header encounters a busy channel, the rest of the packet will hold and block all upstream channel(s) in the routing path. Unlike store-and-forward switching, cut-through routing requires no packet buering at each intermediate switch and thus provides low packet transmission latency. When a source host sends multiple packets to a destination host, cut-through routing forces packets arriving at the destination to be in order, provided all packets follow the same route. Host interface boards connect hosts to the network. Each host interface consists of a processor with a dualport memory. The dual-port on-board memory is accessible to both the on-board processor and the host computer. The host computer can read/write onboard memory through programmable I/O (PIO) and data can be moved between on-board memory and the host computer main memory via direct memory access (DMA) that is supported on the host interface board. The control program (CP) running on the on-board processor can be loaded from the host computer using PIO. Checksum hardware is also provided on the interface board so that the link-layer checksum of every packet can be computed in hardware while the packet is being either sent or received. Host Computer Interface Board possible data communication paths User space Kernel UDP IP TCP device driver user process Multi interface CP Myrinet API I/O Bus Figure 1: Dual-interface CP for IP and API trac The CP on the host interface is designed to allow multiple processes on the host computer to pass data through to the on-board processor at the same time (that is given no two hosts are accessing overlapping sections of on-board memory). The CP is also termed multi-interface CP since it provides an independent application device interface, also known as the application device channel in [5], for each process on the host computer to directly communicate with the host interface. Figure 1 shows a dual-interface CP that is able to support a host simultaneously running packets over both IP and the Myrinet-API. In terms of both observed bandwidth and latency, communication over TCP/IP performs worse than communication over the Myrinet-API. While TCP/IP is a kernel networking protocol stack, the Myrinet- API is a user-level communication library that allows a user-level process to directly access and communicate with the host interface board. Using the dualinterface CP provided by Myricom, the values of peak bandwidth for TCP/IP, UDP/IP, and the API are 45 Mbps, 55 Mbps, and 150 Mbps, respectively, measured on Sun SPARCstation 20/50s (running SunOS 4.1.3)

3 interconnected through the ATOMIC LAN. The original prototype for the ATOMIC LAN was developed by USC/ISI using Caltech's Mosaic chip [6]. The network switches, hosts interfaces, and system software are currently provided by Myricom [7] as commercial products. Although Myricom has made many improvements in both hardware and software based on the ATOMIC prototype, its commercial product uses the original ATOMIC networking architecture. Researchers at USC/ISI are currently using the Myricom product and continuing ATOMIC research [8] in supporting the ATOMIC LAN outside the lab environment for daily computing and networking needs. 3 PVM Service Decomposition PVM allows a collection of host computers to appear as a single virtual distributed-memory parallel machine. The PVM system is composed of a daemon, known as pvmd in [2], and a library of PVM interface routines. Pvmd is a user-level daemon process running on each host computer that is a part of a given PVM conguration. PVM interface library routines are used by each application process for data communication and process control. A PVM application process is called a PVM task, or task for short. On each host computer, the pvmd serves as a message router that multiplexes messages from local task(s) to the network and demultiplexes messages from the network to local task(s). Pvmd also acts as a coordinator responsible for authentication and process control. Application computational load is distributed among tasks running on host computers. The pvmd does not do any computation. There is only one pvmd running on each host computer while there may be multiple tasks running on the same host. Host Computer Interface Board Kernel TCP path default message path User space UDP pvmd IP TCP device driver PVM task Multi interface CP Myrinet API Figure 2: General PVM service model I/O Bus As shown in Figure 2, in the general PVM service model [2], a message sent from a local task to another task on another host is rst, by default, routed to the local pvmd. The message is then forwarded by the local pvmd to the pvmd on the remote host. The message is then nally transferred to the destination task by the remote pvmd. Communication between a task and the local pvmd on the same host is achieved by using TCP/IP. The pvmds on dierent hosts communicate with one another through a built-in PVM reliable communication protocol using UDP/IP. Routing messages through pvmds, however, introduces a significant overhead due to extra message copying between PVM tasks and their respective PVM daemons. To improve message passing performance, a task is allowed to talk to another remote task directly through a TCP/IP connection. Such a direct message passing scheme is known as message direct routing over TCP in [2], or TCP path for short. No pvmd is involved in message passing using the TCP path. In the USC/ISI ATOMIC workstation cluster, communication latency is an average of 1.2 ms and the peak communication bandwidth is 11 Mbps when messages take the default route through respective pvmds. The latency is decreased to an average of 1.1 ms and the peak bandwidth is increased to 30 Mbps when messages take direct routes from task to task using TCP/IP. Note that communication over TCP/IP can reach a peak bandwidth of 45 Mbps in the ATOMIC LAN. PVM's task-to-task peak bandwidth is lower due to the extra memory copies for packing the data on the sender site and unpacking the data on the receiver site. Host Computer Interface Board Kernel ATP path default message path User space UDP pvmd IP TCP device driver PVM task Multi interface CP ATP Myrinet API I/O Bus Figure 3: Decomposed PVM service in ATOMIC Since the Myrinet-API peak bandwidth of 150 Mbps in the ATOMIC LAN is much higher than the TCP/IP peak bandwidth of 45 Mbps, the performance of PVM task-to-task data transfers can be signicantly improved by forwarding messages over the Myrinet-

4 data transfer flow Task 0 Task 1 PVM task data space PVM task data space on interface board memory host user space memory I/O bus pvm_send() pvm_recv() atp_send() atp_recv() memory copy head recv_ request_ uncachable memory for PVM buffer tail application device interface 0 head tail DMA send_ request_ send_packet_ recv_packet_ DMA memory copy uncachable memory for PVM buffer head recv_ request_ tail application device interface 1 pvm_send() pvm_recv() atp_send() atp_recv() head tail send_ request_ Figure 4: Packet ow on the host machine API at the user level, instead of through the kernel over TCP/IP. As shown in Figure 3, we propose communicating data over the Myrinet-API to speed up data transmission performance between PVM tasks. We refer to this approach as using the ATP path. The basic idea is to separate the PVM data-transfer path from the PVM control-message path. As a result, a fast data communication path is provided between PVM tasks while maintaining pvmd functionality for process control. In the next section, the ATP protocol is presented. 4 User-Level Protocol Implementation This section describes the ATP protocol implementation which oers reliable point-to-point communication to PVM tasks. The rst two subsections describe the underlying packet ow on the host machine as well as how the overhead of memory copies is reduced. The last subsection presents the details of the protocol. Our implementation uses the same task-to-task communication modes as specied in PVM [2]. A receive (pvm recv()) is a blocking receive which returns only if the data has been received. A send (pvm send()) is a blocking send which returns when all data has been acknowledged by the receiver and the PVM buer that holds unacknowledged data is free for reuse. 4.1 Packet-Flow on the Host Machine The basic communication structure between the user-level ATP protocol and host interface is shown in Figure 4. In this paper, a message refers to a data transfer at the PVM programming level. Conceptually, the size of a single message is unlimited. A packet refers to a data transfer at either the Myrinet-API level or the host interface level. The size of a single packet is limited to 8K bytes, the maximum transfer unit (MTU) for the ATOMIC LAN. API-level packets and interface-level packets use the same MTU. Nei-

5 ther segmentation nor concatenation takes place when packets are DMA transferred between the PVM buer and on-board memory. However, segmentation (concatenation) occurs when a message, which is larger than the MTU, is copied to (from) PVM user space from (to) the PVM buer. The PVM buer space that is used to buer send and receive packets is declared as uncachable in order to avoid cache coherence problems when this memory is overwritten by DMA upon receiving packets. Both the host CPU and the on-board DMA unit can access the PVM buer. After an outgoing packet has been DMA transferred to the host interface, it is stored in a send packet in on-board memory and waits to be delivered into the network. Similarly, after an incoming packet has been received by the host interface, it is stored in a recv packet and waits to be DMA transferred to the host computer. The send packet and the recv packet are used only by the on-board processor and are not accessed by the host CPU. The only data structures shared between the host CPU and the on-board processor are the send request and the recv request. The library routines atp send() and atp recv() are responsible for providing reliable message sending and receiving Ẇhen a packet is sent through atp send(), the host CPU appends a request at the tail of the send request. The request contains only the memory pointer to the particular packet in the PVM buer and the packet's size, as opposed to the packet itself. This request will be honored by the on-board processor after all previous requests have been processed. When it is time to service the request, the on-board processor initiates the DMA operation to transfer the packet data to the send packet using the memory pointer and the data size contained in the request. When the DMA operation is done, the request is removed from the send request by the on-board processor. The send request is implemented as a circular with a head pointer and a tail pointer. The head pointer is modied only by the on-board processor and the tail pointer is modied only by the host CPU. As a result, requests in the send request are serviced in FIFO order. The host CPU has to wait when the send request is full, and the on-board processor has to wait when the is empty. The dual-port on-board memory guarantees the atomicity of singleword load and store operations, thus the consistency of concurrent accesses is ensured. When a packet receive operation is issued in atp recv(), the host CPU appends a request at the tail of the recv request. Besides the expected packet size and the memory pointer (pointing to the allocated PVM buer reserved for the expected packet), the request also contains the source PVM task ID and the PVM message tag [2] used to identify the requested PVM message. After the packet arrives at the host interface, the on-board processor will verify the source task ID and the message tag, and initiate the DMA operation to transfer the packet to the reserved memory. Finally, the request is removed from the by the on-board processor. For the recv request there is no FIFO ordering of requests. Instead, each request may be examined in order to match an arrived packet since packets can arrive from dierent PVM tasks out of order. A packet that does not match any receive request is dropped. In the multi-interface CP, each application device interface is associated with an independent send request and recv request. As shown in Figure 4, on the local host computer, two PVM tasks (task 0 and task 1) are assigned independent application device interfaces (interface 0 and interface 1) to the host interface. Only the owner task can access its own pair of s. On each host, an application device interface is statically assigned to a PVM task during the task initialization. This task-interface assignment is known to all other PVM tasks. 4.2 Reducing Memory Copy Overhead As shown in Figure 4, a section of uncachable memory space serves as the PVM buer which is physically disjoint from the rest of PVM task data space. On Sun SPARCstations, the PVM buer has to be uncachable, otherwise cache coherence problems will occur when the PVM buer is overwritten by DMA upon receiving packets and the corresponding cache lines would contain stale data. By using a separate PVM buer, data has to be moved between PVM task space and the PVM buer, which introduces the overhead of an extra memory copy. host user space memory I/O bus on board memory DMA PVM task data space step 1 memory copy step 2 PVM buffer step 3 step 2 step 5 step 4 step 3 step 4 send packet Figure 5: Interleaving memory copies with DMA operations on the sender An alternative approach, which can avoid the memory copy, is to make memory in PVM task data

6 space uncachable. Then, data could be DMA transferred to/from the host interface directly without going through any PVM buer. However, this approach causes another problem. Computation speed will be dramatically reduced (20% on Sun SPARCstation 20/50) if the referenced data is allocated in uncachable memory. It is non-trivial and inappropriate for application users to manipulate cachable and uncachable data space at the programming level in order to maintain the proper computation speed. For this reason we have chosen to implement the memory copy approach. For large-sized messages, the overhead of memory copies can be greatly reduced by interleaving them with DMA operations. As shown in Figure 5, a largesized message is segmented into multiple packets after atp send() is called. Instead of copying the whole message into the PVM buer, only the rst MTU bytes are copied and lled into the rst packet in step 1. In step 2, while the rst packet is being DMA transferred by the host interface, the next MTU bytes are copied and lled into the second packet. On a Sun SPARCstation 20/50, a memory copy of 8K (MTU) bytes takes about 126 microseconds while a DMA transfer of 8K-bytes takes about 210 microseconds over the Sun SBus. Thus, the memory copy of the second 8Kbytes into the PVM buer can take place in parallel with the DMA transfer of the rst packet into the send packet. The same procedure repeats until the last packet is DMA transferred into the on-board memory. Except for the rst packet, the overhead of successive memory copies can be dramatically reduced by using this interleaving optimization. The overhead of memory copies at the receiving end is also signicantly reduced by interleaving them with DMA operations. 4.3 Reliable Data Protocol ATP is a user-level, reliable, sequenced packet delivery protocol which has been developed for the ATOMIC workstation cluster to be used directly with the Myrinet-API. By taking advantage of ATOMIC LAN, our protocol implementation is less complicated and oers high-performance. A Stop-and-Wait Protocol A PVM message is identied by the PVM task ID and the PVM message tag. If the message size is larger than the MTU over the ATOMIC LAN, it is segmented into multiple packets each of which is further identied by a unique sequence number. The TAIL bit in the packet header will be set for the last packet. Figure 6(a) shows the basic idea of how the protocol works. For simplicity, only the sequence number of each data packet is shown in Figure 6. In Figure 6, we further assume that all packets shown in each scenario are dierent segments of a single PVM message. The sender atp send() transmits n packets in a row to the receiver atp recv(). Atp send() then stops and waits for an acknowledgement (ACK) that the receiver has indeed received the data. After receiving all n packets, atp recv() at the remote end sends an ACK back to atp send(). When the ACK reaches atp send(), another n packets may then be transmitted to the remote atp recv(). We refer to this approach as bundle-sending and the number \n" as the bundle size. Each ACK consists of two elds. The rst eld contains the sequence numbers for those packets which are not received by atp recv(). The second eld contains the value of the bundle size oered by atp recv(). In Figure 6(a), no packet is lost. Thus, the rst eld in each ACK is empty and the second eld is n, the size of the next bundle to be sent. Error Checking and Packet Loss Recovery The host interface provides a hardware checksum. If the host interface receives a corrupted packet, it will drop the packet without informing the host computer. Therefore, if a packet reaches the user-level protocol, it is guaranteed to be a good packet. No software checksum is carried out in ATP. In the ATOMIC LAN, packets from the same source arrive at the same destination in the order they were sent (provided that all packets use the same route). Thus, atp recv() can easily tell whether any packet is lost and which packets are lost by observing the sequence number of the last received packet of the current bundle being sent. This information is included in the ACK and is sent back to atp send(). This scheme is known as selective acknowledgement [9]. For example, in Figure 6(b), ve packets are sent from atp send() to the remote atp recv(). Packets 2 and 3 become lost. As soon as packet 4 is received, atp recv() realizes the loss of packets 2 and 3 (This is because we are able to assume in the ATOMIC LAN that no packets will arrive out of order). Atp recv() records which packets were lost and continues receiving packets untill either the last packet is received or (if the last packet has also been dropped) atp recv() times out. At this point, in Figure 6(b), atp recv() sends back a \selective acknowledgement" which tells atp send() to restrict its next bundle size to 2, and to retransmit packets 2 and 3. Atp send() will then, upon receipt of the selective ACK, carry out the necessary retransmissions and wait for another ACK. This process will repeat until the receiver has positively acknowledged all expected data. In Figure 6(b), atp send() retransmits packets 2 and 3. After these two packets have been successfully received, atp recv() then adjusts its expected bundle size to 5 and informs atp send() via the next ACK. The ATOMIC LAN has a very low error rate. The ACKs are seldom corrupted across cables or network switches. However, an ACK could be dropped by the sender host interface if the sender host computer is overloaded by other trac being received. In our implementation, if an ACK becomes lost, atp send() will timeout and retransmit all n data packets. This retransmission scheme can potentially degrade the performance since the entire bundle of packets are retransmitted. To avoid dropping ACKs, a special buer could be created in on-board memory for receiving out-of-band data including ACKs sent from atp recv().

7 atp_send() atp_recv() As a result, the sender host interface will have room to store ACKs even if its regular recv packet is full. bundled packets bundled packets bundled packets starting point atp_send() timeout bundled packets atp_send() ack[ ] [n] ack[ ] [n] ack[2,3] [2] ack[ ] [5] packet 1 packet 2 packet n packet n+1 packet 2n (a) bundle sending packet 1 packet 2 packet 3 packet 4 packet 5 packet 2 packet 3 (b) packet retransmission READY [n] FIN packet 0 packet 0 packet 0 packet 1 packet n 1 Lost packet Good packet Good ACK atp_recv() atp_recv() (c) transmission startup and termination Figure 6: Reliable data protocol starting point Congestion Control and Flow Control When congestion occurs in the ATOMIC LAN, Myrinet link-layer back-pressure hardware will prevent senders from injecting more packets into the network [10]. Therefore, our protocol does not provide extra software support to deal with congestion control over the ATOMIC LAN. Flow control in the ATOMIC LAN is handled by atp recv() in how it controls the bundle size it advertises to atp send(). In situations where the receiver workstation is experiencing either heavy CPU load or heavy load at its network interface (many packets being sent to it), or both, atp recv() will begin to notice packets being dropped. At this point, each ACK generated by atp recv() will include the sequence numbers for all dropped packets as well as smaller bundle sizes for atp send() to use. In eect, the bundle size will decrease to a certain point (dependent upon how loaded the receiver actually is) and gradually stabilize to a size representing the number of packets the receiver machine can handle without dropping. Transmission Startup and Termination In the distributed PVM environment there are no assumptions made regarding the execution order of the sender and receiver. Either the receiver atp recv() can start rst or the sender atp send() can start rst. Our protocol works well in either case. When atp recv() is called, it immediately generates an initial READY packet and sends (or broadcasts) it to all host machines that the receiver is interested in. This READY packet will contain the bundle size advertised by atp recv(). The READY packet is important so that any sender that has been polling the receiver (as mentioned below) can be informed immediately to begin sending. For example, in Figure 6(c), atp send() starts earlier than atp recv(). The READY packet from atp recv() immediately informs atp send() which is busy-waiting for the receiver's signal. As shown in Figure 6(c), a bundle-send begins as soon as the handshake has been established. If there is no immediate response from any valid senders, atp recv() will busy-wait. When atp send() is called, as shown in Figure 6(c), it sends only the initial packet (packet 0) to atp recv() and then busy-waits for an ACK. This initial packet is an actual data packet that contains the rst MTU bytes of the sending message (or the whole message if the message is smaller than the MTU). If atp recv() has been ready, it will send back an ACK containing the advertised bundle size. In Figure 6(c), atp recv() is not ready yet. Thus, this initial packet will be dropped by the receiver host interface. Atp send() will eventually timeout and retransmit the initial packet. This \polling" process repeats until atp recv() is ready and responds by sending back the initial READY packet.

8 A tradeo exists regarding the length of the sender's timeout value. A smaller timeout will allow faster recovery if the READY packet sent by atp recv() is lost. However, the smaller the timeout is, the greater the amount of network trac will be generated by atp send() untill atp recv() eventually responds. To conclude transmission atp send() will set the TAIL bit in the last data packet of the entire message. Upon receipt of this last packet, atp recv() will respond by sending a FIN(ish) packet back to atp send before returning. As soon as the FIN packet arrives at the sender, atp send() will also return. 5 Performance Results This section shows the performance results obtained in testing PVM task-to-task communication using TCP/IP and ATP over the ATOMIC workstation cluster. In particular, all measurements have been made between two Sun SPARCstation 20/50s running SunOS These two machines are connected directly to each other through a Myrinet switch. In our experiments, the receiver pvm recv always starts earlier than the sender pvm send. In Figures 7, 8 and 9, the term \TCP path" refers to data transfers through the kernel over TCP/IP, and the term \ATP path" refers to data transfers over the Myrinet-API using ATP. Latency (sec) TCP path 2 ATP path Message Size (bytes) Figure 7: Latency comparison Figure 7 plots latency values for PVM task-to-task communication using the TCP path and the ATP path. The ATP path latency is only 50% of the TCP path latency. Figure 8 compares the bandwidth values of PVM task-to-task communication using the TCP path and the ATP path. The message sizes measured are those equal to and smaller than the 8k-byte MTU. For these small messages, the bundle size is always one with no interleaving of memory copies and DMA operations. However, even without using bundle sizes greater than one or the interleaving optimization, communication using the ATP path consistently outperforms communication using the TCP path. In Figure 9, the \raw API" bandwidth refers to the bandwidth of user-level data transfers using the Bandwidth (Mbps) ATP path 4 TCP path Message Size (Kbytes) Figure 8: Bandwidth comparison for small messages unreliable raw Myrinet-API. The \raw API" bandwidth of 150 Mbps is obtained by DMA transfers of data directly to/from user process data space with no memory copies taking place on either host. This raw API bandwidth oers an upper bound for any user-level protocol using the Myrinet-API. Figure 9 shows that, when the message size increases up to 256 Kbytes, the bandwidth of reliable communication over the Myrinet-API can reach up to 94% of the \raw API" bandwidth. The experimental results demonstrate that the interleaving optimization technique greatly reduces overhead due to memory copying. Without the interleaving optimization, the bandwidth of reliable communication over the Myrinet-API is only 66% of the \raw API" bandwidth. Bandwidth (Mbps) Raw API ATP path 4 ATP path w/o Interleaving + TCP path Message Size (Kbytes) Figure 9: Bandwidth comparison for large messages Our performance results show that the approach of using ATP over the Myrinet-API achieves lower latency and higher bandwidth than using TCP/IP. Thus, we conjecture that our version of PVM will

9 greatly increase the overall PVM performance over the ATOMIC LAN for bandwidth-sensitive data parallel applications as well as latency-sensitive task parallel applications. 6 Related Work This section only discusses the related work regarding reliable communication over the ATOMIC LAN. The related issues of implementing user-level protocols on other network architecture can be found in [11, 12, 13]. The host interface is an alternative place to implement a reliable data protocol. An RPC-based (remote procedure call based) reliable protocol [14] has been implemented in the host interface CP in order to minimize the per-packet overhead and optimize communication latency for small messages. This protocol was specicly designed for device control over the ATOMIC LAN in the USC/ISI Netstation Project [15]. This protocol generates a very low RTT (round trip time) for device control command streams. The major reason driving us to implement reliable communication in the host computer is the limitation of the on-board processor speed and on-board memory capacity. With a 128K-byte RAM and 25 MHz LANai processor [10], a Myrinet ATOMIC interface can only aord to support a very limited number of application device interfaces without signicantly degrading its performance. Thus, an application device interface may have to be shared by multiple processes among many various user applications including distributed-computing and real-time applications. There are no assumptions regarding characteristics of ATOMIC network trac ow. The same application device interface (to the host interface) can be shared, for example, by a PVM task and a teleconferencing application which does not require reliable data transmission. On the other hand, the on-board LANai processor could be overloaded if reliable communication is supported in the host interface for data transmitted over each application device interface. For this reason, we think that reliable communication is applicationbased and such a protocol should be implemented in the host computer at the transport layer. 7 Future Work Similar to task-to-task communication, pvmd-topvmd communication can also be implemented using our user-level reliable data protocol in the ATOMIC workstation cluster. For the point-to-point communication problem addressed in this paper, the performance improvement of implementing pvmd-to-pvmd communication over the Myrinet-API may not be signicant, since pvmds are not in the critical path of point-to-point communication. However, pvmd is responsible for collective communication that involves three or more PVM tasks. For example, the current PVM multicast implementation only routes multicast messages through pvmds. A signicant performance improvement is expected by decomposing pvmd service and implementing PVM collective communication at the Myrinet-API level. The benet of the approach presented in this paper takes place only when hosts in the PVM conguration are directly connected to the same ATOMIC LAN. In a heterogeneous network environment with dierent kinds of high-speed LAN segments, two hosts may not talk to one another without going through TCP/IP. Our current research addresses how to implement user-level TCP/UDP network protocols over the ATOMIC LAN. Similar to the performance of Myrinet-API communication, we expect the performance of user-level TCP/UDP protocols to be much higher than that of their associated kernel network protocol stacks and so yield greater benet to userlevel applications including PVM. 8 Conclusion In this paper, we have decomposed the PVM service over the high-speed ATOMIC LAN, by separating the data transfer path from the control message path. The critical path of PVM point-to-point data communication has been replaced by a fast and reliable userlevel protocol implemented over the Myrinet-API. By interleaving memory copies and DMA transfers, the memory copy overhead has been greatly reduced, resulting in an enhanced PVM implementation that can provide up to 140 Mbps PVM bandwidth over the ATOMIC LAN. The performance results have shown that our user-level reliable protocol has achieved 94% of the bandwidth upper bound established by communication over the unreliable, zero-memory-copy \raw" Myrinet-API. A signicant increase in PVM application performance is expected to be gained by using our version of PVM over an ATOMIC LAN. Acknowledgements The authors would like to thank Jon Postel, Greg Finn, Joe Touch, Celeste Anderson, Ted Faber, Annette DeSchon, and Steve Hotz at USC/ISI for their suggestions regarding this work and Myricom for the useful dialogue regarding their dual-interface LANai control program. References [1] A. Beguelin, J. Dongarra, A. Geist, R. Manchek, S. Otto, and J. Walpole, \PVM: Experiences, current status and future direction," in Supercomputing'93 Proceedings, pp. 765{766, Nov [2] A. Geist, A. Beguelin, J. Dongarra, W. Jiang, R. Manchek, and V. Sunderam, PVM 3 User's Guide and Reference manual. Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831, Sept [3] J. B. Postel, Transmission Control Protocol. RFC 792, Sept [4] R. Felderman, A. DeSchon, D. Cohen, and G. Finn, \ATOMIC: A high-speed local communication architecture," Journal of High Speed Networks, vol. 3, no. 1, pp. 1{30, [5] P. Druschel, L. L. Peterson, and B. S. Dave, \Experiences with a high-speed network adaptor:

10 A software perspective," in Proceedings of SIG- COMM'94, pp. 2{13, Sept [6] C. L. Seitz, N. Boden, J. Seizovic, and W. Su, \The design of the caltech mosaic c. multiprocessor," in Proceedings of the Washington Symposium on Integrated Systems, (Seattle), [7] N. Boden, D. Cohen, R. Felderman, A. Kulawik, C. Seitz, J. Seizovic, and W.-K. Su, \Myrinet: A gigabit-per-second local area network," IEEE Micro, vol. 15, pp. 29{35, Feb [8] J. Touch, A. DeSchon, H. Xu, T. Faber, T. Fisher, and A. Sachdev, \ATOMIC-2: Production use of a gigabit lan (abstract)," in Proceedings of Gigabit Networking Workshop'95 at INFOCOM'95, Apr [9] V. Jacobson and R. Braden, TCP Extensions for Long-Delay Paths. RFC 1072, Oct [10] Myricom, Myrinet Link and Routing Specication, Jan [11] C. A. Thekkath, T. D. Nguyen, E. Moy, and E. D. Lazowska, \Implementing network protocols at user level," in Proceedings of SIG- COMM'93, pp. 64{73, Sept [12] A. Edwards, G. Watson, J. Lumley, D. Banks, C. Calamvokis, and C. Dalton, \User-space protocols deliver high performance to applications on a low-cost gb/s LAN," in Proceedings of SIG- COMM'94, pp. 14{23, Sept [13] C. Maeda and B. Bershad, \Protocol service decomposition for high-performance networking," in Proceedings of SIGOPS'93, pp. 244{255, Dec [14] G. G. Finn, \Device control via the network: Interface limitations and issues," in Proceedings of 4th Annual Principal Investigators Meeting (Networking'94), p. 22, ARPA, Sept [15] G. G. Finn, \An integration of network communication with workstation architecture," in omputer Communication Review, Oct