RAMP: A Reliable Adaptive Multicast Protocol

Transcription

1 RAMP: A Reliable Adaptive Multicast Protocol Alex Koifman akoifman@tasc.com Stephen Zabele gszabele@tasc.com TASC 55 Walkers Brook Drive Reading, MA 01867, USA Tel: ABSTRACT The specifications and performance of RAMP, a Reliable Adaptive Multicast Protocol, are presented. Initially described in IETF RFC 1458 [1], RAMP has been enhanced for use over an all-optical, circuit-switched, gigabit network under our ARPA-sponsored Testbed for Optical NEtworking (TBONE) project. RAMP uses immediate, receiver-initiated, NAK-based error notification combined with originator-based unicast retransmission. The approach is motivated by the loss characteristics of the TBONE network, where extremely low bit-error rates (10e-12 or better) and circuit switching make packet losses almost entirely a result of receiver buffer overflows. Hence, multicast retransmission would hinder performance through unnecessary receiver processing of redundant packets, and delayed NAKs would increase both latency and the likelihood of buffer overflow. Interestingly, TBONE loss characteristics resemble those of switched virtual circuit ATM networks and packet-switched networks employing reservation services. Hence, RAMP s design is also relevant for conventional networks. KEYWORDS NAK-based Control Reliable Multicast Transport Scalable Architecture INTRODUCTION Very high-speed communication networks and largeðscale image management systems are emerging as critical elements of global information systems for such diverse applications as defense and intelligence, supercomputing and education, environmental remote sensing, and medical imaging. Some of these applications are experiencing increasing requirements to store and manage thousands of terabytes of digital image data, to capture and disseminate images in realðtime, and to provide access to hundreds of users who will be geographically distributed. In the near future, users will browse through vast libraries of images and retrieve full sets of high resolution images for exploitation, receive images related to crisis tasks in realðtime, and "tune in" to broadcasts of information related to topics, geographic areas, or activities of interest. The high-speed networks needed to support these and related capabilities must scale effectively as the number of users, images, and image sources grow, as technology for higher data rates becomes available, and as the demand for network services supporting image access and

2 communication services becomes more sophisticated. When the same data must be delivered to two or more sites, repeatedly transmitting the data using unicast protocols unavoidably cuts throughput by the reciprocal of the number of sites. As an alternative, network efficiency can be maintained through the use of multicast protocols where data is transmitted once yet can be received by multiple clients. Multicast-based services using Deering s multicast extensions to IP [2] are already in use over the Multicast BackbONE (MBONE) overlay on the Internet, where applications such as Jacobson s vat audio tool, and nv video tool have met with considerable success. As end-to-end reliability is not needed for these applications (audio and video data is transient, and recovery is best effected by simply skipping over missing segments) the implementations are based on UDP. For other collaborative applications such as Jacobson s wb shared whiteboard tool, shared text editors, and other desktop conferencing components, the data sets are more static and some form of reliable delivery is needed to ensure consistent presentations across multiple views. As there is currently no standard for reliable multicast transport, implementations of several reliable and semi-reliable (best-effort) multicast transport protocols have recently arisen [3, 4, 5, 6, 7]. The approach taken by these protocols has generally focused on NAK-based, receiver initiated, retransmission approaches where dropped packets are both requested and retransmitted using multicast. This approach is largely motivated by the characteristics of the MBONE and MBONE-like networks having relatively low (1.544 megabits per second) data rates and packet-switched routers that provide no quality-of-service (QoS) guarantees. For these networks, end nodes are in general considerably faster than the network and the most likely sources of lost packets are the switches (routers). When packet loss occurs in a switch, the losses observed by the downstream receivers are necessarily correlated, and retransmission using multicast is entirely appropriate. As part of the ARPA-sponsored Testbed for Optical NEtworking (TBONE) project, TASC, MITRE, and Optivision are developing an all-optical, circuit-switched, gigabit network. The significantly higher data rates (initially 800 megabits per second), the extremely low bit-error rates (10e-12 or better), and the circuit-switched nature of the optical crossbar switches within the TBONE network have shifted the most likely source of packet loss from congestion in the switches to overflows in the receiver buffers. Coupled with the desire to minimize latency and eliminate unnecessary processing both at the sender and the receivers in order to maximize performance, this has led to a unicast-based rather than a multicast-based approach for error control. Interestingly, the characteristics of the TBONE network are quite similar to those of ATM networks providing switched virtual circuits as well as those of more conventional packet-switched networks providing resource reservation services. As such, this design is also relevant for use in more conventional networks. To provide high-performance transport services for TBONE, we have been incorporating the error control approach into our implementation of RAMP, a Reliable Adaptive Multicast Protocol. First described in our IETF RFC 1458, RAMP has evolved from an experimental prototype into a robust implementation that is in daily use in collaborative desktop conferencing applications. In the remainder of this paper, RAMP s design and characteristics are presented, followed by an analysis of measurements made of RAMP s performance as a multicast, a unicast, and a concast protocol over Ethernet. RAMP OVERVIEW

3 RAMP is a transport layer multicast protocol that operates over network layer multicast protocols such as IP/multicast, providing a standard method for reliable pointðtoðmultipoint transmission. RAMP guarantees reliable and orderly delivery to all multicast recipients using a negative acknowledgment approach to minimize return (control) traffic. RAMP can be described as a connectionðoriented, reliable stream service as message boundaries are preserved; however, both datagram and streamðstyle interfaces are supported. RAMP features include timely (at most a few seconds) notification of receiver failure to the sender, as well as timely notification of sender failure to all receivers. RAMP provides fast joining and leaving of groups, where participating receivers can leave a group or new receivers can join a group at any point in the session. MidÐsession dynamic modification of group membership allows individual instances of distributed applications, such as shared whiteboards, to be initiated and terminated at different times without requiring lengthy resynchronization. As RAMP maintains explicit group membership at the sender, transmission can be terminated immediately when the last receiver leaves the group rather than continuing to consume valuable network resources. Explicit maintenance of group membership within RAMP also provides TBONE with the information needed for circuit setup. Source knowledge of the destination set within TBONE is consistent with the ATM requirements for setting up multicast groups, and eliminates the need for an Internet Group Management Protocol (IGMP)Ðlike service to supply routing information. RAMP also supports mixed reliable and unreliable transport in that some of the receivers can elect not to set up control channels or send retransmission messages. This feature is useful for multicast transmission of hierarchically encoded data sets, where partial loss by some of the receivers still yields useful information to those receivers. For example, an image sent from a ground station to an archive must be performed reliably; however, other listeners can elect to receive reliably only the lower resolution levels. If partial information from the other levels becomes available to those receivers, it can be used to increase the quality of the imagery at the corresponding positions in the image. The protocol also provides senderðbased reliability, where the sender can elect to send data either reliably or unreliably. Sender-based unreliable transport under RAMP is similar to UDP transport, with RAMP providing the added functions of segmentation and reassembly of large messages. As shown in Figure 1, RAMP is a transport protocol layered on IP/multicast. Although the functional model for RAMP is somewhat similar to TCP, TCP establishes a fullðduplex reliable connection between two endpoints as shown in Figure 2a, whereas RAMP establishes a simplex (one way) reliable flow between the sender and the receiver group as shown in Figure 2b. Although RAMP does provide limited (single segment) data delivery from a receiver back to a sender in the form of piggyback messages, the reverse path is primarily intended for control information. Figure 1 Network layers A simplex model for multicast communications is well suited to the network topologies that can be

4 established with the TBONE network, as the optical cross bar impedes set up of fully multiplexed pointðtoðmultipoint connections where each group member can act both as a sender and a receiver simultaneously. Although busðbased networks such as Ethernet do in fact support full-duplex interconnectivity, the RAMP architecture in no way precludes an application from establishing a full-duplex flow. An application that requires a full-duplex data flow need only create two RAMP flows: a forward flow and a reverse flow. Each flow functions independently and can provide a different QoS. Achieving a full-duplex data flows on the TBONE network requires high-speed, on demand switching between independent simplex flows. RAMP flow consists of segments, where each consecutive segment has an increasing sequence number. Reliability is achieved using a negative acknowledgment scheme where a receiver notifies a sender immediately upon detecting a gap within the received sequence. Explicit acknowledgments are only required during the initiation and termination of connections. Unreliable Delivery Under RAMP In addition to reliable delivery, RAMP also provides two types of unreliable data delivery to support applications where unreliable delivery is acceptable and appropriate. The first type is the more familiar unreliable connectionless delivery model where the sender and all receivers operate unreliably, and is appropriate for voice and video applications. Here unreliable data delivery is similar to UDP, with the addition that RAMP supports larger messages through segmentation and reassembly. The receivers do not send acknowledgments or retransmission requests to the sender, and the sender does not accept or process acknowledgments or retransmission requests from the receivers. The second type of unreliable delivery involves a somewhat unique model where the sender supports reliable delivery, yet some (or all) of the receivers operate in an unreliable mode. This is appropriate, for example, for image delivery services where certain receivers (such as image archive servers) require reliable delivery of all image data, yet other receivers can accept some data loss, such as the loss of higher resolution data in hierarchically encoded images. The second type of reliable delivery allows a single multicast server to simultaneously support both reliable and unreliable receivers with a single data feed. In RAMP, both the sender and receivers can freely switch between reliability modes. The sender moves between reliability modes by toggling the reliability flag (bit) in its messages, indicating whether or not receivers are allowed to issue control messages to the sender. Receivers switch between reliable and unreliable modes by simply processing or ignoring lost messages. Passive And Active Opens Figure 2 (a) TCP model and (b) RAMP model RAMP allows connection origination either by a sender or a receiver. A connection originated by

5 issuing a Connect request is referred to as an active open, whereas a connection originated by issuing an Accept response is referred to as a passive open. To understand the need for the two types of opens, consider the case when a receiver group consists of a number of receivers, where one of the receivers is the connection originator, while the others wait in passive mode. In the purely passive mode all receivers wait for a Connect and upon the receipt, respond with Accept thereby joining the group. Once the connection with at least one receiver is established data can be passed from a sender to a receiver(s) and from a receiver to a sender. For collaborative applications involving interactive image retrieval, this connection model has proved quite useful. RAMP permits a receiver to have a connection with multiple senders. In this case the connection with each sender is maintained independently. This model is well suited for a multipoint-to-multipoint communications typically required for collaborative whiteboards and similar types of applications. Data Flow An example data flow from sender to receiver(s) for IP/multicast is shown (Figure 3). Each block indicates transmission of datagrams, with the size of the block proportional to the length of the datagram. Spaces between the blocks indicate intervals where no datagrams are being transmitted; spacing between blocks is not to scale. Figure 3 IP/multicast data flow In comparison, a RAMP data flow from sender to receiver(s) contains both the above packet groups and additional messages providing state information. For example, the state information allows a RAMP sender to inform receivers when there is currently no data to send, so that the receivers do not perceive data loss. The sender can elect to operate either in Burst or Idle mode, and is allowed to transition between the two modes as the number of receivers varies. Burst mode minimizes the network traffic at the expense of the control traffic, and better for smaller receiver groups (nominally, less than one hundred receivers). Idle mode introduces extra messages in the data flow, but minimizes the control flow transfers and is therefore better for larger receiver groups. RAMP logically breaks the data flow into bursts, with a burst defined as a series of messages where the interval between any two consecutive messages is less than some time T (e.g., 0.5 seconds). When the time between two consecutive messages is greater than T, the first of the two messages defines the last message in the current burst, and the second of the two messages defines the first message in the subsequent burst. The duration of a burst can be arbitrarily long: for example, if the interval between any two successive message is always less than T, then the entire transmission is considered one burst. When the sender has not transmitted data for a time interval T, the end of a burst is declared and the sender is required to send one or more Idle messages. The sender selects the mode in which the connection operates. RAMP messages use both a mode flag

6 (Burst or Idle) and an ACK flag to synchronize mode selection between the sender and receivers. When the sender sets the ACK flag, it expects all reliable receivers to acknowledge the message. In Burst mode (Figure 4), the sender marks the start of the burst by setting the ACK flag. When the sender determines the end of a burst, it sends a single Idle message indicating that the sender will remain silent until the beginning of the next burst. Whenever a reliable receiver observes an ACK flag, the receiver is required to acknowledge receipt of the message by sending an ACK message containing the segment number back to the sender. Receipt of a Data message containing an ACK flag confirms that the forward (data) connection to the receiver has not deteriorated during the sender s quiet interval, and receipt of the ACK response by the sender confirms that the return (control) connection to the sender has not deteriorated. When the sender does not receive an ACK from one of the known reliable receivers, the sender knows that data (and possibly a connection to the receiver) has been lost. The sender can then begin preemptive retransmission leading to a possible closing of the connection for that receiver. Figure 4 RAMP data flow in Burst mode In Idle mode (Figure 5), a series of Idle messages is sent rather than a single Idle message. The Idle messages inform the receiver(s) that no additional data is available, yet the sender will not become silent for a period longer than T. The Idle scheme essentially keeps resetting the timeout(s) at the receiver(s) until there is more data to transmit, so that explicit acknowledgments are not required at the beginning of the next burst. By avoiding ACKs, the number of messages that the sender must process is reduced. Figure 5 illustrates the changes made to the IP multicast data flow by RAMP operating in Idle mode. Figure 5 RAMP data flow in Idle mode In Idle mode, when a receiver does not receive either an Idle message or a Data message within a period of time T, the receiver assumes that data is lost and begins the usual retransmission processing. Whereas the sender is responsible for identifying and acting on a failed connection in Burst mode, the receivers are responsible for identifying and acting on failed connections in Idle mode.

7 Protocol Transfers All data flows from a sender to the receivers over the data channel using a combination of multicast and unicast, and all control traffic flows from the receivers to the sender over a unicast control channel. All initial data transmissions are multicast by the sender to the receiver group, and (currently) all retransmissions resulting from lost data are unicast by the sender to the individual receiver. Receivers receive both the initially transmitted data and any retransmitted data over the same socket due to the intrinsic capabilities of IP/multicast. When the sender issues a Connect message to an IP multicast address, each receiver listening on this address responds with an Accept message. Once at least one Accept has been received, the sender is allowed to start sending Data messages. Each Data message includes a sequence number which monotonically increases with each new message. When a receiver detects a gap in the message sequence, the receiver sends a Resend message over the control channel to the sender. The Resend message contains the sequence number of the undelivered message(s). In Burst mode (Figure 6), Data messages are (logically) grouped into bursts. Each burst followed by the Idle message. The Idle message(s) has the same sequence number as the preceding Data message. The first Data message in each burst has the ACK flag set. When the ACK flag is set, each receiver must respond with the ACK message. If the ACK message from a receiver times out (the sender resends the Data message a few times, e.g., 3 times), the sender considers the receiver to be unreachable and closes the control channel to it. This in turn terminates the flow, allowing the sender and the receiver to learn that the connection is broken. Broken connections can be reestablished at a later time.

8 Figure 6 RAMP transfers in Burst mode In Idle mode (Figure 7), the sender multicasts Idle messages immediately after it receives the first Accept message. When Data messages are being sent, no Idle messages are issued. The sender resumes sending Idle messages after a period time T from the last multicast Data message. Resend messages are unicast to the individual receivers and do not reset the timeout period at the receivers. If a receiver does not receive an Idle message or a Data message within the timeðout interval (e.g., 0.5 sec) from the last Data message, the receiver issues the Resend message with the next sequence number (one greater than the one last received). If after a timeðout, the receiver does not receive the missing message, the receiver closes the control channel with the sender and this closes the RAMP flow to the sender. Receivers periodically (e.g., 2 sec) unicast Idle messages to the sender. Any receiver s message to the sender resets the receiver Idle time-out period. If the sender does not receive an Idle message from a receiver, the sender closes the connection to the receiver. Figure 7 RAMP transfers in Idle mode Receivers that begin listening on the multicast address after the sender is started (late joiners) will not receive the Connect message, however the receivers can still join the reliable multicast session through passive accept. If the first message that a late joining receiver receives from the sender is a Data message, then the receiver considers itself a late join and sends the Accept message. Upon receipt of the Accept message, the sender simply adds the late joining receiver to its receiver list. The late receiver uses the first received Data message s sequence number as the initial sequence number, and is prohibited

9 from requesting retransmission of messages with earlier sequence numbers. If a receiver receives a Connect message or a Data message from a sender for which it has no control channel, the receiver assumes that this is a new sender and responds with an Accept message. This allows a receiver to listen to multiple senders transmitting to the same multicast group on the same socket. RAMP allows a receiver to unicast data reliably to the sender(s). This piggyback mechanism allows to send up to a segment of data at a time. The sender acknowledges each segment. A pair of Pdata/ACK messages is used for piggyback delivery. Message Format There are two types of simplex flows for each RAMP connection (Figure 8). The first type is the data flow from the sender to the receiver group using IP/multicast (except for unicast retransmission of lost Data segments and explicit ACK messages). The second type is the control flow from a receiver to the sender using unicast. As control flows use unicast, there must be one control flow for each receiver in the receiver group. In contrast, as data flows use multicast, only one is needed for each RAMP connection. Figure 8 RAMP data and control flows Data and control message headers and message types are nearly identical, although the asymmetric nature of RAMP induces minor differences. The first 8 bytes of the RAMP message headers are intentionally identical to the UDP message header so that completely RAMP-compatible messages can be constructed using UDP yet avoid kernel programming during development. Data flow message types include Connect, Accept, ACK, Idle, Close and Data. Control flow message types are Connect, Accept, ACK, Idle, Close, Pdata and Resend. RAMP allows applications to send and receive large messages by breaking the messages into basic transfers units, or segments. Messages that do not fit within a single segment are fragmented by the sender and reassembled at the receiver s end. Every data flow segment has the fixed 16 byte header, followed by options (if any), followed by data. As it is difficult to negotiate the segment size when dealing with multiple receivers, RAMP s segment length is set to 8000 octets. Multiple segments may comprise one message, up to the maximum of 216-1, allowing maximum message size of 8*64 Kbytes (actual message size is slightly less as the header takes up to 60 bytes in each segment, reducing the total data portion). Control messages are sent from an individual receiver to the sender using unicast. Every control message has a fixed header portion of 16 octets, the rest may be a list of pairs of sequence numbers (sequence number, length of sequence) or a return multicast address. Control flow is used only by receivers subscribing to reliable delivery.

10 Resend messages can combine multiple sequence numbers into one message, such that when a receiver detects a gap in the received sequence numbers, the receiver can request multiple messages in a single Resend request. For other control messages, e.g., ACK, waiting until multiple segments are acknowledged is discouraged as it may cause a timeðout at the sender side. When combining multiple sequence numbers in one message, messages are listed in increasing order -the lower sequence number first and the higher one last. Interim Flow Control Approach Flow control for the TBONE allðoptical network currently relies on HiPPI hardware flow control, however we have implemented a simple, interim flow control mechanism to allow use of RAMP over packetðswitched networks. The interim flow control approach calculates a transmission delay factor for each receiver based on the number of negative acknowledgments received from that receiver during normal retransmission processing. The delay factor is given by the formula: Delay = LostSeg * Weight / SentSeg where: Delay represents the amount of delay between the transmission of individual segments, and is measured in 10 millisecond units. A delay resolution of 10 milliseconds is used as it represents the minimal delay resolution for most UNIX interval timers. LostSeg is equal to the number of segments lost during a sample interval (we use a sliding window sample interval length of 5 seconds). SentSeg is a count of segments sent during the sample interval. Weight is a scaling constant that provides finer grained control over the delay value, with 0 <= Weight <= 1. A weight value of 0 provides no flow control, whereas a weight value of 1 provides maximum delay. Currently, we use a weight value of 1. As we are maintaining full reliability, the delay value used by the sender is the greatest of all delay values across the set of receivers, such that rate at which a sender sends packets is determined by the slowest receiver in the group. Even with the above flow control, it is still possible that a sender can still overflow a receive buffer. This occurs when a sender sends faster than the receiving application retrieves data from the receive buffer, such that the receiver s queue becomes full: as no segments are lost, the sender does not get an

11 indication to slow down. To avoid overflowing the receive buffer, we employ a mechanism whereby the receiver intentionally drops segments when the queue becomes greater than 80% full. This triggers a Resend request by the receiver, providing the sender with the needed throttling information. The simplicity of RAMP flow control is necessitated by the minimal amount of traffic that is sent from receivers to a sender. Unlike TCP where each packet (or a group of packets) is acknowledged, RAMP tries to minimize the reverse channel traffic. The lack of data makes rate estimation considerably less reliable, precluding more sophisticated approaches. Nonetheless, RAMP s flow control mechanism works reasonably well, and while simple, alleviates the overflow problem at no additional cost. PERFORMANCE OVER ETHERNET In order to evaluate the performance of our RAMP implementation while the TBONE HiPPIÐbased infrastructure is being completed, RAMP performance measurements have been made over TASC s internal twisted-pair Ethernet LAN. The results are presented and analyzed in this section. Where appropriate, RAMP performance is compared with the measured performance of TCP over the same test network. All tests were performed using identically configured Silicon Graphics Indigo (R3000) UNIX workstations. The extent of testing was limited by the physical number of machines (6) available. Reliable Multicast -For reliable multicast transport, RAMP s effective throughput (sum of the throughputs to each receiver) increases nearly linearly with the number of receivers, as shown in Figure 9. RAMP s base unicast performance (one sender and one receiver) was measured to be approximately 6.5 mbps. In the ideal case, the effective throughput from one sender to n receivers should be n times the unicast rate of 6.5 mbps. The measured effective throughput of 32 mbps for 5 receivers corresponds exceptionally well to the theoretically expected value of 32.5 mbps. By intentionally introducing random packet losses at the receivers, we were able to gauge RAMP s performance over congested multihop networks. For a drop rate of 5% where each receiver intentionally drops 5% of the received packets at random, shows that RAMP s performance continues to increase nearly linearly with the number of receivers. The change in slope between the previous case where there is no intentional loss and the case where there is an intentional 5% loss is primarily caused by each receiver s independent requests for retransmission. With the sender providing 5% more packets to each of n receivers, the sender output traffic is increased by n times 5%. For example, 5 receivers require that the sender transmit 25% more packets. For a given sender rate, the added overhead due to retransmission reduces the throughput by a factor equal to the reciprocal of one plus the overhead; consequently the effective throughput for the 5 receiver, 5% loss case can be at most 80% of that for the 5 receiver, no loss case. As shown in Figure 9, the measured performance of 24 mbps for the 5 receiver, 5% loss case is very close to the expected maximum performance of 25.6 mbps. The small difference can be accounted for by the additional traffic generated by the retransmission requests themselves, and by experimental measurement error. A similar set of measurements made using a drop rate of 10% reveals that RAMP s performance continues to increase nearly linearly with the number of receivers. Similarly, with the sender providing an additional 10% more packets to each of n receiver, the total traffic is increased by n times 10%. For the 5 receiver, 10% loss case, 50% more packets are sent, which effectively reduces the throughput to 66% of that for the no loss case. As shown in Figure 9, the measured performance of 17 mbps compares reasonably with the expected maximum performance of 21 mbps.

12 Even when loss rates are quite low, the probability that a retransmission request for any given packet will be made by more than one receiver approaches unity as the number of receivers becomes large. For example, if each of 21 receivers drops 5%, then at least 5% of the requests are guaranteed to be duplicates, with the expected number of duplicates at a much higher level (~30%). Clearly a more optima method for servicing duplicate requests is to retransmit using multicast, rather than unicast as is currently done within RAMP. The disadvantage of using multicast for retransmission is that for extremely low loss networks with moderate number of receivers, (such as our dark fiber, circuit switched, TBONE network), use of multicast needlessly consumes receiver resources to receive, process, and discard redundant packets. For example, with a bit error rate of 10e-12, the expected packet error rate using a 1 MByte packet size is approximately 10e-5. As the bit error rate is dominated by receiver noise rather than network noise in an optical network, bit error events at individual receivers will tend to be independent. Then, even for a multicast group with 10,000 receivers, the probability that the sender will receive a duplicate retransmission request for any given packet is approximately 0.5%. Reliable Unicast -RAMP performance as a reliable unicast protocol was measured as a function of the number of independent senderðreceiver pairs on the same network. As shown in Figure 10, the aggregate throughput to receivers remains nearly constant as the number of receivers is increased. Although the number of senderðreceiver pairs used in this experiment is quite limited, the results indicate that multiple independent RAMP sessions can effectively divide the available network bandwidth equitably when competition for resources occurs. Equitable sharing of network resources is directly determined by the ability of flow control to throttle each sender to an appropriate rate. The interpretation is that although the prescribed flow control algorithm is relatively simple, it is reasonably effective at least for a limited number of sessions in a LAN environment. Figure 9 Multicast over Ethernet

13 Figure 10 Unicast over Ethernet Experiments with intentional packet losses at the receivers were repeated for the multiple competing sessions experiment. For an intentional 5% loss rate at each receiver, the aggregate throughput was reduced from the nominal 6.5 mbps level to a level of about 5.8 mbps; an effective reduction to 90% of the noðloss level. For an intentional 10% loss rate at each receiver, the aggregate throughput was reduced from the nominal 6.5 mbps level to a level of 4.8 mbps; and effective reduction to 74% of the no loss level. Clearly the performance of the flow control algorithm degrades faster than can be justified by the loss of data alone, and investigations into a more enhanced flow control algorithm are warranted for IP networks. As the TBONE network will initially use hardware flow control and flow control aggregation, however, an enhanced flow control algorithm is not needed at this time. Reliable Concast -In contrast to distribution types of multicast services where there is one sender and multiple receivers (such as the transmission of audio or video over the MBONE), collaborative applications require the capability for each instance of a collaborative application to act both as a sender and as a receiver on the same multicast channel. As this data generally contains state information, it must be exchanged using reliable delivery. Consequently RAMP s performance in a manyðtoðone, or concast, environment is an important measure of RAMP s suitability for a variety of collaborative applications. RAMP performance as a reliable concast protocol was measured as a function of the number of senders. As shown in Figure 11, the aggregate throughput to the receiver was observed to degrade minimally as the number of senders increased. From RAMP s base throughput of 6.5 mbps for 1 sender, throughput degraded nearly linearly to 6.1 mbps for 5 senders. Again, equitable sharing of network resources is directly determined by the ability of flow control. Since RAMP uses NAKs as the sole basis for estimating sender rates, the receiver in this experiment is required to send an increasing number of NAKs to each sender as the number of senders increases. This implies a total NAK requirement proportional to N2 for N senders. Although RAMP s performance as the number of senders increases is excellent (only a 6% reduction for 5 senders over the 1 sender case), clearly a more scalable flow control approach will be necessary for high volume concast communications with larger numbers of senders. A simple approach is for the receiver to provide a weight in each NAK that acts as a NAK multiplier for the senders rate calculation. This will reduce the total NAK requirement to a value proportional to N rather than N2. We are currently evaluating this approach. Experiments with intentional packet loss at the receiver were repeated for the concast experiment. For an intentional 5% loss rate at the receiver, the aggregate throughput was reduced nearly linearly from the nominal 5.4 mbps level with one sender to a level of about 5.1 mbps with 5 senders; a effective

14 reduction to 94% of the one sender level. For an intentional 10% loss rate at each receiver, the aggregate throughput was reduced nearly linearly from the nominal 4.9 mbps level with one sender to a level of 4.5 mbps for 5 senders; and effective reduction to 92% of the one sender level. The reduction in throughput from the one sender case to the 5 senders case appears relatively consistent (~94%) across all loss levels. Figure 11 Concast over Ethernet Comparison with TCP -Measurements were made of TCP performance over the same testbed to provide an absolute evaluation of RAMP s performance. As shown in Figure 12, when TCP is used to provide a reliable multicastðlike service using repeated transmissions, the throughput to each receiver is reduced by 1 over n, where n is the number of receivers. This expected result occurs as the total bandwidth for the network is fixed and must be shared by each of the n receivers. In contrast, the throughput to each receiver remains relatively constant when reliable multicast is used, also as expected. Figure 12 primarily illustrates that RAMP is at least as good as TCP as a unicast protocol, and that RAMP behaves properly as a reliable multicast protocol. Figure 12 RAMP vs. TCP throughput over Ethernet An alternate presentation of the results is given in Figure 13, where the effective throughput (sum of throughputs to each receiver) is given as a function of the number of receivers for both TCP and RAMP. Clearly there is a substantial advantage to using RAMP over TCP even for a receiver group as small as two, with the advantages increasing quickly as the number of receivers grows. For example, to achieve the same effective throughput as RAMP with 5 receivers (32 mbps), TCP would need to be running over a 50 mbps LAN rather than the 10 mbps Ethernet used here.

15 Figure 13 RAMP vs. TCP effective throughput over Ethernet A surprising result illustrated by this graph is that the effective throughput for repeated unicast using TCP actually rises (slightly) with the number of receivers. This indicates that the communication process is actually receiver bound, such that the sender can actually provide data faster than an individual receiver can accept it. With multiple independent receivers, the rate to each receiver is reduced from the single receiver case such that each receiver can more easily keep up with the sender. This allows the sender to transmit at a higher aggregate rate yet not overrun any given receiver. ENHANCED FLOW CONTROL Although unnecessary for TBONE s current network configuration, additional work is warranted on RAMP s flow control approach for use with packet-switched networks that provide no reservation capabilities. To this end, we have begun investigating the use of what can be described as proactive flow control. With this approach, as a receiver s buffer approaches critical occupancy, the receiver estimates a reduced transmission rate and rate duration for which the receiver can reasonably be expected to recover. The receiver transmits the information to the sender in the form of a "throttle" message, which the sender then uses to reduce its transmission rate below the flow specification for the specified duration. Upon expiration of the time period, the sender returns to the full rate allowed by the flow. If the sender receives a second throttling message during a reduced rate period, processing is straight forward: if the second rate is larger and the time period is shorter, simply ignore the message. Otherwise reduce to the smaller new rate for the new period, then return either to the previously reduced rate or the original flow rate, as appropriate. Interestingly, an extreme end of this protocol is a type of stop-and-wait approach (rate of zero for some time). An advantage of the approach is that the sender treats throttle messages "anonymously", such that the computational burden on the sender is low. For illustration, consider RAMP s current approach whereby the sender derives rate information by counting received NAKs. NAK counting places a considerable computational burden on the sender, as the NAK accounting procedure must keep a separate count for each receiver in order to produce a meaningful interpretation. For example, receipt of 100 NAKs, 50 from each of two receivers may indicate that the sender must slow considerably, whereas receipt of 1000 NAKs, one from each of 1000 receivers may indicate that the sender need only slow minimally. For large receiver groups, this is inordinately expensive. As the proposed proactive flow control approach uses a simple aggregation scheme that requires no explicit receiver accounting, it is intrinsically much more lightweight and scalable.

16 SUMMARY We have presented the design and the performance of RAMP, a reliable adaptive multicast protocol being developed under the ARPA-sponsored TBONE project. The performance assessment reveals that RAMP is an effective protocol for reliable transmission even when used as a unicast protocol, that RAMP scales linearly for reliable multicast operations, and that RAMP provides quite reasonable performance as a concast protocol making it suitable for collaborative applications. Although designed for an all-optical, circuit-switched, gigabit networks, RAMP performs well over packet-switched networks. RAMP has been implemented for IP/Unix workstations and has been tested over both Ethernet and ATM. RAMP is currently being used in various multimedia collaborative applications where it has proven to be fast and reliable. ACKNOWLEDGMENTS This work was sponsored in part by the Advanced Research Projects Agency, under contract number F C-0086 and contract number F C REFERENCES [1] R. Braudes, S. Zabele, "Requirements for Multicast Protocols", RFC 1458, May [2] S. Deering, "Host Extensions for IP Multicasting", RFC 1112, August [3] V. Jacobson, "A new architecture for real-time applications and protocols", Proc ARPA Networking PI Meeting, Sept [4] V. Jacobson, "Multimedia Conferencing on the Internet", Tutorial Notes - ACM Sigcomm94, (London, Sept. 1994). [5] L. Padmanabhan, "Design and Implementation of a Shared White-Board", M.S. Project, Dept. of Computer Science, UMass, Amherst, MA 01003, May [6] S. Paul, K. Sabnani, D. Kristol, "Multicast Transport Protocols for High-Speed Networks", Proc IEEE Int. Conf. Network Protocols, (Boston, Oct. 1994). [7] S. Pingali, D. Towsley, J. Kurose, "A Comparison of Sender-Initiated and Receiver-Initiated Reliable Multicast Protocols", Proc ACM SIGMETRICS Conf.