Impact of Network Dynamics on End-to-End Protocols: Case Studies in TCP and Reliable Multicast 1

Transcription

1 Impact of Network Dynamics on End-to-End Protocols: Case Studies in TCP and Reliable Multicast Kannan Varadhan Deborah Estrin Sally Floyd USC/Information Sciences Institute Marina Del Rey, CA (Fax) {kannan,estrin}@isi.edu M/S B-9B, Lawrence Berkeley Laboratory One Cyclotron Road, Berkeley, CA (Fax) floyd@ee.lbl.gov April, 998. Abstract End-to-end protocols measure network characteristics and react based on their estimates of network performance. Network dynamics can alter the topology significantly, and thereby affect protocol operation. Topology changes may result in routing pathologies (such as route loops, packet interleaving), changes to the end-to-end path characteristics, network partition etc, that then impact the performance of end-to-end protocols. This paper presents methodologies to evaluate an end-to-end protocol in the presence of network dynamics using a simulator. A number of models of network dynamics are introduced. In the paper, we evaluate two end-to-end protocols over dynamic topologies and study the adaptivity of these protocols to topology changes. The first part of the paper illustrates the effect of packet interleaving on the performance of TCP. The second part of the paper is a systematic evaluation of the adaptive r mechanisms in Scalable Reliable Multicast (SRM). The r mechanisms are evaluated under simple topology changes, as well as under network partition conditions. The paper concludes by posing a number of open research questions about the behaviour of different reliable multicast mechanisms when operating over dynamic topologies. Keywords: Protocol Design, Protocol Evaluation, TCP, Reliable Multicast, SRM, Network Dynamics, Topology Changes. Areas of Interest: Protocol Design and Analysis, Internetworking, Multicast/Broadcast Algorithms The work of K. Varadhan and D. Estrin was supported by the Defense Advanced Research Projects Agency (DARPA) under contract number ABT-9-C-. The work of S. Floyd was supported by the Director, Office of Energy Research, Scientific Computing Staff, of the U.S. Department of Energy under Contract No. DE-AC-SF98, and by ARPA grant DABT-9-C-. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the DARPA or the U.S. Department of Energy. This technical report is USC-CS-TR 98-, and is an extended version of [].

2 Introduction and Related Work A number of transport and application level protocols measure and react to network characteristics, such as the end-to-end delay, available bandwidth, rtt, congestion, etc. A TCP [] source adjusts its data rate based on its estimate of the available bandwidth and congestion; Receiver-Driven Layered Multicast (RLM) [] requires a group member to estimate the level of congestion, and decide the amount of data that it can receive at any given ; in Scalable Reliable Multicast (SRM) [], every group member estimates the rtt to all other group members, for use in its error recovery mechanisms. Common to all of these protocols is that changes to the topology can significantly change the actual network characteristics, and therefore impact the performance of the protocol, and possibly adversely affect the network. Paxson [], through end-to-end measurements of routing stability, tells us that in general, the stability of a particular end-to-end path is fairly high (around %); i.e., if, at a given instant in, an end-to-end transport session uses a particular route from end-to-end, then there is a % likelihood that the session will use the same route at a later instant in. However, he notes further that, there is extensive variability in the measure from site to site (variability of % %), and therefore, while one end-to-end path may remain fairly stable, another might be less so. Govindan et al.[] arrive at a similar conclusion, based on their analysis of inter-domain routing protocol traces gathered at various points in the Internet backbone over 8 month period. More recently, Labovitz et al.[9], through measurements of inter-domain route updates at various network access points, suggests that there are a far greater number of route updates than is to be expected in a network, the size of the Internet. Paxson [] also points out a number of routing and network pathologies that adversely affect TCP performance. By design, multi-party protocols and applications support a greater number of participants; they are used to communicate simultaneously with participants connected by diverse network paths. Therefore, it is reasonable to expect that routing and network pathologies can have even greater impact on the performance of multi-party protocols. Transport protocols are designed to be adaptive to varying network conditions, but their evaluation is often conducted after their deployment on an operational network. We need to understand the full range of protocol behaviour before protocol deployment, in order to be sure that it meets the desired design characteristics. In addition, tradeoffs must be evaluated with respect to possibly conflicting goals in the design process. A common example of this is the tension between the goals of robustness and efficiency in adaptive protocols. We propose to enhance the capabilities of the protocol designer to evaluate protocols in the presence of dynamic topologies. This approach permits the designer to more comprehensively understand protocol behaviour, and make the appropriate design choices. Shankar et al.[9] in analyzing different routing protocols, describe the impact of network dynamics on the throughput and delay of a simple window based transport protocol. Individual protocol designers somes suggest how their protocol should perform in the presence of topology changes []; however, we know of no systematic study of the performance and evaluation of end-to-end protocols over dynamic topologies. In this paper, we look at some of the issues involved in studying transport protocols over dynamic topologies. We concentrate on the study of two protocols: TCP and SRM. TCP [, ] is a reliable transport protocol for unicast connections. TCP has been extensively studied, both via simulations and on operational networks. However, its detailed performance analysis in the presence of dynamic topologies has never been reported. We develop our mechanisms using TCP as an example (Section ); this also helped us validate our methodology. A number of reliable multicast protocols have been proposed in the literature recently, such as Lin et al.[], Yavatkar et al.[], Holbrook et al.[8], Floyd et al.[]. In our paper, we analyze the r mechanisms in one

3 reliable multicast protocol: SRM (Section ). We present results of our preliminary work with SRM when it is running over dynamic networks. We close with a discussion of a number of promising research issues related to the impact of network dynamics on reliable multicast mechanisms (Section ). Impact of Route Dynamics on TCP In this section, we will use TCP as an example to demonstrate our methodology of studying a protocol over dynamic topologies. TCP [] is a unicast transport protocol that guarantees ordered reliable delivery of data. There are a number of studies of TCP in the literature. Fall et al.[], in particular, has compared different flavours of TCP over stable topologies. Our work in this section is an extension to this earlier work, studying the behaviour of TCP over dynamic topologies. We separate the notion of a dynamic topology from the routing protocol that is used to repair the topology. In any topology, a network is dynamic if one or more nodes or links in the topology can periodically fail and (possibly) recover. A number of different models of network dynamics can be used to specify the durations when a particular node or link will be down or up. A routing protocol must then be used to compute the end-to-end connectivity after any topology change. As with the models of dynamics, different types of routing protocols can be used. In a typical simulation that has no network dynamics, the user can compute the routes in the topology off-line, and then start their simulation. We call this one- off-line route computation strategy, Static routing. A similar off-line route computation strategy can be used when the network is dynamic. In this strategy, routes are recomputed over the topology after any topology change. We call such an off-line route computation strategy, Session routing. A more realistic option is to simulate Dynamic routing. Figure shows a TCP session running over a dynamic topology, with three different kinds of unicast routing strategies. The experiments in this figure use the topology of Figure. The figures show Link h, i fail at t :8s, and recover at t :s. The plot shows the failure and recovery events as vertical lines at the appropriate s; the arrows at the top of the graph shows the type of event, the label adjacent to the arrows indicate the relevant link. An alternate path is available to route around the failed link. We notice that the throughput and response of TCP is different depending on the type of routing strategy used in the simulation. Each type of routing strategy has specific consequences for the transport protocol that is being simulated. Static routing, if used over a dynamic topology, can lead to temporary partitions while a node or link is down. This explains why there is no packet throughput seen in Figure (a) when Link h, i is down Session routing, on the other hand, will always ensure end-to-end connectivity, as long as the topology is connected. However, results from a simulation with Session routing at the network layer, can be incorrect. For example, throughput measured by a protocol simulation that uses Session routing to repair network dynamics will often be higher than is normally possible. Figure (b) shows that a greater number of packets are successfully transmitted than occurs in the adjacent traces. While Dynamic routing is more realistic, it entails overhead, and can introduce its own artifacts. For example, a protocol designer must take into consideration the fact that, at startup (t = s), no node in the topology has routes to other nodes, except possibly to its directly connected neighbours. Dynamic routing takes some

4 <> packets acks drops link down link up <> <> <> <> <> packets packets packets (a) Static routing (b) Session routing (c) Dynamic routing Figure : Comparison of different routing protocol options in a simulation Brief notes on the topology: Source of interest is at Node Link h, i is dynamic in selected experiments Figure : Topology used for first set of TCP experiments to become quiescent after some number of routing protocol messages are exchanged by all of the nodes in the topology. From Figure (c), we can infer that the TCP session looses all of the initial packets. This is because the session starts at t = s in our simulation, and at this the route computation has not yet become quiescent. Since the retransmit out is very high when a session s data packets are lost very early in the connection history (and this out value is set at s), we see the first successful packet transmission at t = s via the alternate path. For the rest of the experiments in this paper, we use a Dynamic routing protocol. We have implemented a simple Distributed Bellman-Ford routing protocol in ns[]. Our choice of parameters for the routing protocol are motivated by our desire to study the effect of topology changes on end-to-end protocol. One well-understood effect of topology change on these protocols is a transient loss of connectivity. The duration of this connectivity is a function of the routing protocol and the topology. While this area of research is quite interesting and useful when trying to understand protocol behaviour, our focus is on other effects that arise from topology changes, such as intra-network behaviours that affect protocol operation (for example, packet interleaving effects), or variations in the path characteristics (for example, sudden variations in the rtt). Therefore, we choose our parameters to ensure rapid routing protocol convergence without interfering with the simulation of the end-to-end protocols. In our simulations, the route update interval is s. In addition, updates are triggered whenever the incident topology at a node changes state, or when that node receives an update from a neighbour that causes it to recompute new routes.

5 Session Session Figure : TCP Tahoe throughput over stable topologies We use the topology shown in Figure for the first set of our TCP simulations. In this topology, Links h, i and h, i have bandwidth 8M b, and propagation delay ms. All the other links have bandwidth 8Kb, delay ms. Link h, i has a queue limit of 8 packets. This link is also dynamic, i.e., it periodically fails and recovers. The link will be down (and up) for a period of drawn from an exponential distribution with a mean of s (and s). We simulate two FTP sessions, one from Node to Node, starting at t = :s, and the other from Node to Node, starting at t = :s. The window size used by the latter session is packets; Figure presents the vs. sequence number plots of the session from Node to Node, when it uses three different flavours of TCP: Tahoe TCP, Reno TCP, and TCP SACK. The simulation configuration above (topology + unicast routing) differs from [] in two significant ways: () their topology has no alternate paths between the sources and sink though Node, and () they use Static routing to precompute the routes in the topology prior to simulation execution. However, we can verify that neither of these changes affects protocol behaviour when the topology is stable. Figure shows the throughput of the two FTP sessions using Tahoe TCP in stable topologies. In this figure (and all other figures in this section), we plot the sequence numbers modulo 9. We trace the sequence numbers of packets traversing Link h, i. Each packet is plotted as two points: the and sequence number of the packet when it is placed on the queue of the link that is being traced, and the and sequence of the packet when it is removed from the queue. The spacing between the two points is a measure of the queuing delay experienced by a packet on that particular link. The sequence number of the acks for each packet are also shown as tiny dots. These acks are seen approximately one rtt after the packet was first sent by the source. The upper plot in Figure shows the throughput of Session, the FTP session from Node to Node ; the lower plot shows the throughput of Session, the other FTP session from Node to Node. Each session experiences congestion and consequent packet drops at t :s. Following the packet drop, each session adjusts its window, and resumes transmission. This behaviour is identical to that described by Fall et al.[]. Our focus of research is on protocol behaviour in the context of topology changes. We will therefore consider the periodic failure and recovery of Link h, i; specifically, the link will fail at t :8s and recover at t :s This corresponds to the simulation of tahoe, over the net topology, from the test-suite-routed.tcl test suite in the ns- distribution.

6 <> <> Session <> <> Session Figure : TCP Tahoe throughput over dynamic topologies respectively. These events are shown as vertical lines at appropriate points in. The alternate path through Node is available when Link h, i fails. Our plots will show the trace of packets through Link h, i when Link h, i is down. Unlike in the stable case, we see from Figure that each session experiences different types of network behaviours. Session from Node to Node experiences packet loss due to link failure early in its connection phase (Figure ). In fact, Session in each of the variants of TCP experienced this same type of loss, and exhibited the same response to this loss. Hence, we do not plot the throughput of this session in our subseqent figures. Session from Node to Node, on the other hand, experiences the arrival of interleaved acks and significant packet drops following link recovery. We now discuss these two effects exhibited by this session from Node to Node (Session in Figure, and Figure ): the obvious packet drops and the interleaving of acks seen at about the same. We first make a couple of observations about the packet drops occuring at t :s in each of the sessions in Figure. In this session, all of the packet drops occur in a single window of packets. These drops occur because of routing and topology changes. Hence, routing changes can lead to multiple packet drops. Moreover, these packet drops occur just after the recovery of Link h,i. This was contrary to our expectation that packet drops occur due to link failure, but never due to link recovery. Therefore, packet drops may occur due a link recovery. One of the reasons for this packet drop is the arrival of interleaved acks. These acks are in transit across both the original longer path, and the newer shorter path, at the instant of link recovery. In general, there are several possible consequences of such packet or ack interleaving. One consequence from interleaved acks/packets is that the sender can get three dup-acks, and assume that a packet has been lost, even though the packet is not lost. This forces all variants of TCP to do a fast retransmit of the lost packet. TCP Reno, in particular, is susceptible into going into a retransmit out needlessly; TCP Reno is defined by this behaviour in response to multiple packet drops. This corresponds to a simulation of tahoe over the topology net-dv, with link failures at t = :8s and t = :s from test-suite-routed.tcl in the ns- distribution. The results of Figure correspond to the reno and sack simulations in test-suite-routed.tcl and test-suite-sack.tcl, in the ns- distribution.

7 <> <> Session (a) Reno TCP <> <> Session (b) SACK TCP The sessions all create transient congestion after Link h,i recovers; we can see the difference in the recovery mechanisms of the three sessions after the topology change event: - Reno TCP (Figure (a)), after a fast retransmit, waits for its RTO r to fire, and then initiates slow start. - SACK TCP (Figure (b)), appears to behave identically as Reno does, but has a slightly better throughput than Reno TCP. - Tahoe TCP (Session in Figure ) initiates slow start immediately after a fast retransmit, and hence has the highest performance of the three. Figure : Effect of interleaving acks on Reno and SACK TCPs Another possible consequence of interleaved acks is that the sender suddenly sees a big jump in the sequence number acked, and therefore sends a large burst of packets back to back. This can then result in actual packet drops as opposed to simply perceived packet drops. We saw this earlier in Figure, when packets from three types of TCP were dropped. In a different simulation of TCP SACK using the same topology of Figure, we see both of these consequences (. an unnecessary retransmit of out-of-order packets, and. a real packet drop), exhibited by a TCP SACK source (Session in Figure ). In the figure, the receiver, Node, receives a few packets out of order. These are six packets transmitted across Link h, i just before Link h,i recovers at t :s; they are in transit across the alternate longer delay path. Subsequent packets in the same window arrive at the destination earlier using the shorter Link h, i. () The destination continually acks for the interleaved packets still in transit across Links h,,i; after receiving three dup-acks, the sender retransmits the six packets at t :8s even though they are not actually lost. () Subsequently, based on the jump in the sequence number acked by the receiver, the source sends a burst of packets (at t :s) that then results in the actual packet drops. In contrast, Session in The topology differs from the earlier one in that Link h, i is a M b bandwidth, ms delay link, Link h, i is a M b bandwidth, ms delay link; the other links have :M b bandwidth, ms delay. Link h, i has a queue limit of packets. This simulation corresponds to a simulation of sack over the topology net-dv, with link failures at t = :8s and t = :s from test-suite-sack.tcl in the ns- distribution.

8 <> <> Session <> <> Session Figure : Effect of interleaving packets on TCP SACK Figure only experiences drops due to congestion at t :s. This simulation points to a slightly less conservative behaviour in TCP SACK about which packets it should retransmit. The source waits for three dup-acks before retransmitting the first packet, as do TCP Tahoe and TCP Reno. However, based on the selective acks from the receiver, the source infers loss of packets not explicitly acked and retransmits them, leading to possibly unnecessary retransmits of packets that are not actually lost. We saw this earlier in Figure, when it retransmit all six packets that were interleaved at t :s. In contrast, TCP Tahoe, which also retransmits packets that are not likely lost, is controlled by slow start; TCP Reno goes into fast recovery assuming that only one packet has been dropped. In summary, we have shown that studying TCP over dynamic topologies can reveal interesting aspects of TCP. An important consequence of dynamic topologies that we have highlighted is the extensive reordering of packets that can arise due to a small topology change. It is important to recognize that the results in this section are not necessarily a function of the size of the topology. Packet interleaving occurs in the immediate vicinity of a topology change, and this localized re-ordering is maintained all the way to the end-to-end protocol operating at the edges of the network. The ability to study such behaviours is very useful in protocol design. Scalable Reliable Multicast Unicast protocols, such as TCP, are only affected by changes in the topology if the changes occur on the path between the two end-points. In contrast, multicast protocols interact with multiple parties simultaneously and so involve a higher number of links. Therefore, the likelihood is greater that some of the paths in the source s multicast tree are unstable at any given. In addition, the instability in any portion of the multicast tree may affect many members of the group because of the collaborative adaptive algorithms used. Therefore, it is even more critical to study aspects of end-to-end multicast protocols, such as error recovery or congestion control mechanisms. It is generally the case that, if multiple packets in a window have been dropped, the TCP Reno sender ultimately waits for a retransmit out, and the initiates slow start [].

9 In this section, we evaluate a reliable multicast protocol: Scalable Reliable Multicast (SRM). In SRM, active sources in a multicast group periodically send data to the group, as specified by the application. Each unit of data is identified by the tuple hsource id, message sequence numberi; this corresponds to the notion of naming in Application Data Units model used in []. The source tags each unit of data with its unique identifier. In addition, every member of the group periodically sends session messages specifying the amount of data it has received from each source. Loss can be detected either through receipt of data from the source, or through receipt of a session from some member of the group. A group member that detects a loss can send out a request for that lost data. The error recovery mechanism in SRM is receiver-oriented; therefore, any group member that has the requested data can respond to that retransmission request. It is not necessary that the original source of a particular data unit be the one to respond to requests for retransmission of that data. In the rest of this paper, we use packets as a synonym for data units. In order to avoid multiple simultaneous requests by all nodes that detect the loss of a particular packet, each node, n i that detects the loss will set a random r in the interval [C d s ; (C + C )d s ], where C and C are two protocol request parameters, and d s is n i s distance to the source of that packet. n i will send out its request when the r expires. If n i hears a request for that packet before its r expires, it will cancel its r. In either case, it will reschedule its r using exponential backoff. Since duplicate requests from other nodes can mislead a node into backing off an already-backed-off r, each node also delineates an ignore-backoff interval, during which it will ignore requests from other nodes. When it finally receives a repair, the node will set a hold-down r, with a value of d s, so that it does not attempt to aggressively schedule and send a repair in response to a duplicate (or ill-d) request. Likewise, each node n j that can send a repair in response to a request will set a random r in the interval [D d r ; (D + D )d r ]. D and D are two protocol repair parameters, and d r is n j s estimate of its distance to the n i that sent the request. n j sends out a repair message when its r expires. If it receives a repair before its r expires, then its cancels the r. In either case, n j sets a hold-down r, with a value of d r, during which it will ignore all requests for that packet, so that it does not attempt to send out a second repair in response to a duplicate (or ill-d) request. We consider two mechanisms by which values for the r parameters, C, D, D, and D, are chosen. In the fixed r mechanism, all of the nodes use identical preset values. Fixed rs are useful to illustrate the behaviour of the protocol. In practise, we need a mechanism to adapt to the observed delay and loss characteristics exhibited by the network. With adaptive rs, each node, n i, that detects a loss adjusts its request parameters based on the number of duplicate requests that it receives, as well as the average delay between detecting the loss and hearing the request expressed in units of its rtt to the source, d s. Similarly, each node, n j, that can repair a loss adjusts is repair parameters based on the number of duplicate repairs that it receives, as well as the average delay between receiving a request and sending a repair, normalized in units of its rtt to the requestor, d r. The adaptive r mechanism will tolerate a duplicate request (repair), as long as the delay is within one rtt to the source (requester). The terms, C and D, are called the deterministic components of the r mechanisms in SRM. Each node has to set its r value to at least C d s (or D d r ) from the it detects a loss (receives a request). If C and D are set to, then the C and D components ensure that nodes closer to the loss send out requests and repairs. Hence, this component helps distinguish between, and favour those, nodes that are nearer to the loss. By contrast, when there are multiple nodes that are equidistant to the loss, then the nodes should use probabilistic choice in order to decide which of the nodes will send the first request or repair. The terms, C and D, specify the maximum range of the interval in which the rs can be set, and hence are called the probabilistic components of the r mechanisms in SRM. 8

10 Earlier, we had said that each node periodically sends session messages that contain information about the latest message from each source that it has received. In addition, nodes use these session messages to estimate their distances to each other. Each node, m i, -stamps every session message it sends. In addition, for every other group member, m j, that m i is aware of, m i advertises the sequence number of the last session message it received from m j, as well as the that m j sent that session message, and the that m i received it. m j can use this information to estimate its distance to m i. In the remainder of this section, we will present our simulation results as part of our protocol evaluation. We evaluate the protocol in three different ways: on a stable topology, on a dynamic topology that is always connected, or on a dynamic topology that is occasionally partitioned. The first, evaluation using a stable topology, is a base case evaluation (Section.). We use this to illustrate the protocol behaviour, as well as to define our metrics. We then study the same simulation configuration, when the topology is made dynamic (Section.). This evaluation measure helps us understand the characteristic of the deterministic component of the r functions. Finally, we look at the performance of the protocol under partition and partition healing, as well as the impact of the protocol on the network (Section.).. Base Case: Analysis with Stable Topologies In this section, we will evaluate SRM in a stable topology; i.e., the topology is not dynamic. This serves to briefly review the protocol behaviour, as well as to illustrate the different data presentation methods that we will use in the following subsections. We use the ring topology shown in Figure. The bandwidth of each of the links in the topology is :Mbps. The delay for all but one of the links is ms; the delay for Link h, i is 8ms. Link h, i is called the fallback link, and will only be used if one of the other links in the topology has failed. This topology is a very simple model of a network with alternate paths for fallback (or fallback paths). Such fallback paths are used in the event of failure of some of the other paths in the network. When the topology is stable, our topology resembles a string topology []. These topologies stress the deterministic component of the SRM rs. There are other topologies that stress other aspects of the protocol, or model more realistic topologies. We have left the studies on these other topologies for future study. As before, the simulations in this section use our simple Distributed Bellman-Ford routing protocol for unicast routing. The multicast routing protocol is a dense mode variant of DVMRP []. For all sources in every multicast group, each node n i computes a parent-child relationship graph; the graph specifies whether n i is upstream or downstream of each of its neighbors in the shortest path spanning tree of any given source in a multicast group. n i uses the receipt of a unicast route update from a neighbour as a signal to recompute its parent-child relationships. The multicast algorithm is a flood and prune algorithm. Therefore, n i periodically floods multicast packets to all of its neighbours that are downstream to the packet source. Neighbours that do not have any members in the group will send a prune back to their upstream neighbour. Prune state associated with each of its neighbours s out every :s. In order to keep our current analysis simple, we have assumed a group density of one, i.e., there is an SRM agent at every node in the topology. Different aspects of the protocol are dependent on group density; in this paper, The simulation code for the experiments in this section are available at ~kannan/ code/ impact.tar.gz. The scripts used for processing this data are separately available at ~kannan/ code/ srm-scripts.tar.gz. 9

11 In the topology, the source is located at Node, all links are :Mbps, all links except Link h, i have ms delay, the delay on Link h, i is 8ms, Link h, i is dynamic, Links h, i and h, i periodically drop data packets from Node. Figure : Cyclic topologies used in the study of reliable multicast we explore the behaviour of the r mechanisms in the protocol, which are a greater function of the topology, than the group density. Hence, our results are not invalidated by this simplifying assumption. As a notational convenience, we do not distinguish between a node in the topology and an SRM agent running on that node in the topology. A constant bit rate source generates two packets per second, and is attached to Node. We generate periodic loss of the data stream to observe the behaviour of the adaptive rs. Links h,i and h,i drop every other packet from the source. This approach allows us to precisely quantify the loss. In this configuration, only Nodes,, and will experience all of the data losses and therefore attempt error recovery; the other nodes in the topology will attempt to repair the loss. The losses on the two links are independent of each other. 8 The data rate is set so that each loss and recovery will be complete before the source sends the next packet. Each of our simulations runs for s. In order to let the initial routing become quiescent, as well as to let the distance estimation algorithms in SRM converge, losses begin at t = s; we start our evaluations after t = s. For clarity, we only show the results till t = s in our plots. We ran each experiment s, with a different seed for the random number generator in our simulator each. Unless otherwise specified, our results are a plot of the average of the runs. These plots also include the results from each of the individual runs as tiny dots to illustrate the distribution. Recovery delay is the difference between when a node detects a lost data unit to the it actually receives the repair from another node. Figure 8 shows the average recovery delay for the nodes in the topology to recover from each loss. The x-axis corresponds to the approximate that the loss was detected by any of the nodes. This allows us to assign a unique stamp to that event; we use this idea in later evaluations to correlate other events in the simulation, such as topology change events. The y-axis indicates the recovery delay. In Figure 8(a) all of the nodes use fixed rs; In Figure 8(b) all of the nodes use adaptive rs. From the figures, we see that fixed rs has a constant recovery delay; adaptive rs gradually improve the recovery delay at all the nodes experiencing the loss. 8 In particular, note that we are using dense-mode multicast, based on periodic flood-and-prune. In our topology, the fallback link, Link h,i, is not usually used. Node will periodically flood data to its neighbour, Node across this fallback link. Node sends a prune whenever Node sends data packets across the fallback link; (the exception of course, is when the link must be used because some other link in the topology has failed). Link h, i will drop alternate packets seen in this periodic flood as well, and hence, over a period of, there may not be a one-to-one correlation between packets dropped on Link h,i and Link h,i.

12 seconds.8. seconds (a) Fixed Timers (b) Adaptive Timers Figure 8: Average recovery delay per loss units of rtt.8. units of rtt (a) Fixed Timers (b) Adaptive Timers Figure 9: Normalized recovery delay, expressed in units of rtt per loss Since the recovery delay (and by extension, the average recovery delay) is a function of which node initiates the request, and which other node performs the repair, this metric can have high variability. An alternate measure, used in [], is to normalize the recovery delay at a node, and express it in units of that node s rtt. In particular, for any given loss, we determine the last node to receive the repair, and normalize the delay experienced by that node its estimate of the rtt to the source. This is a measure of the worst case recovery delay experienced by these nodes. Figure 9 shows the plots of this metric for Fixed and Adaptive r mechanisms. From Figure 9(a), we see that, when using Fixed r mechanisms, the worst case recovery delay for any node is over.rtt to the source for that node. On the other hand, we can see from Figure 9(b) that the delay is within rtt for that node, when the nodes use adaptive rs. These figures illustrate some of the original motivation in [] for developing adaptive r mechanisms, i.e., to reduce the recovery delay, at the expense of a marginal increase in the number of duplicate requests. We therefore look at the number of requests and repairs sent by each of r mechanisms.

13 .... # messages. # messages (a) # of requests sent in fixed r mechanisms (b) # of requests sent in adaptive r mechanisms.... # messages. # messages (c) # of repairs sent in fixed r mechanisms (d) # of repairs sent in adaptive r mechanisms Figure : Request and repair counts per loss We can characterize a protocol s performance as a count of the number of request and repair messages that are sent for each loss. Figure shows these plots of request and repair messages for fixed and adaptive rs. In order to show all of the different points in the distribution, we plot the distribution as jittered dots. From Figure (a) and (c), we can see that fixed mechanism send few duplicate requests or repairs. On the other hand, adaptive rs send slightly higher number of requests (Figure (b)) in order to reduce the recovery delays following a loss. Figure (d) tells us that adaptive r mechanisms are more consistent in not sending duplicate repairs than fixed rs in our topology. From the figures for normalized recovery delay (Figure 9(b)) and request counts (Figure (b)), we see that adaptive r mechanisms tends to admit some number of duplicate requests and repairs in order to ensure that the recovery delay is within expected bounds. For the rest of this paper, we will evaluate the behaviour of the adaptive r mechanisms in response to topology change events. As an aside, Figure shows the traces from two specific simulations; each is a plot of the nodes that send the individual reques and repair messages for each loss. Each simulation is run using the same seed, Figure (a) shows the results using fixed rs, and Figure (b), using adaptive rs. In each case, we see that Node sends out most of the requests and Node does most of the repairs for fixed rs, which is as we expect in a string

14 node id Requests Repairs 8 node id Requests Repairs 8 (a) Fixed rs (b) Adaptive Timers Note that this plot is the result from a single simulation run, and does not indicate any distributions. Figure : Messages sent by each node for each loss parameters by node C C D D parameters by node C C D D 8 8 (a) Stable Topologies (b) Dynamic Topologies Note that this plot is the result from a single simulation run, and does not indicate any distributions. Figure : Parameter adaptation by each node: Adaptive rs topology. With adaptive rs, these nodes send out all of the requests and repairs, while Node also sends out the occasional duplicate requests. A final measure of protocol behaviour that could help improve our comprehensibility of the protocol is a plot of the request and repair parameters at each of the nodes. Figure (a) shows this plot for one run of the simulation. The y-axis shows the parameters for each node offset by an appropriate amount. Note that each node is only involved in request or repair for each loss. From the figure, we see that Node reduces its D and D, so that it is always the one to send out repairs. All other nodes that can repair a loss increase their D, so that they would rarely send out a repair message. Node reduces its C and C, so that it always sends out a request message. By reducing its parameters significantly, and rather quickly, Node ensures that the recovery delay is already fairly small. Over, Node adapts its parameters to enable it to send an occasional duplicate request. This explains the occasional duplicate request from Node that we see in Figure (b). We also notice from the figure (Figure (a)) that in stable topologies, the nodes adapt to the topology very quickly, and subsequently exhibit very little adaptivity. In contrast, Figure (b) shows the same parameter adaptation for the same simulation, when the topology is dynamic. We notice immediately that some of the nodes are continually

15 .. seconds.8. units of rtt (a) Average recovery delay (b) Normalized recovery delay Figure : Recovery delay per loss: Adaptive rs over dynamic topologies adapting to every change in the topology. In the following section, we will evaluate protocol behaviour over dynamic topologies.. Evaluation of the Adaptive Timer Mechanisms In this section, we repeat the experiments of the earlier section, when running SRM over dynamic topologies. For the sake of brevity, we concentrate on adaptive r mechanisms in the remainder of this paper. The choice of the network dynamics model for these experiments is guided by the following criteria: (a) the topology changes must occur in such a manner that, regardless of the topology, the same set of nodes experiences the losses and the same set of nodes can repair the loss; and (b) the interval between two successive topology changes should be sufficient to let the nodes adapt to the new topology. This permits us to study the adaptation of the r mechanisms. In addition to the above two criteria, it is also useful to have topology changes occur at predictable intervals. While this last criteria does not model any operational characteristic, it helps us isolate the effect of each topology change from the next. Therefore, for the rest of the experiment in this paper, we use a deterministic network dynamics model, one in which Link h, i periodically fails and recovers every s. The first event is the link failure at t = s. Link h, i will fail at t = s and t = 8s in our plots, and recover at t = s and t = s. During the intervals, [s; s] and [8s; s], the alternate higher delay path through Link h, i is used. In our plots, we show the at which topology changes occur as a vertical line at the appropriate instant in. Markers on the line (seen towards the top of the plot) indicate whether the link fails (goes down) or recovers (comes up); a label indicates the link that has changed state. Figure shows the plots of average and normalized recovery delays for the nodes experiencing loss in the topology. In particular, Figure (a) illustrates the expected increase in the average to recover from a loss every the alternate path through Link h, i is used. In these intervals, we see the average delay increase from :s to :8s. Correspondingly, Figure (b) indicates that the normalized delay increases from rtt to rtt.

16 8 8 # messages # messages 8 8 (a) # of requests sent in adaptive r mechanisms (b) # of repairs sent in adaptive r mechanisms Figure : Request and repair counts per loss: Adaptive rs over dynamic topologies By design, the increase in normalized delay will cause the nodes to attempt to send more duplicate requests in an attempt to reduce their recovery delay. Figure (a) confirms that the nodes send more than requests in the intervals when the normalized delay increases over rtt, and decreases to sending about. requests on average, at other intervals. In contrast, the number of repairs sent is about., and never changes significantly (Figure (b)). While the results in terms of the duplicate requests and repairs, and recovery delays are as expected, we can observe two anomalies: () exaggerated spikes at t = s and t = 8s. corresponding to the instants when Link h, i fails, and () increases in the number of repairs sent at the same instants in. The first anomaly is an artificial consequence of protocol behaviour operating over dynamic topologies. The second is a real consequence of protocol behaviour. We discuss each of these in turn. The anomalous spikes in the normalized delay that occur when Link h, i fails are an artifact of operating over dynamic topologies. The reason for these spikes is that, just after the link failure, the to recover increases for all the nodes. However, the distance estimates at each of the nodes is still the original, much smaller estimate. In particular, in our ring topology, the node that will get the repair last is the node that had the shortest distance estimate of all the three nodes before the failure. All three factors above, contribute to the exaggerated spikes in the normalized recovery delay at the instant of the link failure. Since the adaptive r mechanisms attempt to optimize normalized delay by sending duplicate requests, we might expect that the spikes in the number of requests at t = s and t = 8s in Figure (a) are reasonable. However, this does not explain the corresponding peaks in the number of repairs at those instants (Figure (b)). The reason for these becomes apparent when looking at a detailed trace of a single request and repair cycle following a link failure (Figure ). Recall that, just after Link h, i fails, the nodes have very short distance estimates to each other; this estimate is based on information learned before the topology change. Therefore, each of the nodes will set its rs too short. Nodes experiencing a loss will set multiple rounds of requests, sending a request after each round, and setting their rs again, and again, until they receive the repair. In a similar manner, nodes that can send the repair use correspondingly short distance estimates. They will set short hold-down s following the repair, and send multiple repairs, corresponding to each of the requests sent. We can see this clearly in the trace of Figure. The figure shows the lines at all of the nodes following a failure of Link h, i at t = s Nodes,, and go through at least two rounds of request, and they send three requests between them. Nodes,, and set their hold-down rs too short, and set and send multiple repairs. Node in particular

17 DETECT loss NACK r SEND NACK RECV NACK REPAIR r SEND REPAIR RECV REPAIR [ [ ] [ ] [ [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ This figure shows a possible sequence of events following a link failure, in which nodes set and send multiple rounds of requests and repairs. The x-axis shows the ; along the y-axis, we plot the line for each node, and the sequence of events at that node. The arrows indicate the duration of the hold-down r following the sending or receipt of a repair message. The square brackets show the interval from which a node sets its nack or repair r. In this topology, Nodes,, and send requests, and the other nodes send repair messages. Figure : Trace of messages for a single loss following a link failure 8 8 # rounds # rounds 8 8 (a) # rounds of request (b) # rounds of repair Figure : Maximum request and repair rounds per loss: Adaptive rs sends three repairs corresponding to each of requests it receives. Since the nodes execute multiple request and repair rounds, following a topology change, Figure plots the maximum rounds of request and repair executed by any node. From Figure (a), in dynamic topologies the nodes execute multiple rounds of requests; :8 request rounds when Link h, i is down, : request rounds at other s. As expected, we also see the transient rise in the number of rounds of repair, that we discussed in the earlier paragraphs occurs at t = s and t = 8s (Figure (b)). However, we also observe a secondary phenomenon in this figure; the nodes execute more repair rounds when all the links in the topology are up, and the distances between the nodes are optimal, i.e., in the intervals [s; 8s] and [s; s]. It is not immediately obvious to us as to why this occurs and requires further investigation; we have deferred this investigation for future work. We have already observed that it is critical for the error recovery mechanisms to obtain good estimates of their

18 distances to all other members. Distance estimation in SRM is achieved through the exchange of session messages by group members. It takes at least three iterations of session messages for two nodes to find their distances to each other. Our session message frequency in these experiments was one message every second; hence it comes as no surprise to find that the spikes in all of our graphs only last for a couple of losses. Appendix A describes the effect of the frequency of session messages on the loss recovery characteristics of the protocol. In practise, it is impractical for nodes to be sending such frequent session messages, if only because session messages consume expensive bandwidth; they also incur processing overheads at both the nodes that send it, and the nodes that receive and process the session messages. Therefore, we need to determine the optimal frequency of sending session messages, and balance the requirements of bandwidth for distance estimation against that required for other functions, such as error recovery. In this context, the idea of using scaled session messages such that nodes consume a limited amount of bandwidth when sending session messages, attempt to limit the scope of their session messages, and at the same designate a representative node to send more frequent session messages to the entire multicast group, bears promise [].. Recovery after Partitions In the previous sections, we considered protocol behaviour when the topology was stable (Section.), and when the topology was dynamic, but continuously connected (Section.). In this section, we study protocol behaviour when the topology is dynamic, and results in occasional network partitions. Partitions and partition healing are more complex situations for multicast protocols. This is because reliable multicast protocols are often designed to continue operation in spite of the partition. Contrast this with unicast protocols where, because of fate-sharing, nodes affected by a partition will make some number of attempts at recovery and then fail. Therefore, it is crucial to study the effect of the partition and healing on reliable multicast protocols. Before we describe our protocol evaluation of the adaptive r mechanisms in SRM, we must outline our simulation model that we will use in this section, i.e., the topology, sources, and loss patterns. Our protocol evaluation is based on our classification of the losses that occur due to network partitions. We also address the effect of the protocol behaviour on the network itself. For this experiment, we need a topology that is susceptible to partitions. We use the tree topology of nodes, shown in Figure. In this topology, Link h, i is dynamic. The periodic failure and recovery of Link h, i generates the partition events that are of interest to us. We use the same model of network dynamics as in the earlier section (Section.). Link h, i fails at s t = s and t = 8s resulting in partitions. The partition heals when Link h, i recovers at s t = s and t = s. Therefore, the topology is partitioned into two connected components in the intervals [s; s] and [8s; s]. The aim of the experiment is to study the impact of network partition and healing on the adaptive r mechanisms in SRM. Since the r mechanisms adapt to observed losses, we want a topology in which all the nodes continually receive some data, and there is intermittent loss of some fraction of that data, regardless of whether the network is fully connected or partitioned. This requires a source on either side of the partition, and some number of lossy links that periodically, but continuously, drop data from each of the sources. To satisfy these constraints, we place two constant bit rate sources, each generating two packets per second, one on Node, the other on Node. Link <, > is configured to drop every other data packets originating from Node ; likewise, Link h8, i is configured to drop every other data packets originating from Node. We can group the receivers in the topology by the pattern of losses they observe for data from each of the sources.