! g!gj. Observing TCP Dynamics in Real Networks. Abstract ,..

Transcription

1 Observing TCP Dynamis in Real Networks Jeffrey C. Mogul dewrl.de.om Digital Equipment Corporation Western Researh Laboratory 25 University Avenue Palo Alto, California, 9431 Abstrat 1. ntrodution Although the TCP protool has a relatively speiflat~on [14], and a Funtionally orret implementation The behavior of the TCP protool in simple situations is well-understood, but when multiple onnetions share a set of network resoures the protool an exhibit surprising phenomena. Earlier studies have identified several suh phenomena, and have analyzed them using simulation or observation of ontrived situations. This paper shows how, by analyzing traes of a busy segment of the nternet, it is possible to observe these phenomena in real life and measure both their freauenv and their effets on performane. A TCP implernent&ion might use simila; teh- niques to support rate-based ongestion ontrol. brief an be as short as ten pages of soure ode, the dynamis of TCP in real networks are not well understood. Real networks in partiular suffer from ongestion, that is, ontention for insuffiient resoures. While modem TCP implementations attempt to avoid or reover from ongestion, they do this using delayed feedbak mehanisms that exhibit omplex behavior. Most TCP implementations now in use inlude the ongestion avoidane and ontrol algorithms desribed by Van Jaobson [8]. Jaobson developed his algorithms by observing the behavior of TCP implementations attempting to send data aross a ongested network. n these observations, the ongestion was mostly due to a small number of onnetions, all sending data in the same diretion via a single shared bottlenek link. The essene of the ongestion avoidane algorithms is the observation that data pakets arrive at the reeiving host at the rate that the bottlenek link will support. f the reeiver s aknowledgements (ACKS) arrive at the sender with the same Permission to opy without fee all or part of this material is granted provided that the opies are not made or distributed for diret ommerial advantage, the ACM opyright notie and the title of the publiation and its date appear, and notie is given that opying is by permission of the Assoiation for Computing Mahinery. To opy otherwise, or to republish, requiree a fee and/or speifi permission. COMM 92-8/92/MD, USA 1992 ACM /92 /8 /35...$1.5 spaing, then by sending new data pakets at the same rate the sender an avoid overrunning the bottlenek link. t is by orretly exploiting this self-loking property of TCP that ongestion may be avoided (figure 1). nsender LB, * Slow,.. Reeiver AGKe have u! g!gj link The horizontat dimension represents time, and the shaded area is proportional to paket size (after [8]). Figure 1: TCP self-loklng Zhang, Shenker, and Clark [18] studied the slightly more omplex ase of a single link with TCP data flowing in both diretions at one. Their simulations showed that several surprising phenomena ould arise in suh a situation, even when Jaobson s algorithms were employed. One suh phenomenon they alled ACK-ompression. A TCP sender s self-loking depends on the arrival of ACKS at the same spaing with whih the reeiver generated them. f these ACKS spend any time sitting in queues during their transit through the network, however, their spaing may be altered (figure 2). When ACKS arrive loser together than they were sent, the senderl might be misled into sending more data than the network an aept, whih ould lead to ongestion and loss of effiieny. (This analysis assumes that aknowledgement pakets are small enough that queueing is the major ause of their delay.) lthe t~~~~ sender end reeiver will always be used wim respet to the data pakets, unless further qualified. That is, ACKS are sent from the reeiver to the sender. 35

2 f Slow link b,,,,, = t :;$:, t Related work Timothy Shepard s work on the problem of TCP paket trae analysis [16] extended the repertoire of graphial methods for visualizing TCP traes, and desribed a way to quantify the burstiness of a onnetion. Automated detetion of ACK-ompression relies on a similar method (see setion 2.3). Trevor Mendez [12] worked on the problem of identifying pathologial TCP onnetions among the many onnetions aptured in a long trae, onentrating on the retransmission behavior of TCP hosts. Figure 2: Self-loking disrupted by ACK-ompression The insights of Zhang et al. were based on simulations, Two other studies have deteted ACK-ompression in atual implementations. Heybey [7] made traes of onnetions through the atual ntemet, but these onnetions were set up speifially for the traes and might not have been representative. Wilder et al. [17] made traes of S1 TP4 onnetions, also in a testbed network. n all three ases, the results were analyzed by hand to detet interesting patterns, a tehnique that, while enlightening, annot be applied to more omplex situations. This paper desribes the results of a trae-based studies of large numbers of unontrived onnetions through the nternet. These were obtained by monitoring the pakets flowing in and out of a bttsy gateway system, widely used by sites all over the ntemet for eletroni mail and similar protools. Tools were developed to automatially analyze the traes to detet ACK-ompression and other interesting phenomena, and in some ases to show the orrelation between various phenomena in order to detet ausal relationships Motivation The primary motivation for these experiments was to determine if ACK-ompression ould atually be deteted automatially, and to disover if it auses any problems. The experiments show that ACK-ompression an indeed be deteted automatially from paket traes, and that it does happen in real networks. They do not show any serious onsequenes of ACK-ompression in today s ntemet, although this may be a result of our temporary inability to exploit the full bandwidth of the network. Other phenomena besides ACK-ompression, suh as synhronization of losses between several onnetions, out-oforder paket delivery, and some forms of improper TCP behavior, an also be automatially deteted from traes, as will be desribed in this paper, A seondary question is whether, if ACK-ompression does happen, there is anything that one ould do about it, A TCP implementation ould use tehniques, similar to the traeanalysis tools desribed here, to infer when aknowledgements are being ompressed. t ould then limit the rate of outgoing pakets to avoid overloading the bottlenek link. The emphasis of the disussion in this paper will be on the methods and tools used to detet ACK-ompression and related phenomena, rather than a broad study of the frequeny of ACK-ompression. Several studies have used traes to study the behavior of many simultaneous TCP onnetions in wide-area networks. Caeres et al. [1] analyzed traes to disover summary information about TCP onnetions, suh as the number of pakets transfemed or the paket interarrival time. Vem Paxon [13] did similar analyses of a set of traes, and alulated the distribution of per-onnetion average bandwidths. Muh ongoing work on TCP behavior ontinues to use simulation, sine trae analysis annot apture all the aspets of a network (suh as instantaneous window sizes and queue lengths). Reent studies inlude an analysis of traffi phase effets by Sally Floyd and Van Jaobson [6], Floyd s analysis of onnetions through multiple ongested routers [4], and the paper that desribes ACK-ompression [ 18]. Several simulation studies have identified other phenomena that might be visible to a trae-based analysis. For example, Shenker et al. [15] found in their simulations that when multiple onnetions use the same path, the pakets from a given onnetion tend to be lustered together, rather than interleaved with those of other onnetions. 2. Methodology Automati detetion of ACK-ompression depends on suessful approahes to several problems: 1. Choosing the monitoring point: The traes must be made of onnetions that have a reasonable likelihood of exhibiting ACK-ompression. 2. Obtaining traes: The traes should be long, must have high-resolution timestamp information, and should ontain essentially all of the pakets (i.e., there should be few dropped pakets). 3. Defining the phenomena: Unlike a simulation or a ontrived experiment, in these trae it is impossible to observe the atual spaing of ACKS at both ends of the network path. Therefore, ompression must be defined indiretly, based on observable aspets of the trae. The rest of this setion disusses the approah taken to eah of these issues, and similar issues in the detetion of related phenomena. Setion 3 will dkuss the tools that have been reated to analvze the traes. These tools must solve two problems: 1. Deteting the phenomena--in the traes: Algorithms must be reated to analyze the traes to detet when the defined phenomena are present. 2, Visualizing the results: Beause there may be thousands of onnetions and events represented in a trae, the results of the analysis should be presented in a form that allows easy and rapid understanding, 36

3 2.1. Choosing the monitoring point The ideal ~ite for obtainin~ - these traes would be one through whih passes many TCP ometions, between a wide variety of TCP hosts throughout the ntemet, and whih transfer more than just a few pakets worth of data at a time. (f the onnetions are too short, they will not generate enough ACKS to allow ompression.) Also, the traes should be made as lose as possible to the end-point of these onnetions, sine it is only there that the final spaing of ACKS may be observed. These riteria suggest monitoring at a major gateway omplex, one whih makes long TCP onnetions with sites all over the ntemet. Fortunately, the author has easy aess to the DECWRL gateway, operated by the Network Systems Laboratory of Digital Equipment Corporation. This site exhanges over 16, eletroni mail messages every day, as well as thousands of USENET news artiles. The gateway also inludes a popular anonymous FTP site, from whih several thousand files are retrieved eah day. Although most mail messages are too short to give rise to interesting TCP behavior, the gateway also provides an FTPby-mail faility that produes about 15 relatively lengthy messages every day. News artiles are also short, but they are sent in bathes via the NNTP [9] protool, and so give rise to long onnetions. Many FWP transfers involve large files. Sine the traes are being made at one endpoint of the TCP onnetions, ACK-ompression an only be observed on those onnetions that arry bulk data away from the gateway omplex. This halves the number of potentially interesting ACKompression events in the trae. Additional traes were made on a subnet of Digital s internal P network, adjaent to a host that serves as the primary USENET distribution enter for the ompany, as well as several heavily-used eletroni mail hosts. Wti-le fewer longdistane onnetions are visible to these traes than at the DECWRL gateway site, many of the internal paths traverse low-bandwidth, ongested links, and so the frequeny of ACK-ompression might be muh higher than on the- nterned. During the initial phases of this study, a question arose as to whether the TCP window sizes in ommon use would be large enough to stimulate ACK-ompression. Many hosts use window sizes of just 4k bytes, whih might be too small to eliit the phenomenon, but quite a few hosts now use 16k-byte windows. (n ottr traes, at least half of the onnetions used windows of 16k bytes or more.) n any ase, although the NSFNet an transport 15-byte datagrams to many sites, most TCP hosts use a segment size of 512 bytes in order to avoid possible fragmentation; this means that even with a relatively small window size of 4k bytes, up to eight pakets may be in transit at one. Our results show that for whatever reason, the issue of window size has not been an impediment to the detetion of ACK-ompression. One advantage of making the traes at an endpoint site, rather than at a bakbone, is that by definition that site is a party to all of the onnetions traed. This simplifies the issue of obtaining permission, but it would be unethial (and perhaps illegal) to monitor the ontent of the onnetions without the permission of both parties [2]. The traes therefore ontain only paket header data, and not anything that ould be used to ompromise the privay of the messages traed Obtaining traes The traes were obtained by promisuous-mode monitoring of 1 Mbit/sex Ethernet networks. This passive monitoring tehnique eliminates any effet that the traegathering proess might have on the onnetions being traed, t also means that no hanges need be made to the hosts whose ometions are traed Traing environment The trae-gathering program runs on any ULTRXTM version 4.2 system, and for these experiments was run on DECstationm 31 workstations. During the traes, little additional ativity was present on the trae-gathering system. Trae files were written to disk for later analysis, rather than being transferred over the network in real-time (whih might have affeted the network being traed). Some simple arithmeti will demonstrate that to detet ACK-ompression, the traes must ontain high-resolution timestamps. For example, suppose that a TCP onnetion is using 15-byte pakets over a path with an effetive bandwidth of 1 Mbit/se. This means that properly-spaed ACKS will be sent approximately 1.2 mse apart. A timestamp resolution oarser than about half of that will make some properly-spaed ACKS appear to be ompressed together, sine they will arrive during the same tik. ULmux normally updates its timebase just 256 times per seond, yielding a resolution of about 4 mse. To obtain better timestamps, the ULTRH kernel used on the traing systems was modified to use a 496 hz timebase, yielding a resolution of about 244 use. Additionally, the traes reord the length of eah paket; this allows some subsequent deompression of timestamps beause the minimum spaing between two pakets, as they were transmitted on the monitored Ethernet, an be alulated from their lengths (see setion 3). For the traes made on Digital s internal P network, the DECstation 31 used was equipped with speiallyonstrttted lok hardware. This provides a basi timestatnp resolution of 1 nse, although the usable resolution is not quite this good beause the timestamps are applied by the kernel and so are smeared by the lateny of the interrupthandling ode Creation of a trae file The traing program is oneptually quite simple. The P header of eah TCP paket reeived is parsed to obtain soure and destination addresses, and the TCP header is parsed to obtain the soure and destination ports, the sequene and aknowledgement numbers, the flags, and the window size. Sine the addressing information is repetitive, the program maintains a map from TCP address tuples (the two P addresses and the two TCP port numbers) to a ompat onnetion identifier. (At any one time, thk is enough information to uniquely identify a ometion, but during the life of a trae there may be several ometions that reuse the same address tuple. n a post-proessing phase, it is easy to split the trae up into distint onnetions, beause the TCP protool has rttles to prevent any onfusion.) Whenever a new onnetion is seen, the traing program emits a reord ontaining the new 37

4 onnetion D and the assoiated addressing information. For eah paket reeived, the program emits a reord ontaining the timestamp, onnetion D, and the non-address fields from the TCP header. Note that the onnetion D identifies one diretion of paket flow; TCP onnetions are always full-duplex, and the relationship between the two half-onnetions an be rereated in a post-proessing phase. The onnetion D mapping is done using a hash table, with 124 hash slots eah pointing to a hain of entries. A simple hash funtion on the address information distributes the entries remarkably evenly, with no hain more than three times longer than the average. A one-behind ahe bypasses the hash table lookup whenever two or more suessive pakets belong to the same onnetion. n spite of the ompression of address information, the trae files tie still quite bulky. A file representing a trae of 1 million pakets requires just under 32 Mbytes of disk spae. n order to avoid luttering the trae with irrelevant information, the trae program an be made to ignore pakets between pairs of loal hosts, where loal refers to a host whose address is the same as the traing host, when ompared under a given bit mask. For traes made at the DECWRL gateway site, thk mask overed the P network number; for traes made internal to Digital s P network, the mask overed the network and subnet numbers Potential problems with passive monitoring Passive monitoring tehniques an miss pakets if the throughput of the monitoring system is insuffiient, or if there is not enough buffering in the monitor to handle the oasional burst of pakets. When looking for AC!K-ompression, it is espeially important to apture all the pakets in a burst, sine the ompressed ACKS will by definition be reeived in a burst. The ULTRX kernel provides a moderate amount of queueing to absorb bursts, and keeps an aurate ount of the number of pakets dropped from the queue. The traing program emits an error reord, ontaining the ount of dropped pakets, whenever this ours, and so the analysis tools an reonstrut exatly how many pakets were dropped and approximately when this happened. n the traes made for this paper, the trae-gathering system was able to keep up with the traffi. n no ase were even.2~ of the pakets dropped. While it is impossible to tell if the dropped pakets oneal an unusually high number of ACK-ompression events, it is unlikely that this is the ase. Another problem with passive monitoring is that it oasionally aepts pakets with heksum errors. While it is theoretially possible to have the traing program hek the header heksums on eah paket, in pratie this would so slow the program that it would ertainly lose far too many pakets. (Cheksumming is now onsidered to be the most expensive part of proessing a TCP paket [3].) Statistis kept by the gateway mahines show that pakets with bad heksums, while present, are quite rare. The danger for this experiment is that a damaged paket might give rise to a trae reord that onfuses the analysis tools. The tools take some are to ignore onnetions that ontain bizarre pakets, but it is possible that orruption of a few bits ould ause false onlusions to be drawn on rare oasions Defining the phenomena When ACKS are observed arriving at the end of their transit through the ntemet, it is impossible to know preisely how they were spaed when they were sent. (When ACKS are observed near the reeiver, we an tell their spaing, but we then do not see them arriving at the other end and so an say nothing at all about their possible ompression.) This means that the trae analysis tools must infer ACK-ompression from indiret evidene. Choosing a satisfatory definition is thus key to the utility of the analysis, and proved to be somewhat diffiult. What we an observe, by looking at the number of bytes aknowledged by a set of ACKS and the time spaing between them, is the apparent network bandwidth that these ACKS promise to the sender. (This is preisely the information that the sender uses to meter out its data traffi.) This is alled the instantaneous ACK bandwidth of the observed ACKS. We want to know if this value is muh higher than the atual bandwidth of the path, whih would indiate that ACKompression has ourred. Another thing we an observe is the atual progress of the onnetion over intervals long ompared to the TCP window size. Clearly, the bandwidth obtained during suh an interval annot be higher than the atual available bandwidth. f we an measure this true bandwidth, then we an define ACKompression as ourring when the instantaneous ACK bandwidth is signifiantly greater than the true bandwidth. The simplest way of alulating the bandwidth of a onnetion, given a trae of all its ACKS, is to divide the total number of bytes aknowledged by the total time required. Unfortunately, most onnetions do not proeed at a steady rate, and so this average rate is almost always far too small. Plots of ACK-value versus time for atual TCP onnetions show phases of relatively steady progress (diagonal lines) interspersed with dead periods (horizontal lines). Clearly, what we want is to ignore the dead periods when omputing the true bandwidth. Previous work by Shepard [16] suggests a solution. n order to visualize the burstiness of TCP onnetions, he plotted the distribution of the instantaneous bandwidths for a onnetion. The x-axis of these plots shows the number of bytes arried in a paket divided by the time sine the previous paket; the y-axis shows the total number of bytes transferred at a given instantaneous bandwidth. Sine the dead periods transfer few or no bytes, they ontribute little to the distribution, and the peaks of the distribution show the approximate true bandwidths attained during various phases of ~he onnetion. f there is one main peak to this distribution, we an find its approximate position on the x-axis by taking the median of the distribution. This provides a simple, automati mehanism for estimating the true bandwidth of a onnetion that transfers a lot of data over a stable path. t is possible that if the onnetion s rate is atually limited by something external to the network, suh as a dkk drive on one of the end systems, this mehanism will underestimate the true bandwidth. There is not muh one an do about this, if the information in the traes is all that is available. One ould in priniple set up probe onnetions along the paths followed by interesting onnetion, and attempt to measure the 38

5 available bandwidth; this is not always possible, and was not done for this study. Anyway, the results suggest that the alleged ACK-ompression events are often real, sine they are often assoiated either with subsequent segment loss or with periods of redued throughput. f they were a onsequene of an underestimate of the available bandwidth, one would expet to see them assoiated with periods of inreased throughput. Some additional filtering redues falsely-deteted ACKompression. First, we an insist that a ompression event last for at least a ertain number of pakets (i.e., the ACKbandwidth is measured over a window of A pakets, where A might be greater than the minimum of two.) Seond, we an insist that the ACK-bandwidth exeed the estimated true bandwidth by a ratio R (suh as 5 or 1) in order to orret for serious underestimates of this value. Third, we an insist that a ompression event show a ertain minimum ACKbandwidth B, in order to omit onnetions that don t atually transmit muh data. This threshold should be set slightly above an estimate of the lowest link-bandwidth in the network, sine setting it muh higher ould hide ACK-ompression on low-bandwidth links Segment loss and out-of-order delivery We would like to be able to detet TCP segment losses sine these are the main onsequene of improper segment spaing. Sine we annot dketly know when a segment has been lost, we have to infer segment loss from the observed retransmission of a segment. Note also that if a segment flowing into the monitored site is lost, we will never know this beause even if we reeive a retransmitted opy, without having seen the original we will not reognize the retransmission as suh. n other words, we an only detet retransmission on ometions that transfer data out of the monitored site. Fortunately, sine we an detet ACK-ompression on only these same full-onnetions, it should be possible to detet orrelations between ACK-ompression and segment loss, if the orrelations exist. We define retransmission to have ourred if we see a segment whose sequene number lies within the ranges of bytes transmitted by another segment. We an also detet out-of-order deliverv of a se~ment (presumably beause of multiple paths throu gh the Xmet). This is deemed to have happened when we see a sequene number arrive that preedes a previously-reeived sequene number, and does not overlap with another segment. 3. Analysis Tools One the traes are made, they must be analyzed to detet the phenomena of interest. Analysis is done offline, beause a onnetion annot be analyzed until it is omplete, and beause on-line analysis would interfere with paket-gathering and ause paket loss. Many kinds of off-line analysis an be applied to a single trae file. onnetions in a trae: Aggregate statistis: For eah onnetion, we ompute the total duration, number of bytes sent (i.e., the net hange in sequene number), number of bytes aknowledge (the net hange in aknowledgement number), and the median ACK-bandwidth. ACK-ompression: Given parameter values for the ACKompression detetion window, minimum ACKbandwidth, and minimum ratio to the median, we find those ACK pakets that appear to have been ompressed. Segment retransmission, out-of-order delivery: By examining sequene numbers, paket lengths, and timestamps we dkover those segments that have been retransmitted or have arrived out of order. These primary analyses an then be ombined in several ways (for example, to find orrelations between ACKompression and segment loss) and the results then plotted Data strutures to support analysis Eah analysis tool starts by reading a saved trae file. An in-memory database is built up from the onnetion and paket-header reords in the file. For eah new onnetion, a onnetion reord is reated and inserted into a hash table (for rapid loation based on the ompat onnetion D.) Eah paket-header reord is stored in a header reord, and added to the end of a linked-list of headers attahed to the appropriate onnetion reord. Sine the trae file is ordered by paket arrival time, the lists of headers are also time-ordered. During this proess, the timestamps being read in are heked to make sure that thev. are no loser to~ether than the minimal possible spaing on an Ethernet. (f traes were made on a different LAN, suh as FDD, this alulation would have to be modified.) f one header reord follows the previous one with a timestamp differene shorter than would be possible based on the length of the previous paket, the timestamp of the new reord is inreased aordingly. Another problem with the raw trae data is that while at any given time the TCP address tuple uniquely identifies a onnetion, one a onnetion is terminated the same address tuple may be reused shortly thereafter. Sine the sequene numbers from the two onnetions bear no relation to eah other, in order to make sense of the sequene numbers the onnetions must be separated. This is done by searhing for tell-tale signs of the end or beginning of a onnetion: pakets bearing the FN or SYN flags, or large jumps in the sequene or aknowledgement numbers. The largest legal jump in a TCP sequene number is the maximum window size of 64k bytes. Any jump larger than this suggests that a new ometion has been made, although it is also possible that the trae-gathering program has either dropped a bunh of pakets, or has aepted a orrupted paket. n either ase, sine we are laking suffiient information to analyze the entire onnetion, splitting into two onnetions avoids the problem of data integrity. Several different primary analyses an be applied to the ZRemember that the term onnetion refers in this ase to a half-duplex, one-way paket flow. An atual TCP onnetion is represented by two of these half-onnetions. 39

6 One in a while, a trae ontains data that just doesn t make sense. For example, the final sequene or aknowledgement number is muh lower than the initial value. While this ould our beause of sequene-number wraparound, it an also be due to orrupted data, so the tools simply disard the entire onnetion in this ase. The in-memory database of onnetion and header reords an be quite large. Eah header reord in its in-memory form ontains 36 bytes; this ould be redued somewhat with ompression tehniques, but that would make the whole proess run slower. As a result, a trae of 1 million pakets requires somewhat more than 36 Mbytes of virtual address spae, and sine the loality of referene to this database is poor, the analysis must be arried out on a mahine with at least that muh real memory. The largest mahine available for these experiments, a DECstation 5/2, has 48 Mbytes of RAM, so traes of up to about 1 million pakets ould have been analyzed, but suh traes would have required a lot of disk storage. The running time of the analysis programs depends heavily on the ost of reading the trae file and onstruting the internal database. On the systems with 48 Mbytes of RAM, the file system s buffer ahe an hold an entire trae file of 1 million pakets, so the ost of an analysis depends entirely on CPU speed. On the DECstation 5/2, the simplest analysis (not requiring the omputation of median ACKbandwidths) takes about 3 seonds, 2 %. of whih is spent in the reado system ah and 3~ of whih is spent orreting the timestamps and inserting header reords into the database, (The more omplex analyses take several minutes, beause the algorithm used for finding medians is quite rude.) 3.2. Deteting the phenomena in the traes One all of the onnetion and header reords have been reated, passes an be made over the entire database, or over the header reord list for a speifi onnetion Deteting ACK-ompression f the analysis requires detetion of ACK-ompression in a onnetion, the first pass omputes the median of the instantaneous ACK-bandwidths. First, the program alulates the instantaneous ACK-bandwidth for eah paket with the ACK flag set; this is omputed by dividing the differene in aknowledgment numbers between suessive ACK pakets by the time elapsed between their reeption, The program then reords the instantaneous bandwidth samples in an array, and sorts the array. Finally, it loates the median value, weighted by the number of bytes aknowledged at eah rate. (A more sophistiated mediamfinding algorithm might redue the time required for an analysis, but sine it takes less than 9 seonds to analyze a 1-million paket trae, this has not been a problem.) The next pass again traverses the list of header reords and alulates the ACK-bandwidth, but this time instead of using the instantaneous differene between two suessive pakets, the bandwidth is alulated over the ACK window size A, speified as a parameter to the analysis, For example, if A = 3, then the ACK-bandwidth is alulated between an ACKpaket and the third following ACK-paket. f the instantaneous bandwidth exeeds the onnetion s median by a speified ratio R, and also exeeds a speified minimum value B, the header reords in the window are flagged as being ompressed Deteting segment retransmission and out-of-order delivery A separate pass over the header reord list is made to find retransmitted TCP segments. During this pass, an array is reated ontaining an entry ontaining the sequene number and length of eah paket, a serial number indiating the order of reeption, and a bak-pointer to the assoiated header reord. This array is then sorted, using the sequene number as the primary key, the length as the sonda~ key, and the reeption order as the tertiary key. One more pass over the sorted array suffies to detet retransmissions and out-of-order delivery: f an array entry s sequene number falls within the range of bytes arried by a previous paket, then the entry is a retransmission. (We ignore pakets with SYN set, sine typially a lot of these are sent during an attempt to make a onnetion with a dead host.) The earlier paket is flagged as having been lost, and the later paket is flagged as a retransmission. During the pass over the sorted array, we an also detet and flag pakets delivered out of order, sine these are ones whose reeption order is earlier than their predeessor in the array (whih is sorted by sequene number), if the predeessor is not a retransmission. 4. Visualizing the results One the automated analysis has found onnetions with anomalous behavior, one needs some way to visualize the results. Visualization methods for these experiments an be divided into those showing just one onnetion, and those that show all (or a subset) of the ometions. Also, the methods an be divided into graphial presentations, whih abstrat a lot of the details of individual pakets, and textual presentations, whih provide detailed insights but do not easily reveal interesting patterns. All four ombinations of sope and presentation are useful Graphial presentation of single onnetions Van Jaobson [8] seems to have pioneered the use of plotting seauene numbers versus time to visualize a single TCP onnetion. Sine the automated analysis desribed above marks paket header reords that are part of ACKompression, segment-loss, or out-of-order delivety events, the Jaobson plots are easily extended to show when these events our. Note that sine ACK-ompression happens to pakets arrying aknowledgements, whih flow on the other half-onnetion from the pakets that suffer from segment loss and out-of-order delive~, it is profitable to plot on one graph both the sequene numbers from one half-onnetion, and the aknowledgement numbers from the partner halfonnetion. An example of one suh plot is shown in figure 3. The sequene-number plot is a solid line, and the aknowledgement-number plot is a broken line. (This ometion starts with a period of about 25 seonds during whih few 31

7 r 1- ; time (seonds sine start of onnetion) Figure 3: Sequene numbers vs. time for single onnetion bytes are transfemed.) Regions of ACK-ompression are bounded by + symbols, segment losses are marked with X symbols. n olor media, these symbols an be made muh easier to distinguish, and olor may also be used to highlight regions of the urve suffering from ACKompression. Examination of suh plots quikly provides insight into suh issues as the atual underlying bandwidth of the onnetion (from the slope of the diagonal segments of the urve), and the relationship between ACK-ompression and overall performane. For example, if ACK-ompression were deleterious, one would expet to see dereases in the slope, or segment loss events, shortly after an instane of ACKompression. n figure 3, this seems to be the ase shortly after the first ACK-ompression event, an entire window-full of pakets is sent in a brief interval, and all of these pakets are lost. Following Shepards s approah [16], we an also plot the distribution of instantaneous ACK bandwidths, as in figure 4. This more diretly reveals the underlying network bandwidth, if one assumes that the spikes near 1 Mbyte/see are the result of ACK ompression rather than an indiation of the true bandwidth. 8 Paket traes an be diffiult to analyze, even for an expert, but some help an be provided. For example, although TCP S 32-bit sequene numbers are hard to deal with diretly, the tpdump program [ 11 ] reports not the absolute sequene numbers but the relative value sine the first observed paket of a onnetion. t seems even more useful to report the differene in sequene numbers between two suessive pakets, beause that an be diretly related to the amount of data transferred. Also, by showing the two halves of a onnetion in side-byside olumns, following Shepard s example [16], one gets a piture of the time-ordering of the pakets. By displaying relative instead of absolute timeststmps, the short-term dynamis are made apparent. Figure 5 shows part of the same onnetion as figure 3, paket-by-paket. n figure 5, the sender s pakets are shown on the left, and the reeiver s on the right. The first olumn numbers the pakets, for onveniene in explanation. The seond olumn shows the timestamp, in seonds sine the preeding paket. The next two olumns show the data length (exlusive of headers), and the inrement in sequene number. The final olumn on the sender s side marks lost or retransmitted pakets (with L or R, respetively). On the reeiver s side, the first olumn shows the inrement in aknowledgement number, and the next olumn shows the instantaneous ACKbandwidth. The final olumn on the reeiver s side marks ompressed pakets (with C ). On lines 1 through 12, we see the series of data pakets being sent, then a burst of ACKS (lines 13 through 18). The last olumn shows that some of these ACKS (lines 14 through 17) appear to have been ompressed, and indeed most of the pakets in the next burst (lines 19 through 3) are lost (marked with L ). After a 3.5 seond timeout, the sender realizes this and enters a slow-start phase (starting with line 3 1), whih revives the transfer of data Q 1CSO 1e+6 1 ACK bandwidth (bytes/see) 17 Figure 4: nstantaneous bandwidth distribution (same onnetion as fig. 3) 4.2. Textual presentation of single onnetions Sometimes the Jaobson plots do not show enough detail to understmd the preise dynamis of a onnetion. n this ase, it helps to look at a trae of the individual pakets of a onnetion, showing the exhange of pakets in both dketions in the orret time sequene. 3A11 the ACKS in a ompression event are marked with C, inluding those whih arrive first and thus whose instantaneous ACK bandwidth is small (beause they have been delayed the longest). 311

8 TimestamD Sender Reeiver One an also plot distributions of various statistis for the len seq loss? ak ACKbw omp? onnetions in a trae (figures 9 and 1)4. Some of these are C C OXX) C6123.CK) L L L L L L L L L L 324 L CX :; R R R R CQ C R R R R R 324 R e R useful for providing a ontext for understanding the important of ACK-ompression. The data in plot C show that most onnetions transfer only a few thousand bytes (about 75~ of the onnetions transfer less than 1k bytes), and so are unlikely to suffer from ACK-ompression. Plots G and H, respetively, show that very few onnetions suffer from segment retransmission or out-of-order delivery Statistial analyses of multiple onnetions t is interesting to ask whether there are frequent orrelations between ACK-ompression and segment losses. One ould obtain timestamps for all the ompression events and all the loss events in a trae, and try to establish if a statistially signifiant orrelation exists. A more diret approah is to look at eah full onnetion that experienes both ompression and segment loss, and analyze the timestamps to see if there is a ausal relationship between ompression and loss. nferene of a ausal relationship an be done in a number of ways. One would be to assume that any pakets lost in the next TCP window after a ompression event are aused by that event, but it is not always possible to be sure that those pakets were not sent before the ompressed ACKS arrived at the sender. Another approah is to measure the time between an ACK-ompression event and the next segment loss event. One ould then ompare this interval to an estimate of the round-trip time. Even simpler is to see if the interval is small ompared to the median interval between loss events; this last method shows that in many ases there does seem to be a ausal relationship. For example, in one trae made on Digital s internal network, 63 onnetions experiened at least one loss event subsequent to a ompression event, and in 42 of those ases the time between a ompression event and a subsequent loss event was less than the median time between losses. (These intervals between reeption of ompressed ACKS and transmission L of a lost segmen~ are typial~y on the order of several milliseonds or tens of milliseonds.) CQ Simulations by Shenker et al. [15] suggested that when multiple TCP onnetions are sharing a ongested resoure, their ongestion-re every behavior ould beome Figure 5: Partial paket trae (same onnetion as fig. 3) synhronize~, with eah &netion loosing a paket at eah 4.3. Graphial presentation of multiple onnetions Although the sequene-number urve in figure 3 is useful for displaying one onnetion, an entire trae might ontain thousands of onnetions, and it would be impossible to study all those urves. There is a variety of more useful ways to display the aggregate behavior of many onnetions. One interesting question is how sensitive the ACKompression detetion algorithm is to the three input parameters: the ACK-window size A, the minimum ACKbandwidth 1?, and the minimum ratio R of instantaneous to median bandwidths. Figures 6, 7, and 8 show how the total number of ompression events deteted vanes with hanges in two of these parameters (while the third is held onstant). Observe that varying A or R an hange the number of alleged events by several orders of magnitude, while varying B has muh less of an effet (until it beomes lose to Ethernet rates). ongestion epoh. The timestamped segment-loss events obtained from traes ould, in priniple shed some light on this effet. Synhronized segment loss an be deteted by making an array of all the loss events in a trae, sorting this array by timestamp, and then looking for losses from different onnetions that our within no more than a speified interval. (The hoie of this interval is an arbitrary parameter, beause of ourse no two pakets an be lost at preisely the same instant,) 4T avoid expanding the vertial sale, plots C through D do not inlude samples for whih the x-axis value is zero (e.g., plot C only inludes those onnetions that transferred at least one byte). The figures inlude parenthetial notes giving the number of omitted samples. 312

9 L 1~ R=2 ACK window= t 1 [ 1 1~ 1~ ; le+6 ACK bandwidth threshold (bytes/see) Figure 6: Sensitivity of ACK-ompression analysis (ACK-window size held onstant) 1~ 1 El Bandwidth threshold = 1 1 ~ A=3 A=4 A=5 1-1 JA= ACK bandwidth minimum ratio Figure 7: Sensitivity of ACK-ompression analysis (minimum bandwidth held onstant) 1 1 Minimum ratio = ~. 1g t 1 11 t le+6 le+7 ACK bandwidth threshold (bytes/see) A=2 Figure 8: Sensitivity of ACK-ompression analysis (minimum ratio held onstant) 313

10 1 8 ~ 8 (832 onnetions witi zero bandwidth) 6 d o 2 (yj $ t i 4 % 4 B % 2 s Oooil Conneuon duration (seonds) (A) Distribution of onnetion durations YJ 1 1e+6 Average ACK Bandwidth (bytes/see) (E) Distribution of average onnetion bandwidths z,2 6 8 g J 4 % g g 2 z (837 onnetions wnh zem bandwidth) U 1CO 1 Paket ount (B) Distribution of paket ounts o lcqooo le+6 Me&an ACK bandwidth (bytes/see) (F) Distribution of median bandwidths (198 om:tions sending zero bytes) Total bytes transferred -6 Number of retransmitted segments (C) Distribution of onnetion lengths in bytes (G) Distribution of lost segment ounts $. % 6CQ j 4 2 (34544 zero-length segments) & 8 (2757 onnetkms with no out-of-order segments) g g 8 6 % J 4 E z ht. bytes p., segment 11 (D) Distribution of segment lengths Number.f segments delivered OUr of rdex (H) Distribution of out-of-order delivery ounts Figure9: Distributions forallhalf-onnetions inone trae Figure 1: Continuation of Figure 9 314

11 n the trae mentioned above, out of 5142 losses, in 287 ases losses from two or more onnetions ourred less than 1 mse apart. t is not lear, however, if this reflets a statistially signifiant orrelation between losses from multiple ometions. Figures 11, 12, and 13 show for this trae the distribution of interarrival times (respetively) for paket losses among all onnetions, for two losses enountered by a single onnetion, and for two losses where seond ours to a different onnetion than the first. The spikes at 46 use and 12 use reflet the bak-to-bak transmissions of 576-byte and 15-byte pakets, respetively: when TCP sends a burst of pakets, the entire burst is likely to be lost as a group. Aside from these spikes, the data might reflet a Poisson distribution, but it may be too noisy to demonstrate a lear orrelation between losses from differing onnetions. 6, 5. Summary of Experiments Traes were made on several days at both the external and internal loations. The traes are summarized in table 1. Note that the ount of pakets refers to the number saved for later analysis, not the number that went by on the network. Times are Paifi Standard Time; durations are in seonds. Dropped pakets are those disarded beause the monitoring system ould not keep up; there is no way to know preisely what kinds of pakets were dropped. The figures in setion 3 are all drawn from Trae 11, sine the longer traes produe more onfusing graphs. Tables 2 and 3 summarize the number of lost segments deteted. They also show the number of ACK-ompression events and ausally-orrelated segment-loss events for two different ombinations of ACK-ompression detetion parameters. D Losses Compressions Correlations ?ukkudJ o.2 O.W Time between losses fmm any onnetion (seonds) ~ % Figure 11: Loss interarrival times, all onnetions ~ 2 -, z O.W Time between losses fmm same ometien (seonds) Figure 12: Loss interarrival times, single onnetion El E ACK-window A = 3 Bandwidth threshold B = 1Kbytes/se Minimum ratio to median R = 5. Table 2: Summary of deteted events D Losses Compressions Correlations El E ACK-window A = 5 Bandwidth threshold B = 1Kbytes/se Minimum ratio to median R = 1. Table 3: Summary of deteted events 14 x ~ 1 %8 8 $6 $4 2,CQ2.Q Time between losses from different onnetions (seonds) Figure 13: Loss interarrival times, differing onnetions As a hek on the reliability of the method that looks for ause-effet relationships between ACK-ompression and segment loss, plots were made of the 12 onnetions in Trae E2 identified as having suh orrelations even with the striter parameters used in table 3. nspetion of these plots suggests that in about half of the ases, a ausal relationship is likely, and ould not be ruled out in several other ases. The possibility exists that some of the deteted instanes of ACK-ompression are atually periods during whih the available bandwidth is muh higher than normal. This is probably true in some ases, but there are definitely onnetions for whih the reverse is true. Figure 14 shows a onnetion for whih there is a period with many ACK-ompression events. During this period, the average bandwidth is far lower than it 315

12 D Site Date/Time Duration Pakets Connetions Dropped pkts 11 nternal 2 Jan 92, 13: look nternal 2 Jan 92, 1: lm nternal 3 Jan 92, 1: lm El External 2 Jan 92, 1: lm E2 External 3 Jan 92, 1: lm Table 1: Summary of traes le+7 8e+6 / /, median ACK-bw!= 7269 bytes/see 6e+6 / 4e+6 / / 2ei-6 / o time (seonds sine stat of ometion) Figure 14: Connetion with redued average bandwidth assoiated with frequent ACK-ompression is during other phases of the onnetion, whih do not exhibit ACK-ompression. ntuition suggests that ACK-ompression should indeed be assoiated with periods of redued available bandwidth, sine the ompressed ACKS result from non-empty queues on the return path. nreased queue length should inrease the roundtrip time, thus dereasing the available bandwidth [5]. 6. Future work 6.1. Appliation to ongestion avoidane Sine ACK-ompression and subsequent paket loss ours even on networks that are believed to be unontested over the long term, this implies that portions of the ntemet experiene temporary ongestion over short periods. These periods may in fat be shorter than the response time of feedbak-based ongestion-avoidane shemes, whih means that some form of rate-based ongestion avoidane may be required. While most proposals for rate-based shemes have the network provide expliit rate-limit information to the end-hosts, it may be possible for an end-host to infer the true available bandwidth and thus set its own rate limit. n effet, the endhost ould use the ACK-ompression detetion methods desribed in this paper to avoid being misled by ompressed aknowledgments. This may prove diffiult, for several reasons. The sending host needs:. A high-resolution lok in order to measure the arrival times of aknowledgements and thus ompute ACKbandwidths. An effiient way of omputing the median ACKbandwidth as the onnetion evolves. An effiient, aurate mehanism to throttle the rate of transmitted pakets, slow enough to avoid overrumting the bottlenek link but fast enough to keep the pipe full. The trik will be to provide mehanims that are suffiiently aurate to avoid ACK-ompression, yet suffiiently heap to avoid reduing the transfer rate when ACK-ompression is not a threat. f one were designing a new transport protool to replae TCP, it might pay to inlude timestamps on aknowledgement pakets. The reeiver would timestamp eah ACK to reflet the arrival time of the aknowledged segment. The sender would use the ACK timestamps, rather than the times at whih they arrive at the sender, to ontrol the rate at whih data segments are sent. This would, in effet, onvert the real- ife situation depited in figure 2 into the idealized situation depited in figure 1, sine queueing delays on the return path would not affet the self-loking mehanism. A 32-bit timestamp, saled for a resolution of 1 nse, would not wrap around during the maximum lifetime of an P paket Other interesting analyses The information in the trae files an be analyzed to disover other interesting information about TCP dynamis. For example, the instantaneous round-trip time (RTT) for a onnetion an be estimated by measuring the time from the transmission of a segment to the reeipt of an aknowledgement for the orresponding sequene number. (When segments are lost, there are a number of ompliations that arise, for whih good heuristis are known [1].) t would be interesting to know if there are orrelations between the variations in RTT among several onurrent onnetions; this would be expeted if multiple ometions through the same newly-ongested router suffer queueing delay simultaneously. There might also 316

13 be orrelations between jumps in the RTT and segment loss or ACK-ompression. Another open question is whether ACK-ompression events our nearly simultaneously on multiple onnetions. One might expet this to be the ase, and the onsequenes might worse than for isolated ompression events (sine the resulting burst of data pakets would be multiplied by the number of onnetions involved). One might ask what fration of segment losses are aused by ACK-ompression, either diretly (i.e., the onnetion suffering losses is the one that transmits too many pakets) or indiretly (an innoent bystander loses pakets when another onnetion overflows a queue). Answering this ould be diffiult, sine it would require oordinated observations at many points in the network. 7. Conlusions The results of this study show that it is indeed possible to detet ACK-ompression in traes made on real networks. Moreover, it is possible to orrelate ACK-ompression events with subsequent paket losses, even on networks that are not partiularly ongested. Both detetion of ACK-ompression and orrelation with paket loss an be done automatially, whih means that it is possible to apply these tehniques to traes of busy networks over lengthy periods. f, in the future, the ntemet is operated muh loser to its long-term apaity, the onnetion between ACK-ompression and ongestion may beome aute, and automati detetion of ACK-ompression ould be a key to effiient utilization of the network. Aknowledgements would like to thank Lixia Zhang, Sally Floyd, and K. K. Ramakrishnan for their numerous helpful omments on early drafts of this paper. Comments from anonymous SG- COMM reviewers also helped to guide the presentation. Andrei Broder and Susan Owiki provided valuable onsultation on how to interpret the data obtained. Referenes [1] Ramon Caeres, Peter B. Danzig, Sugih Jamin, and Danny J. Mitzel. Charateristis of Wide-Area TCP/P Conversations. n Pro, SGCOMM 91 Symposium on Communiations Arhitetures and Protools, pages Zurih, September, [2] Vinton G. Cerf. Guidelines for nternet Measurement Ativities. RFC 1262,ntemet Ativities Board, Otober, [3] David D. Clark and David L. Tennenhouse. Arhitetural Considerations for a New Generation of Protools. n Pro. SGCOMM 9 Symposium on Communiations Arhitetures and Protools, pages Philadelphia, September, 199. [4] Sally Floyd. Connetions with Multiple Congested Gateways in Paket-Swithed Networks Part 1: One-way Traffi, Computer Communiation Review 21(5):3-47, Otober, [5] Sally Floyd. Private ommuniation r61 Sallv Flovd and Van Jaobson. Traffi Phase Effets in P~ket-S~hh&i Gateways. Computer Communiation Review 21(2):26-42, April, [7] Andrew Heybey. Private ommuniation [8] Van Jaobson. Congestion Avoidane and Control. Pro. SGCOMM 88 Symposium on Communiations Arhitetures and Protools, pages Stanford, CA, August, [9] Brian Kantor and Phil Lapsley. Network News Transfer Protool. RFC 977, ntemet Ativities Board, February, [1] Phil Kam and Craig Partridge. mproving Round-Trip Time Estimates in Reliable Transport Protools. ACM Transations on Computer Systems 6(4): , November, [11] Steven MCanne and Van Jaobson. An Effiient, Extensible, and Portable Network Monitor. Work in progress. [12] Trevor Mendez. Detetion of Pathologial TCP Connetions using a Segment Trae Filter. Computer Communiation Review 22(1):28-35, January, [13] Vem Paxson. Measurenwnts and Models of Wide Area TCP Conversations. Tehnial Report LBL-384, Computer Systems Engineering Depannent, Lawrene Berkeley Laboratory, May, [14] Jon B. Postel. Transmission Control Protool. RFC 793, Network nformation Center, SR nternational, September, [15] Sott Shenker, Lixia Zhang, and David D. Clark. Some Observations on the Dynamis of a Congestion Control Algorithm. Computer Communiation Review 2(5):3-39, OCtober, 199. [16] Timothy Jay Shepard. TCP Paket Trae Analysis. MT/LCSflR- 494, Laboratory for Computer Siene, Massahusetts nstitute of Tehnology, Februa~, [17] Rik Wilder, K. K. Ramakrkbnan, and Allison Mankin. Dynamis of Congestion Control and Avoidane of Two-Way Traffi in an S1 Testbed. Computer Communiation Review 21(2):43-58, April, n [18] Llxia Zhang, Sott Shenker, and David D. Clark. Observations on the Dynamis of a Congestion Control Algorithm: The Effets of Two-Way Traffi. n Pro. SGCOMM 91 Symposium on Communiations Arhitetures and Protools, pages Zurih, September,