Root Cause Analysis for Long-Lived TCP Connections

Transcription

1 Root Cause Analysis fo Long-Lived TCP Connections M. Siekkinen, G. Uvoy-Kelle, E. W. Biesack, T. En-Najjay Institut Euécom BP 93, 694 Sophia-Antipolis Cedex, Fance ABSTRACT While the applications using the Intenet have changed ove time, TCP is still the dominating tanspot potocol that caies ove 9% of the total taffic. Thoughput is the key pefomance metic fo long TCP connections. The achieved thoughput esults fom the aggegate effects of the netwok path, the paametes of the TCP end points, and the application on top of TCP. Finding out which of these factos is limiting the thoughput of a TCP connection efeed to as TCP oot cause analysis is impotant fo end uses that want to undestand the oigins of thei poblems, ISPs that need to toubleshoot thei netwok, and application designes that need to know how to intepet the pefomance of the application. In this pape, we evisit TCP oot cause analysis by fist demonstating the weaknesses of a peviously poposed flight-based appoach. We next discuss in detail the diffeent possible limitations and highlight the need to account fo the application behavio duing the analysis pocess. The main contibution of this pape is a new appoach based on the analysis of time seies extacted fom packet taces. These time seies allow fo a quantitative assessment of the diffeent causes with espect to the esulting thoughput. We demonstate the inteest of ou appoach on a lage BitToent dataset. Categoies and Subject Desciptos C.2.3 [Compute Communication Netwok]: Netwok Opeations Netwok monitoing; C.2.5 [Compute Communication Netwok]: Local and Wide-Aea Netwoks Intenet Geneal Tems Algoithms, Measuement, Pefomance Keywods Intenet, TCP oot cause, thoughput, time seies.. INTRODUCTION Motivation: Duing ecent yeas, the Intenet taffic has expeienced a massive gowth as the numbe of uses skyocketed. This Pemission to make digital o had copies of all o pat of this wok fo pesonal o classoom use is ganted without fee povided that copies ae not made o distibuted fo pofit o commecial advantage and that copies bea this notice and the full citation on the fist page. To copy othewise, to epublish, to post on seves o to edistibute to lists, equies pio specific pemission and/o a fee. CoNEXT 5, Octobe 24 27, 25, Toulouse, Fance. Copyight 25 ACM X/5/...$5.. applies also to the amount of taffic pe use, as the capacities of access links have inceased by seveal ode of magnitudes. New Intenet applications, such as pee-to-pee (P2P), have emeged and the elative impotance of HTTP and FTP has deceased. On the othe hand, TCP still emains the dominating tanspot potocol that conveys the vast majoity of the taffic typically moe than 9% of the bytes. As a consequence, the behavio and pefomance of TCP in the Intenet is a majo concen. The heteogeneity of today s applications unning on top of TCP and the divesity of the envionments they ae used in, imply that a meaningful analysis can only be done duing thei opeation in the Intenet. Thoughput is the most impotant pefomance measue fo long TCP connections. The achieved thoughput epesents the aggegate effects of the netwok path, the end points, and the application. Ou eseach ties to find out which of them ae esponsible fo limiting the thoughput of a given TCP connection at a given time instant. Knowledge about these oot causes can be used by Intenet Sevice Povides (ISP) fo quality of sevice evaluation and toubleshooting. Taffic modeling is anothe aea that would benefit fom this knowledge, leading to moe accuate wokload models of TCP taffic. It is equally impotant fo Intenet application designes to know when the limiting facto is the netwok o the TCP end points and when it is the application. Appoach: We adopt an appoach that equies as input bidiectional packet heade taces captued at a measuement point along the path fom souce to destination and poduces as output quantitative infomation about the limitation causes of TCP s thoughput pe connection. We assign a scoe to each of these limitations and tack the evolution of limitation scoes with time. Fo this pupose, we base ou appoach on a set of time seies geneated fom the (TCP and IP) packet heades. We focus on long tansfes whee TCP slow stat no longe dominates the thoughput achieved. As stated above, we distinguish thee main classes of oot causes: (i) limitations due to the application, (ii) limitations due to the TCP end-hosts and (iii) limitations due to the netwok. We apply a divide and conque appoach to infe the limitation causes. Fist, the peiods whee the thoughput is detemined by the application ae isolated. The emaining tace data consists of so called bulk tansfe peiods. We then apply a set of tests to deive the most likely cause (o causes) that explains the pefomance of each bulk tansfe peiod. Wheneve possible, the algoithms ae validated using live measuements in the Intenet that ae compaed against the esults given by Web []. We also used NIST Net [4] to ceate specific conditions fo a given tansfe. Zhang et al. [8] pefomed pioneeing eseach wok into the oigins of Intenet TCP thoughput limitation causes. They defined taxonomy of ate limitations (application, congestion, bandwidth,

2 sende/eceive window, oppotunity and tanspot limitations) that we build on and intoduced the TCP Rate Limitation Tool (T-RAT). T-RAT tuned out to suffe fom a numbe of limitations. Fist, to identify ate limitation, T-RAT needs to identify so called flights of packets. These flights often cannot be identified, as we will see in section 3, which undemines the main pemise of T-RAT. Second, T-RAT beaks long connections into flows of at most 256 consecutive packets. In contast, we pefom tue connection level analysis. Challenges: The poblem is vey challenging fo seveal easons. Opeating at connection level complicates the analysis because with long connections, we have a highe pobability to obseve seveal limitation causes ove diffeent peiods of time. Fo instance, some Intenet applications such as BitToent [8] o HTTP. opeate by switching between active tansfe peiods and passive keep-alive peiods. Ou fist challenge is to detect those diffeent peiods and analyze them sepaately. The geat numbe of paametes that influence the behavio of a TCP connection is also a majo issue: ound-tip time (RTT), eceive advetised window, link capacities, available bandwidth, delayed acknowledgment, and TCP vesion to name a few. Contibutions: The contibution of this pape is theefold: Fist, in section 2, we discuss the poblem of infeing causes fo TCP s tansmission ate limitation by elaboating moe on the limitation concepts themselves with espect to the appoach taken by T-RAT. Second, in section 3, we demonstate though simulations that an impotant chaacteistic of a TCP tansfe, the so called flights, have in many cases a vey diffeent fom than the one assumed in T-RAT. Thid and most impotantly, we povide a set of algoithms to infe causes that limit the thoughput of a given TCP tansfe (sections 4 and 5), and apply the algoithms to an example set of eal Intenet taffic (section 6). 2. CAUSES FOR RATE LIMITATION In this section, we discuss the diffeent ate limitation causes that we want to infe. While the classification is inspied by T-RAT, we extend the scope of thei wok, and exemplify the difficulties of identifying cetain causes o assessing the impact of othes. We pesent the causes in a top down manne, stating fom the application level down to the netwok level. 2. Application The application opeating on top of TCP can be the cause fo the thoughput achieved. In this case, TCP is not able to fully utilize the tanspot o netwok laye esouces because the application does not poduce data fast enough. Thee exist two scenaios whee the application is the limitation cause. In the fist scenaio, the application is poducing small amounts of data at a elatively constant ate fo the TCP laye. This esults in small busts of packets, in the exteme case a single packet of size less than the maximum segment size of the connection. Typical examples ae live steaming applications such as Skype [] that tansfe data ove TCP at a constant ate of 32 Kbit/s. Also, applications that use pemanent TCP connections and send keep-alive packets duing inactive peiods, fall in this categoy (BitToent exhibits this behavio duing choke peiods -see [8]). In the second scenaio the application is poducing data in busts sepaated fom each othe by idle peiods. An example of such behavio is web bowsing with pesistent HTTP connections.the sequence numbe :5:27 Figue : A piece of a eceive window limited connection. use clicks on a link to load a web page, causing a tansfe peiod, eads the page, causing an idle peiod, and clicks on anothe link, causing anothe tansfe peiod. 2.2 TCP End-Point Limitations The achieved thoughput of TCP can be limited by the size of the buffes allocated at the two end-points of a connection. The eceive buffe (between the TCP laye and the application laye) constains the maximum numbe of outstanding bytes the othe end is allowed at any given time instant. On the othe hand, the sende buffe (between the TCP laye and the MAC laye) constains the maximum numbe of bytes in the etansmit queue. Consequently, the size of the sende buffe also constains the amount of unacknowledged data that can be outstanding at any time. Following the convention of T-RAT, we call the fist limitation eceive window limitation and the second one sende window limitation. If the tansmission ate of a connection is limited by a window size (eithe sende o eceive window limitation), the sliding window of TCP will be consistently smalle than the bandwidth delay poduct of the path. Figue shows a time vs. sequence diagam of an eceive window limited connection. The staicase-like lines indicate the left (lowe) and ight (uppe) limit of the sliding window and the vetical aows epesent data segments that wee sent. Since most of the time, the lines fo the data segments tansmitted coincide with line tacking the uppe limit of the sliding window, the sende is eceive window limited. Sende and eceive window limitations esult in the same obsevable behavio. As we will see in section 5, identifying a eceive window limitation is possible using the advetised window infomation caied by TCP packets. On the othe side, identifying a sende window limitation is a much moe complex task, that we do not addess in this wok. In [8], the authos use the notion of flight size to infe a sende-window limitation. Howeve, we will see in section 3 that identification of flights is, most of the time, impossible. We expect that fo most tansfes in the Intenet, the sende buffe to be at least the size of the eceive window. Indeed, in most Unix implementations of TCP, the minimum size fo the sende buffe is 64 Kbytes, which is equal to the maximum eceive window size when the window scale option (RFC 33) is not used. When the window scale option is used, a coect implementation of a TCP stack should esize the sende buffe when eceiving the window scale facto of the othe side. Howeve, a e- time

3 cent study [2] has obseved that 97% of the hosts that suppot the window scale option used a window scale facto of, meaning that the maximum eceive window was at most 64 Kbytes. Thee is an additional type of limitation at the tanspot laye that is efeed to as oppotunity limitation in T-RAT. This limitation occus fo shot connections caying so few bytes that the connection neve leaves the slow stat phase. Since it is the slow stat behavio of TCP that limits the ate of the TCP tansfe we do not classify these connections as application limited. As will become clea late, we concentate on analyzing long TCP connections in which case oppotunity limitation will not be an issue. 2.3 Netwok Limitation A thid categoy of limitation causes fo the thoughput seen by a TCP connection ae due to the netwok. We focus on the case whee one o moe bottlenecks on the path limit the thoughput of the connection (see [7] fo a study on the location and lifetime of bottlenecks in the Intenet). While othe netwok factos, such as link failues o outing loops [6], might impact a TCP connection, we do not conside them in the pesent wok as we can easonably expect thei fequency to be negligible as compaed to the occuence of bottlenecks. Fo the following, we need to boow a few definitions fom [5]. We define fist geneal metics independent of the tanspot potocol: capacity C i of link i: the maximum possible IP laye tansfe ate at that link end-to-end capacity C in a path: C = min,...,hc i, whee H is the numbe of hops in the path the aveage available bandwidth A i of a link i: A i = ( u i)c i, whee u i is the aveage utilization of that link in the given time inteval the aveage available end-to-end bandwidth A in a path: A = min,...,ha i, whee H is the numbe of hops in the path We also define two TCP specific metics: bulk tansfe capacity (BT C i) of link i: the maximum thoughput obtainable by a single TCP connection at that link bulk tansfe capacity (BT C) of a path: BT C = min,...,hbt C i, whee H is the numbe of hops in the path As in [5], we call the link i with the capacity C i = C the naow link of the path and the link j with the the aveage available bandwidth A j = A the tight link of the path. Futhemoe, we define link k as the bottleneck link if it has a bulk tansfe capacity BT C k = BT C. Note that while at a given time instant, thee is a single bottleneck fo a given connection, the location of the bottleneck as well as the bulk tansfe capacity at the bottleneck can change ove time. Please note that the bottleneck link is not (necessaily) the same as the tight link. Also the available bandwidth diffes fom the bulk tansfe capacity of a path: The classical example is the case of a link of capacity C used at % by a single TCP connection. The available bandwidth is zeo on the link while the bulk tansfe capacity should be C 2. If the bottleneck link explains the thoughput limitation obseved fo a connection, we declae this connection as netwok limited. Packet losses ae natual indicatos of a bottleneck and we will use the packet loss ate as a measue of the impact of the netwok on sequence numbe time Figue 2: A piece of a bandwidth limited connection whee packets ae egulaly spaced due to the bottleneck link. a connection. Howeve, note that the packet loss ate by itself does not fully chaacteize the impact of congestion on the thoughput of a given connection. Especially, two connections with diffeent RTT values will not see thei thoughput affected in the same way even if they expeience a simila loss ate, as can be seen diectly fom the TCP thoughput fomula []: T put = MSS C RT T p, whee MSS is the maximum segment size of the connection, C is a constant and p is the loss event pobability (which is elated to the loss ate, while not simila, as it indicates the fequency of loss peiods whee one o moe packets ae lost). Fo the above easons, we will use two metics to infe if a connection is netwok limited: (i) the etansmission ate of the connection and (ii) the dispesion atio (see section 5.2) that can be used to detect if the bottleneck link is shaed o not. Figue 2 shows a time vs. sequence diagam of a connection whose thoughput is limited by a non-shaed bottleneck link. The egula spacing between sent data segments and similaly of the acknowledgments eceived (tacked by the ight limit of the sliding window) is easy to obseve. 3. ON THE FLIGHT NATURE OF TCP It is commonly assumed that TCP tansfes packets in flights, i.e. in goups of packets that ae sent back to back within a goup. This is justified by the window based flow and congestion contol mechanisms used in TCP. Flights ae a vey impotant notion fo T-RAT as it needs to elate the flights to the diffeent phases of TCP, namely slow-stat, congestion avoidance, and loss ecovey. Howeve, in [4], the authos seach fo flights in Intenet taffic taces and aive to the conclusion that flights can be aely identified, which means that a tool such as T-RAT is unable to function popely. In the pesent wok, we investigate the notion of flights though simulations, and come to a simila conclusion: while it is possible to obseve goups of packets it is difficult to elate them to the well-known phases of TCP. We simulated TCP connections limited by a specific cause using ns-2 and vaied diffeent paametes (RTT, eceive advetised window, TCP vesion, etc.) affecting the behavio of the connection. Ou objective was to study the similaity of the signatues of connections limited by the same cause but having diffeent paamete values. By signatue, we mean the distibution of packet inteaival times (IATs). Fo example, in the case of a eceive window limited connection one would expect to obseve a bimodal disti-

4 (a) The Basic Configuation fom this fist expeiment that elating flights to one of the phases of TCP is a difficult task. We next conside a moe ealistic scenaio (Figue 3(b)) with coss-taffic using the web client-seve class of ns-2 at node. Tuning the paametes of the clients, we simulated diffeent load values. Figue 6 shows an example evolution of the pobability density function (pdf) of the inte-aival times of packets when inceasing the offeed load of the coss-taffic. In these simulations the delayed acknowledgments mechanism is used as this is the most common case. The loss ate fo the ftp connection expeiencing coss-taffic was zeo fo all cases of offeed load. The main obsevation fom these plots is that even with small amounts of coss-taffic the stuctue of the pdf (and consequently of the goups of packets) is much moe complex than in the fist simple scenaio. In geneal, coss-taffic adds to the queuing delay, which lowes the modes (i.e. ceates much moe diffeent goup sizes). This means that the flight sizes become moe complex to identify, which makes it difficult to tack the size of the congestion window. These simulations futhe confim that it is often impossible to ely on the flight sizes to identify the state of the TCP connection. Theefoe, T-RAT 2 [8] will be unable to wok popely in many cases. (b) The Configuation With Coss Taffic Figue 3: The Configuations fo the Simulation. bution of the IATs, with the pincipal mode at t = S and the C (W ) S seconday mode at t 2 = RT T, whee S is the packet C size (typically can be assumed to be equal to MSS), C the capacity of the naow link, and W is the eceive advetised window. The pincipal mode coesponds to the time it takes to tansmit a single packet on the naow link on the path. As all the packets of a single window should be sent back to back in a single flight, thei IATs coespond to this value. The position of the second mode coesponds to the time inteval between obseving the last packet of the pevious flight and the fist packet of the next flight. Moeove, the atio of the heights of these peaks should be close to a facto of W because fo each window woth of packets one obseves W times an IAT of t and one time an IAT of t 2. In the following, we show a few examples to demonstate that this type of simple easoning aely holds. We stat with a simple topology with one client (node ), seve (node 2), and one intemediate oute (node ) shown in Figue 3(a). A two-minute long FTP tansfe was set up on top of a TCP connection established fom node 2 to node. Figues 4 and 5 show histogams of IATs of packets whee the connection is limited by the eceive advetised window of 2 packets. In Figue 4, delayed acknowledgments wee not used by the TCP eceive while in Figue 5 delayed acknowledgments wee used. As expected, in Figue 4 we obseve the two modes at t = 5.ms and t 2 = 83.7ms and the atio of thei heights is appoximately 2. Howeve, if the TCP eceive is delaying acknowledgments the situation becomes moe complex. We can still obseve the pincipal mode t in Figue 5 but instead of a single seconday mode we obseve seveal additional modes. Due to the delayed acknowledgment time at the eceive, the set of W packets sent is divided into seveal smalle sets of packets sent back-to-back. The numbe of these goups of packets depends on the atio of the RT T to the delayed acknowledgment time value but also on W. We can aleady conclude 4. TIME SERIES-BASED APPROACH Ou appoach to infe the TCP oot causes of a given connection is to geneate a numbe of time seies using the packet tace of the connection. We then use these time seies to compute scoes that chaacteize the impact of the diffeent causes. As the location of the measuement point on the path impacts the way the time seies ae geneated, we devote the next subsection to this issue. We then biefly pesent all the time seies used in ou tests, along with some tests made to validate some heuistics used to deive these time seies. 4. Measuement Point Location Most of the time seies that we use, ae vey easy to compute if the packet tace was captued at the sende side. Howeve, we do not want to limit ouselves to this specific case as, fo instance, we may use publicly available taces collected by thid paties and fo which we do not have exact infomation about the measuement configuation. The fist poblem then is to infe the location of the measuement point. Let us conside the case depicted in figue 7, whee the measuement point A is close to the sende while points B and C ae not. We can detemine the measuement point with espect to the connection initiato by measuing and compaing the delay between the SYN and SYN+ACK packets, and the delay between SYN+ACK and ACK packets, efeed to as d and d2, espectively, fo the measuement point A. We conclude that A is close to the connection initiato if d2 d <.. Moe geneally, we need to compute the RTT samples fo each connection. In the case that ou measuement point is close to the sende, we compute the RTT fo each acknowledged data packet as the time inteval between the timestamps of the last tansmission od a data packet and the fist acknowledgment fo this data packet 3. In We compute the pdf estimates using a kenel density estimation technique [5] with Gaussian kenel function. 2 No publicly available vesion of T-RAT has been eleased so fa. 3 Hee we must take into account that packets may be sent multiple times, in the case of losses, and similaly acknowledged multiples times, in the case of lost o piggybacked acknowledgments.

5 no coss taffic f(x) faction of samples faction of samples inte aival time medium coss taffic (offeed load.5) f(x) t t 2 inte aival time inte aival time t inte aival time Figue 4: Without Delayed ACKs Figue 5: With Delayed ACKs Figue 6: Evolution of the pdf of the inte-aival times of packets fom a eceive window limited connection without and with coss taffic. d d2 Sende A SYN d3 ACK(+Data) B d4 SYN+ACK Receive Figue 7: Detemining the measuement position fom the thee-way handshake of TCP the case that ou measuement point is not close to the sende and TCP timestamps ae available, we implement the method descibed in [7]. 4.2 Time Seies In this section, we list all the time seies that we use in ou tests to find the oot causes fo the thoughput seen by a TCP connection. Faction of Pushed Packets: A pushed TCP packet is sent with a PUSH flag. RFC-793 says: The sending use indicates in each SEND call whethe the data in that call (and any peceding calls) should be immediately pushed though to the eceiving use by the setting of the PUSH flag.. Pushed packet thus indicates that the application on top of TCP has fo the moment no moe data to send. We compute fo each diection of a TCP connection the time seies of the faction of pushed packets obseved ove all consecutive non-ovelapping time intevals of fixed duation. To compute those factions, we only conside packets caying data and discad pue acknowledgments. If no packets have been seen duing a given time window the value is set to. C d5 d6 Inte-aival Times of Acknowledgments: We compute the inte-aival times of acknowledgments sepaately fo each diection of a connection. The ACKs included in the computation ae eithe acknowledging one o two data packets of size MSS o duplicate acknowledgments. Futhemoe, we cancel the effect of delayed ACKs by dividing by two the inte-aival time of ACKs that acknowledge two data packets. Retansmission Rate: We compute fo each diection of a TCP connection the time seies of the etansmission ate as the faction of etansmitted bytes pe all (data) bytes tansmitted in consecutive time intevals of second. A packet is consideed to be a etansmission if (i) the packet caies an end-sequence numbe lowe than o equal to any peviously obseved one; and (ii) the packet has an IPID value [2] [3] highe than any peviously obseved values. Note that we cannot ely on obseving etansmitted packets twice and counting them since the packets may be lost befoe the measuement point especially if the measuement point is fa fom the sende. With the help of the IPID we emove false positive etansmissions caused by eodeing of packets by the netwok that can occu if the measuement point is fa fom the sende. Receive Advetised Window: We compute the time seies fo eceive advetised window, which consists of time-weighted aveaged values ove a given time inteval: Each time a packet is eceived fom the othe end, the eceive window indication in the packets will be consideed as the actual eceive window value until eithe the end of the time window occus o the eception of a new packet. This technique is valid if the measuement point is located at the sende side. Howeve, if the measuement point is away fom the sende, we vitually shift in time the obseved timestamp values by the time delay between the sende and the obsevation point. Fo example, in Figue 7, when the measuement point is at C, we would shift in time the timestamp values of packets sent by the eceive by + d6, which is the estimated time at which this 2 packet should aive at the sende. Note that d6 will be estimated using the technique boowed fom [7] as indicated in section 4. Outstanding Bytes: Anothe value of inteest is the amount of data bytes sent and not yet acknowledged at a given time instant. Since the computation is done by inspecting both diections of the taffic, we need to take into account the location of the measuement point. If the measuement point is close to the sende, we poduce the time seies by calculating the diffeence between the highest data packet sequence numbe and the highest acknowledgment sequence numbe seen fo each packet and then aveaging these values ove a time window in the same way that we do fo the eceive advetised window values.

6 If the measuement point is away fom the sende, we do the computation by shifting in time the timestamp values of aiving packets. Fo example, in Figue 7, we would shift the timestamp values of data packets aiving fom the sende at C by d6 and of 2 acknowledgments aiving fom the eceive at C by + d6. 2 The above algoithms (at the sende and at the eceive side) ae heuistics that we tested with eal tansfes on the Intenet. We analyzed scp tansfes fom a Web enabled machine to anothe machine and an tcpdump at the sending o eceiving machine. Web is a kenel patch that allows to access the actual intenal vaiables of the active TCP connections of a host. It povides the exact values fo the sending TCP s etansmission queue size, which coesponds to ou definition of outstanding bytes. We did scp tansfes and compaed the values obtained fom Web and ou algoithms: fom Institut Euecom to Univesity of Oslo in Noway and fom Euecom to Univesity of Navaa in Spain. As the tansfe to Spain poved to be ove a lossy path (with appox. 6% of etansmitted bytes) and the one to Oslo not, we wee able to captue two diffeent envionments. Figues 8(a) and 8(b) show the compaison in the case whee the measuement point is at the sende and eceive side, espectively. Unfotunately, we wee unable to dump taffic in the machine located in Oslo and pesent only the esults fom the tansfe to Spain in the eceive side case. In Figue, 8(a) the two cuves fo the tansfe to Spain ae plotted on top of each othe, indicating a nealy pefect ageement. In the case of the tansfe to Oslo thee is a diffeence of appoximately one MSS on aveage (note that the window scale option was negociated between the two paties in this expeiment). The eason fo this discepancy is not clea and may be due to timestamp inaccuacies caused by the high thoughput of this tansfe. Figue 8(b) fo the case whee the measuement point is at the eceive side also shows a good ageement between the estimated and actual values. Fom the above expeiments, we conclude that, in most case, we can expect to obseve a maximum a discepancy between the actual and estimated values that emains below one MSS, a good enough pecision fo the tests based on these time seies (see section 5.2.). 5. IDENTIFYING AND ANALYZING BULK TRANSFER PERIODS In this section, we show how we use the time seies intoduced in section 4 to sepaate bulk tansfe peiods fom application limited peiods. We also intoduce the diffeent tests fo TCP end host and netwok limitations. 5. Identifying Bulk Tansfe Peiods The fist opeation we pefom on a connection is to sepaate peiods limited by the application fom othe peiods. We identify the active phases of a connection whee TCP consistently tansfes and call them bulk tansfe peiods. We identify bulk tansfe peiods using the time seies of factions of pushed packets using time window of.5 seconds. A smalle value fo the duation of a time window would isk to intepet the idle time due to the sende waiting fo new acknowledgments afte having sent a congestion window full of packets, as an indication of application limitation. The algoithm used to sepaate bulk tansfe peiods fom application limited peiods vaies between two states: active and inactive. We define the stating state to be inactive. The algoithm switches to the active state (stat of a new bulk tansfe peiod) if the faction of pushed packets is consistently below a value p fo t consecutive time peiods. The algoithm switches back to the inactive state (end of the cuent bulk faction fom total samples faction of pushed packets fom all packets Figue 9: Histogam of the time seies values of the factions of pushed packets fo all connections of a GB BitToent packet tace. tansfe peiod, and stat of a new application limited peiod) if the faction of pushed packets obseved has been consistently above p o if no taffic was sent fo t 2 consecutive time peiods. The algoithm is able to ecognize both types of application limited peiods discussed in Section 2 since we conside both the idle time and the atio of pushed packets. Howeve, the thesholds p, t, and t 2 need to be tuned accoding to the type of input taffic and the focus of the analysis. In section 6, we pesent esults fo a Gbytes BitToent tace and we chose p as follows: We extacted the time seies of the factions of pushed packets duing one-second time intevals fo all those connections of the tace that had moe than data packets (we do not pocess these small connections in any case, see Section 6). We then computed a histogam of all these values, (Figue 9). Based on this histogam we set p to.7 but clealy choosing any value fom [.5,.95] would pactically give the same esult. We set t = 5 seconds and t 2 = seconds, which implies that we discad any tansfe peiods shote than five seconds. 5.2 Bulk Tansfe Peiod Analysis Afte sepaating bulk tansfe peiods fom application limited peiods, we apply the tests fo TCP end-point and and the netwoking limitations on the identified bulk tansfe peiod. These limitation tests ae not exclusive. Each of the tests yields a scoe between and that quantifies the level of the limitation, which is a majo impovement ove T-RAT [8] that was only poviding qualitative esults, i.e. a binay answe fo each test Receive Window Limitation We use two time seies to test fo eceive window limitation: the outstanding bytes time seies and the eceive advetised window time seies. The diffeence of the values of these two time seies indicates how close the TCP sende s congestion window is to the limit set by the eceive window. Specifically, fo each pai of values in the two time seies, we compute thei diffeence and geneate a binay vaiable with value one if this diffeence is less than n MSS and zeo othewise (in section 6 we discuss the impact of the n value). The eceive limitation scoe is the aveage value of the esulting binay time seies fo the analyzed bulk tansfe peiod.

7 x Web s et queue vs. outstanding bytes at the sende 4 web, Navaa outst. bytes, Navaa 9 web, Oslo outst. bytes, Oslo web outst. bytes Web s et queue vs. outstanding bytes at the eceive bytes 5 bytes time (s) time (s) (a) Measuement point is at the sende. (b) Measuement point is at the eceive. Figue 8: Validation of the outstanding bytes algoithms Netwok Limitation We use two metics to infe whethe the netwok limits the thoughput of a connection: (i) the etansmission scoe and (ii) the dispesion scoe. Retansmission scoe: The etansmission scoe fo a bulk tansfe peiod is computed as the atio of the amount of data etansmitted divided by the total amount of data tansmitted duing this peiod. Note that since TCP may pefom unnecessay etansmissions, etansmission scoe does not exactly coespond to the loss ate. Howeve, we can expect these quantities to be close to one anothe in geneal and especially if the vesion of TCP uses SACK. Dispesion scoe: The objective of the dispesion scoe is to assess the impact of the bottleneck on the thoughput of a connection. We intoduce this facto stating fom the simple case of a nonshaed bottleneck in the netwok, next moving to the case whee thee is a shaed bottleneck which is the naow link of the path up to the geneal case whee the bottleneck link is not the naow link of the path. The dispesion scoe is computed fom the times seies of the inte-aival times of acknowledgments and the aveage thoughput tput of the bulk tansfe peiod unde consideation. Let us fist conside the case whee the netwok limitation consists of a non shaed bottleneck on the path. The bottleneck is evidently the naow link of the path. Let be the capacity of the naow link. The histogam of the inte-aival times of acknowledgments (computed as explained in Section 4.2) should exhibit a mode located at MSS that contains most of the mass. Since the bulk tansfe peiod is netwok limited and the naow link is not shaed, the atio of tput to should be appoximately equal to,. We define the dispesion scoe as tput. i.e. tput Conside now the moe complex case whee the bottleneck link is still the naow link of the bulk tansfe peiod but it is now shaed 4. The histogam of the inte-aival times of acknowledg- 4 In pactice, detecting this case is not an easy task and would ement should still exhibit a mode located at MSS. Howeve, since the bottleneck is now shaed, this mode will contain a smalle faction of the total mass of the histogam. Also, the atio tput epesents the shae the connection obtains at the naow link duing this bulk tansfe peiod. Conside now the moe geneal case whee the bottleneck is not the naow link. The mode at MSS in the histogam of the inteaival times of acknowledgments should still pesist, though less ponounced, and it is thus still possible to identify the capacity of the naow link. (We efe the eade to [9] fo a elated wok that takes advantage of the distibution of the inte-aivals of packets tput to identify link capacities.) In this case, does not epesent any moe the shae that the connection obtains at the bottleneck. Howeve, the dispesion scoe can still be seen as an indicato of the distotion (o dispesion) intoduced by the netwok. We validate the method to infe the naow link capacity by setting up an atificial naow link with NIST Net at Institut Euecom, the eceiving end of the path, and tansfeing a lage file with scp to Euecom fom Univesity in Oslo (UiO), Univesity of Navaa (UN), and Helsinki Univesity of Technology (HUT). We pefomed two expeiments with thee paallel tansfes fo two diffeent naow link capacities (2 Mbit/s, 5 Mbit/s) in ode to obseve the effect of coss taffic, and thee expeiments with a single tansfe each to obseve the impact fo dispesion scoe. The esults in table show that the accuacy of the estimated value in these tests is good egadless of the etansmission scoe on the path. We also obseve that the low dispesion scoes coectly eveal that thee is no shaing at the bottleneck. 6. EXPERIMENTAL RESULTS 6. Dataset We applied ou algoithms to a Gbytes tcpdump packet tace quie the use of tools like Pathneck [7] that detects bottlenecks in combination with a tool that measues the capacity of a path [5]. This study is out of the scope of the pesent wok

8 .8.8 bulk tansfe peiods appl. limited peiods F(x).6.4 F(x) bulk tansfe peiods appl. limited peiods duation (s) Figue : Cdfs of duations of the peiods 5 size (bytes) Figue : Cdfs of the size of the peiods in bytes Table : Validation esults fom infeing the capacity of the naow link. sc set estimate tput et scoe RTT UN 2. Mbit/s 2. Mbit/s ms UiO 2. Mbit/s 2. Mbit/s ms HUT 2. Mbit/s 2. Mbit/s ms UN 5. Mbit/s 5. Mbit/s ms UiO 5. Mbit/s 5. Mbit/s ms HUT 5. Mbit/s 5. Mbit/s ms UN 2. Mbit/s 2. Mbit/s ms UiO 2. Mbit/s 2. Mbit/s..4 7 ms HUT 2. Mbit/s 2. Mbit/s ms of BitToent taffic captued at the Univesity of Navaa, Spain. The machine at Navaa was involved in a single toent and the taffic was ecoded once the machine had obtained a full copy of the file and thus only acting as a seve (seed in the BitToent teminology). Hence, all the taffic was captued at the sende side. The tace contains nealy 6, connections with a total amount of 2 million packets. 6.2 Sepaating the Wheat Fom the Chaff Out of the 6, initial connections, we fist filteed out the 57, 8 connections with less than data packets. We then applied ou algoithm to isolate the bulk tansfe peiods and the applications limited peiods to the emaining 2882 connections. We discaded the connections that consisted of a single application limited peiod (ou algoithm discads tansfe peiods of less than 5 seconds, efe to Section 5.). We ended up with only 686 connections (to 43 hosts) consisting of 3295 bulk tansfes and, 365 application limited peiods. Figues and show the cumulative pobability density functions (cdf) of the duations and sizes in bytes of both types of peiods. We obseve that even though BitToent is sending only small potocol messages (e.g. to equest a block o a piece, o to keep connection alive) duing the peiods when it is non active o only downloading data, i.e. the application limited peiods, the duation of those peiods is so lage as compaed to the bulk tansfe peiods (figue ) that eventually the total amount of bytes of some of the application limited peiods can be non negligible as compaed to the amount caied by some of the bulk tansfe peiods (figue ). Indeed, a close look evealed that the ones tansfeing up to 2.5 Mbytes ae seveal hou-long connections sending small packets with push flags at a vey low ate (< 5Kbit/s). Fo the bulk tansfe peiods the coefficient of coelation between the thoughput and the size is.65 and between the thoughput and the duation is.52. Such stong coelations ae the consequence of the BitToent potocol that favos fast tansfes between pees. Hence, the faste the tansfe, the moe likely it is to last, and thus to be lage. 6.3 Receive Window Limitation Figue 2 shows a cdf of the eceive window limitation scoe fo diffeent values of n (see section 5.2.). Clealy, the choice of n is not citical as the shape of the cuve emains pactically the same fo n {, 2, 3}. We use in the following analysis n = 2. We obseve that appoximately 65% of the tansfes ae neve limited by the eceive window and 7% ae limited half of thei life time. Only a small faction of the tansfes ae limited moe than 9% of the time by the eceive window. In Figue 3, the eceive window limited scoe is plotted against the mean value of the eceive advetised window size. The thee most common advetised window values ae distinguishable fom Figue 3 as hoizontal stipes: 8, 6, and 64 Kbytes, an obsevation that agees with [2]. The coefficient of coelation between the limitation scoe and aveage advetised window size is.37 which indicates that it is moe pobable to be eceive window limited when the aveage advetised window value is smalle. Howeve, we note that thee is a significant amount of tansfes with a high limitation scoe and an aveage advetised window of 64 Kbytes, the lagest usable value without window scaling. This obsevation suggests that pehaps, in some cases, a highe thoughput could be obtained by using the window scaling option, though it is not cetain and depends on the amount of available bandwidth on the path. Indeed, using a lage window value might equally lead to congestion and lowe thoughput [3]. We found that all of the 43 client hosts used window scaling o suppoted its usage in ou dataset. Nevetheless, appoximately 93% of the hosts did not scale thei own advetised window, while 6% used a value of 2 and % a value of 7. This agees with the esults in [2] whee 97% of the hosts that suppot window scaling set the value to zeo. On the othe hand, only 26.6% of the hosts in

9 F(x) *MSS theshold *MSS theshold 3*MSS theshold eceive window limitation scoe Figue 2: Cdf of eceive window limitation scoe aveage eceive advetized window eceive window limitation scoe Figue 3: Receive window limitation scoe vs. mean advetised window size F(x) etansmission scoe dispesion scoe scoe Figue 4: Cdfs of netwok limitation scoes aveage thoughput (bytes/s) 4.5 x etansmission scoe Figue 5: Retansmission scoe vs. thoughput thei dataset suppot window scaling, as compaed to % in ou case. 6.4 Netwok Limitation We obseved supisingly elevated levels of netwok limitations in ou dataset as can be seen fom Figue 4. In 2% of the tansfe peiods, at least % of the bytes wee etansmitted. When we plot the etansmission scoe against achieved thoughput in Figue 5, we obseve that the highe the etansmission scoe the lowe the thoughput as stated by the TCP thoughput fomula []. The coefficient of coelation between the etansmission scoe and the thoughput is.2. The cdf of the estimated capacities of the naow link is pesented in Figue 6. The most dominant capacity is aound 2 Mbit/s with 6% of the values (highlighted with a box). The values aound Gbit/s ae eoneously infeed since the capacity of the access link of ou measuement host was less than Gbit/s and may be due to ack compession. Out of the 79 naow links fo which we wee able to infe a capacity we identified only potential non-shaed bottlenecks, i.e. cases whee the dispesion scoe was smalle than.2. As the etansmission scoe was high thoughout the dataset and the infeed naow link capacities faily modest (in moe than 7% of the cases below 2.5 Mbit/s), a possible explanation fo high dispesion scoes (see Figue 4) could be a vey congested high capacity link close to ou measuement host. Figue 7, which plots the etansmission scoe against the dispesion scoe fo bulk tansfe peiods with a eceive window limitation scoe lowe than.5, eveals that thee is a connection between these two scoes. The coefficient of coelation between them was.25. Moe pecise intepetations of this phenomenon would equie moe infomation about the coss-taffic, available bandwidth, and bottleneck locations on the path and is left as futue wok. We looked moe closely at some of the bulk tansfe peiods with zeo etansmission scoe and a high dispesion scoe. We found 3 to 6 seconds-long bulk tansfes whee TCP is in congestion avoidance and expeiences vey long RTTs that pevents it fom gowing its congestion window lage enough to each the limit set by the eceive window o available bandwidth befoe the end of the tansfe. The connection depicted in Figue 8 had an initial RTT of 65ms while duing the tansfe peiod the RTT gew up to

10 F(x) dispesion scoe naow link capacity (Mbit/s) 4 Figue 6: Cdf of infeed naow link capacities. The box highlights the 2Mbit/s tansfes etansmission scoe Figue 7: Retansmission scoe vs. dispesion scoe sequence numbe :8:5 :9: :9: Figue 8: A complete bulk tansfe peiod with a high dispesion scoe and no etansmissions..3s. This suggests that somewhee along the path lage queuing delays wee intoduced causing a netwok limitation that could not be detected by only obseving etansmissions. 6.5 Exclusiveness of the Limitation Causes We want to shed moe light on the dynamics of bulk tansfe peiods that expeience both netwok and eceive window limitation. Figue 9(a) eveals that even though the tend is, as expected, that a high scoe in one test excludes a high scoe in the othe test, thee ae still quite a few tansfe peiods with significant scoes fo both limitations. We had a close look at the thee such tansfe peiods highlighted with a box in Figue 9(a). Two of them exhibited occasional etansmissions altenating with peiods whee the sendes wee eceive window limited as visible in Figue 9(b). Retansmissions ae maked with vetical aows with an R on top. Non-etansmitted data segments often hit the uppe limit of the sliding window. In contast, the thid tansfe peiod was eceive window limited until just befoe the end of the tansfe whee it etansmitted a lage numbe of packets. time 7. CONCLUSIONS AND OUTLOOK In this pape, we have evisited the issue of the oot cause analysis of TCP connections intoduced in [8]. We have fist demonstated the weakness of the flight-based appoach adopted in [8]. We have povided a thoough discussion on the diffeent limitation causes, i.e. the application, the TCP end point paametes and the netwok, emphasizing the need to account fo the impact of the application on the obseved taffic. We then came up with a new analysis method based on vaious time seies extacted fom the heades of a packet tace. Ou technique is obust and allows to pecisely assess the impact of each limitation. A scoe epesenting the limitation level between and is povided fo each of the causes (as opposed to T-RAT [8] that povided only a binay answe, yes o no, to each test). A fist application of the tool on a lage BitToent dataset has demonstated the inteest of the technique. Futue wok includes applying ou tool to publicly available taces that contain data of vaious applications. As explained in Section 5., thee ae cuently seveal paametes that need to be tuned accoding to the application type. We ae woking on a eimplementation of the algoithm fo bulk tansfe identification that equies neithe a time window no the thesholds and could theefoe wok egadless of the time scale of the tansfe and the type of the application. We want to analyze othe applications because we believe that the bulk tansfe applications (P2P file tansfes, FTP, scp, etc.) do not fom a homogeneous class of applications but can geneate many diffeent taffic pattens due to a numbe of factos, e.g. application-level mechanisms, compession, and encyption, which all have an impact on how data is deliveed to TCP and subsequently tansfeed. Anothe challenge in analyzing publicly available taces is that RTT estimation is still difficult in case the measuement point is not at the sende and TCP timestamps ae not available [6]. We ae cuently investigating whethe othe methods fo RTT estimation ae suitable in this case. We would also like to analyze the evolution of the TCP oot causes of bulk tansfe peiods within a connection, and eventually, study in moe depth the tempoal dynamics of the causes and thei inteaction within a bulk tansfe peiod. Acknowledgments This wok has been patly suppoted by the Euopean Union unde the E-NEXT poject FP and by Fance Telecom, poject CRE The authos would like to thank Mikel Izal fom Univesity of

11 eceive window limitation scoe etansmission scoe sequence numbe R 3 6:5:2 R 6:5:25 R 6:5:3 R 6:5:35 R time (a) Retansmission scoe vs. scoe eceive window limitation (b) Close up of a eceive window and netwok limited tansfe peiod. Figue 9: Netwok and eceive window limitation Navaa, Spain fo poviding the BitToent dataset used in ou expeiments. 8. REFERENCES [] S. Baset and H. Schulzinne, An Analysis of the Skype P2P Intenet Telephony Potocol, CUCS-39-4, Depatment of Compute Science, Columbia Univesity, 24. [2] J. Bellado and S. Savage, Measuing packet eodeing, In IMW 2: Poceedings of the 2nd ACM SIGCOMM Wokshop on Intenet measument, pp. 97 5, New Yok, NY, USA, 22, ACM Pess. [3] S. M. Bellovin, A technique fo counting natted hosts, In IMW 2: Poceedings of the 2nd ACM SIGCOMM Wokshop on Intenet measument, pp , New Yok, NY, USA, 22, ACM Pess. [4] M. Cason and D. Santay, NIST Net: a Linux-based netwok emulation tool, Comput. Commun. Rev., 33(3): 26, 23. [5] K. Claffy, R. S. Pasad, M. Muay, and C. Dovolis, Bandwidth Estimation: Metics, Measuement Techniques, and Tools, IEEE Netwok, 7(6):27 35, Novembe 23. [6] M. Dyna, Netwok Tomogaphy Tools, M.S. Thesis, TU Muenchen/Euecom, Septembe 25. [7] N. Hu, L. E. Li, Z. M. Mao, P. Steenkiste, and J. Wang, Locating intenet bottlenecks: algoithms, measuements, and implications, In Poceedings of ACM SIGCOMM 24 Confeence, pp. 4 54, New Yok, NY, USA, 24, ACM Pess. [8] M. Izal, G. Uvoy-Kelle, E. Biesack, P. Felbe, A. Al Hama, and L. Gacés-Eice, Dissecting BitToent: Five Months in a Toent s Lifetime, In Passive and Active Measuements 24, Apil 24. [9] S. Katti, D. Katabi, C. Blake, E. Kohle, and J. Stauss, MultiQ: Automated Detection of Multiple Bottleneck Capacities Along a Path, In Poceedings of Intenet Measuement Confeence (IMC 4), pp , Octobe 24. [] M. Mathis, J. Heffne, and R. Reddy, Web: extended TCP instumentation fo eseach, education and diagnosis, Comput. Commun. Rev., 33(3):69 79, 23. [] M. Mathis, J. Semke, J. Mahdavi, and T. Ott, The Macoscopic Behavio of the TCP Congestion Avoidance Algoithm, Comput. Commun. Rev., 27(3):67 82, July 997. [2] A. Medina, M. Allman, and S. Floyd, Measuing the Evolution of Tanspot Potocols in the Intenet, Comput. Commun. Rev., 35(2):37 52, Apil 25. [3] R. S. Pasad, M. Jain, and C. Dovolis, Socket Buffe Auto-Sizing fo High-Pefomance Data Tansfes, Jounal of Gid Computing, (4):36 376, Decembe 23. [4] S. Shakkottai, R. Sikant, N. Bownlee, A. Boido, and kc claffy, The RTT Distibution of TCP Flows in the Intenet and its Impact on TCPbased Flow Contol,, Coopeative Association fo Intenet Data Analysis (CAIDA), Univesity of Illinois, 24. [5] B. Silveman, Density Estimation fo Statistics and Data Analysis, CRC Pess, 986, ISBN [6] R. Teixeia and J. Rexfod, A measuement famewok fo pin-pointing outing changes, In NetT 4: Poceedings of the ACM SIGCOMM wokshop on Netwok toubleshooting, pp , New Yok, NY, USA, 24, ACM Pess. [7] B. Veal, K. Li, and D. Lowenthal, New Methods fo Passive Estimation of TCP Round-Tip Times, In Poceedings of Passive and Active Measuements(PAM), 25. [8] Y. Zhang, L. Beslau, V. Paxson, and S. Shenke, On the Chaacteistics and Oigins of Intenet Flow Rates, In Poceedings of ACM SIGCOMM 22 Confeence, Pittsbugh, PA, USA, August 22.