FAST TCP: Motivation, Architecture, Algorithms, Performance

Transcription

1 FAST TCP: Motvaton, Archtecture, Algorthms, Performance Cheng Jn Davd X. We Steven H. Low Engneerng & Appled Scence, Caltech Abstract We descrbe FAST TCP, a new TCP congeston control algorthm for hgh-speed long-latency networks, from desgn to mplementaton. We hghlght the approach taken by FAST TCP to address the four dffcultes, at both packet and flow levels, whch the current TCP mplementaton has at large wndows. We descrbe the archtecture and characterze the equlbrum and stablty propertes of FAST TCP. We present expermental results comparng our frst Lnux prototype wth TCP Reno,, and STCP n terms of throughput, farness, stablty, and responsveness. FAST TCP ams to rapdly stablze hgh-speed long-latency networks nto steady, effcent and far operatng ponts, n dynamc sharng envronments, and the prelmnary results are promsng. I. INTRODUCTION Congeston control s a dstrbuted algorthm to share network resources among competng users. It s mportant n stuatons where the avalablty of resources and the set of competng users vary over tme unpredctably, yet effcent sharng s desred. These constrants, unpredctable supply and demand and effcent operaton, necessarly lead to feedback control as the preferred approach, where traffc sources dynamcally adapt ther rates to congeston n ther paths. On the Internet, ths s performed by the Transmsson Control Protocol (TCP) n source and destnaton computers nvolved n data transfers. The congeston control algorthm n the current TCP, whch we refer to as Reno, was developed n 9 [] and has gone through several enhancements snce, e.g., [], [], []. It has performed remarkably well and s generally beleved to have prevented severe congeston as the Internet scaled up by sx orders of magntude n sze, speed, load, and connectvty. It s also well-known, however, that as bandwdth-delay product contnues to grow, TCP Reno wll eventually become a performance bottleneck tself. The followng four dffcultes contrbute to the poor performance of TCP Reno n networks wth large bandwdth-delay products: ) At the packet level, lnear ncrease by one packet per Round-Trp Tme (RTT) s too slow, and multplcatve decrease per loss event s too drastc. ) At the flow level, mantanng large average congeston wndows requres an extremely small equlbrum loss probablty. ) At the packet level, oscllaton s unavodable because TCP uses a bnary congeston sgnal (packet loss). ) At the flow level, the dynamcs s unstable, leadng to severe oscllatons that can only be reduced by the accurate estmaton of packet loss probablty and a stable desgn of the flow dynamcs. We explan these dffcultes n detal n Secton II. In [], we descrbed [] and STCP [7], two loss-based solutons to these problems. In ths paper, we propose a delay-based Appears n the Proceedngs of IEEE Infocom, Hong Kong, March soluton. See [], [9], [], [], [], [], [], [], [], [7], [] for other proposals. In Secton III, we motvate delay-based approach. Delaybased congeston control has been proposed, e.g., n [9], [], []. Its advantage over loss-based approach s small at low speed, but decsve at hgh speed, as we wll argue below. As ponted out n [], delay can be a poor or untmely predctor of packet loss, and therefore usng a delay-based algorthm to augment the basc AIMD (Addtve Increase Multplcatve Decrease) algorthm of TCP Reno s the wrong approach to address the above dffcultes at large wndows. Instead, a new approach that fully explots delay as a congeston measure, augmented wth loss nformaton, s needed. FAST TCP uses ths approach. Usng queueng delay as the congeston measure has two advantages. Frst, queueng delay can be more accurately estmated than loss probablty both because packet losses n networks wth large bandwdth-delay product are rare events (probablty on the order or smaller), and because loss samples provde coarser nformaton than queueng delay samples. Indeed, measurements of delay are nosy, just as those of loss probablty. Each measurement of packet loss (whether a packet s lost) provdes one bt of nformaton for the flterng of nose, whereas each measurement of queueng delay provdes mult-bt nformaton. Ths makes t easer for an equaton-based mplementaton to stablze a network nto a steady state wth a target farness and hgh utlzaton. Second, the dynamcs of queueng delay seems to have the rght scalng wth respect to network capacty. Ths helps mantan stablty as a network scales up n capacty [], [], []. In Secton III, we explan how we explot these advantages to address the four dffcultes of TCP Reno. In Secton IV, we lay out an archtecture to mplement our desgn. Even though the dscusson s n the context of FAST TCP, the archtecture can also serve as a general framework to gude the desgn of other congeston control mechansms, not necessarly lmted to TCP, for hgh-speed networks. The man components n the archtecture can be desgned separately and upgraded asynchronously. Unlke the conventonal desgn, FAST TCP can use the same wndow and burstness control algorthms regardless of whether a source s n the normal state or the loss recovery state. Ths leads to a clean separaton of components n both functonalty and code structure. We then present an overvew of some of the algorthms mplemented n our current prototype. In Secton V, we present a mathematcal model of the wndow control algorthm. We prove that FAST TCP has the same equlbrum propertes as TCP Vegas [], []. In partcular, t does not penalze flows wth large propagaton delays, and t acheves weghted proportonal farness [7]. For the specal case of sngle bottleneck lnk wth heterogeneous flows, we prove that the wndow control algorthm of FAST s globally stable, n the absence of

2 feedback delay. Moreover, startng from any ntal state, a network converges exponentally to a unque equlbrum. In Secton VI, we present prelmnary expermental results to llustrate throughput, farness, stablty, and responsveness of FAST TCP, n the presence of delay and n heterogeneous and dynamc envronments where flows of dfferent delays jon and depart asynchronously. We compare the performance of FAST TCP wth Reno, (HghSpeed TCP []), and STCP (Scalable TCP [7]), usng ther default parameters. In these experments, FAST TCP acheved the best performance under each crteron, whle and STCP mproved throughput and responsveness over Reno at the cost of farness and stablty. We conclude n Secton VII. II. PROBLEMS AT LARGE WINDOWS A congeston control algorthm can be desgned at two levels. The flow-level (macroscopc) desgn ams to acheve hgh utlzaton, low queueng delay and loss, farness, and stablty. The packet-level desgn mplements these flow-level goals wthn the constrants mposed by end-to-end control. Hstorcally for TCP Reno, packet-level mplementaton was ntroduced frst. The resultng flow-level propertes, such as farness, stablty, and the relatonshp between equlbrum wndow and loss probablty, were then understood as an afterthought. In contrast, the packet-level desgns of [], STCP [7], and FAST TCP are explctly guded by flowlevel goals. We elaborate n ths ton on the four dffcultes of TCP Reno lsted n Secton I. It s mportant to dstngush between packet-level and flow-level dffcultes because they must be addressed by dfferent means. A. Packet and flow level modelng The congeston avodance algorthm of TCP Reno and ts varants have the form of AIMD []. The pseudo code for wndow adjustment s: Ack: w ψ w + w Loss: w ψ w w Ths s a packet-level model, but t nduces certan flow-level propertes such as throughput, farness, and stablty. These propertes can be understood wth a flow-level model of the AIMD algorthm, e.g., [], [9], []. The wndow w (t) of source ncreases by packet per RTT, and decreases per unt tme by where x (t)p (t) w (t) packets x (t) := w (t)=t (t) pkts/ T (t) s the round-trp tme, and p (t) s the (delayed) end-toend loss probablty, n perod t. Here, w (t)= s the peak wndow sze that gves the average wndow of w (t). Hence, a flow-level model of AIMD s: _w (t) = T (t) x (t)p (t)w (t) () It should be ( p (t)) packets, where p (t) s the end-to-end loss probablty. Ths s roughly when p (t) s small. Ths model assumes that wndow s halved on each packet loss. It can be modfed to model the case, where wndow s halved at most once n each RTT. Ths does not qualtatvely change the followng dscusson. Settng _w (t) =n () yelds the well-known = p p formula for TCP Reno dscovered n [], [], whch relates loss probablty to wndow sze n equlbrum: p Λ = () w Λ In summary, () and () descrbe the flow-level dynamcs and the equlbrum, respectvely, for TCP Reno. It turns out that dfferent varants of TCP all have the same dynamc structure at the flow level (see [], []). By defnng» (w ;T ) = T and u (w ;T ) = : and notng that w = x T, we can express () as: _w (t) =»(t) p (t) () u (t) where we have used the shorthand» (t) =» (w (t);t (t)) and u (t) = u (w (t);t (t)). Equaton can be used to descrbe all known TCP varants, and dfferent varants dffer n ther choces of the gan functon» and margnal utlty functon u, and whether the congeston measure p s loss probablty or queueng delay. Next, we llustrate the equlbrum and dynamcs problems of TCP Reno, at both the packet and flow levels, as bandwdthdelay product ncreases. B. Equlbrum problem The equlbrum problem at the flow level s expressed n (): the end-to-end loss probablty must be exceedngly small to sustan a large wndow sze, makng the equlbrum dffcult to mantan n practce, as bandwdth-delay product ncreases. Even though equlbrum s a flow-level noton, ths problem manfests tself at the packet level, where a source ncrements ts wndow too slowly and decrements t too drastcally. When the peak wndow s,-packet (correspondng to an average wndow of, packets), whch s necessary to sustan 7.Gbps usng,-byte packets wth a RTT of ms, t takes, RTTs, or almost 7 mnutes, to recover from a sngle packet loss. Ths s llustrated n Fgure a, where the sze of wndow ncrement per RTT and decrement per loss, and :w, respectvely, are plotted as functons of w. The ncrement functon for Reno (and for ) s almost ndstngushable from the x-axs. Moreover, the gap between the ncrement and decrement functons grows rapdly as w ncreases. Snce the average ncrement and decrement must be equal n equlbrum, the requred loss probablty can be exceedngly small at large w. Ths pcture s thus smply a vsualzaton of (). To address the dffcultes of TCP Reno at large wndow szes, and STCP ncrease more aggressvely and decrease more gently, as dscussed n [], []. C. Dynamc problems The causes of the oscllatory behavor of TCP Reno le n ts desgn at both the packet and flow levels. At the packet level, the choce of bnary congeston sgnal necessarly leads to oscllaton, and the parameter settng n Reno worsens the stuaton as bandwdth-delay product ncreases. At the flow level, the system dynamcs gven by () s unstable at large bandwdth-delay products [9], []. These must be addressed by dfferent means, as we now elaborate. w

3 O nc per RTT O dec per loss w* wndow adjustment (pkts) 7 Reno S TCP wndow adjustment (pkts) w = w* 9 7 wndow (pkts) x dstance from equlbrum p /u (a) Reno,, and STCP (b) FAST Fg.. Packet-level mplementaton: (a) Wndow ncrement per RTT and decrement per loss, as functons of the current wndow. The ncrement functons for TCP Reno and are almost dentcal at ths scale. (b) Wndow update as a functon of dstance from equlbrum for FAST. loss loss Queue Delay C delay D Wndow Queue Delay delay F T Wndow R R (a) Bnary sgnal: oscllatory (b) Mult-bt sgnal: stablzable Fg.. Operatng ponts of TCP algorthms: R: Reno [], [], STCP [7]; D: DUAL []; C: CARD [9]; T : TFRC []; F : Vegas [], FAST. Fgure (a) llustrates the operatng ponts chosen by varous TCP congeston control algorthms, usng the sngle-lnk around a target value gven by (). Its operatng pont s on the estmated loss probablty n an attempt to stablze sngle-flow scenaro. It shows queueng delay as a functon of T n Fgure (b), near the overflowng pont. Ths approach wndow sze. Queueng delay starts to buld up after pont C elmnates the oscllaton due to packet-level AIMD, but two where wndow equals bandwdth-propagaton-delay product, dffcultes reman at the flow level. untl pont R where the queue overflows. Snce Reno oscllates Frst, equaton-based control requres the explct estmaton around pont R, the peak wndow sze goes beyond pont of end-to-end loss probablty. Ths s dffcult when the loss R. The mnmum wndow n steady state s half of the peak probablty s small. Second, even f loss probablty can wndow. Ths s the bass for the rule of thumb that bottleneck be perfectly estmated, Reno s flow dynamcs, descrbed by buffer should be at least one bandwdth-delay product: the equaton () leads to a feedback system that becomes unstable mnmum wndow wll then be above pont C, and buffer wll as feedback delay ncreases, and more strkngly, as network not empty n steady state operaton, yeldng full utlzaton. capacty ncreases [9], []. The nstablty at the flow level In the loss-based approach, full utlzaton, even f achevable, comes at the cost of severe oscllatons and potentally stablzng the flow-level dynamcs. We wll return to both can lead to severe oscllatons that can be reduced only by large queueng delay. The DUAL scheme n [] proposes ponts n Secton III. to oscllate around pont D, the mdpont between C and R when the buffer s half-full. DUAL ncreases congeston III. DELAY-BASED APPROACH wndow lnearly by one packet per RTT, as long as queueng In ths ton, we motvate delay-based approach to address delay s less than half of the maxmum value, and decreases the four dffcultes at large wndow szes. multplcatvely by a factor of /, when queueng delay exceeds half of the maxmum value. The scheme CARD (Congeston Avodance usng Round-trp Delay) of [9] proposes A. Motvaton to oscllate around pont C through AIMD wth the same Although mproved loss-based protocols such as parameter (; =) as DUAL, based on the rato of roundtrp delay and delay gradent, to maxmze power. In all these we showed n [] that they don t address all four problems and STCP have been proposed as replacements to TCP Reno, schemes, the congeston sgnal s used as a bnary sgnal, and (Secton I) of TCP Reno. To llustrate ths, we plot the hence congeston wndow must oscllate. ncrement and decrement functons of and STCP n Congeston wndow can be stablzed only f mult-bt feedback s used. Ths s the approach taken by the equaton-based TCP Reno: each ncreases more aggressvely and decreases Fgure (a) alongsde TCP Reno. Both protocols upper bound algorthm n [], where congeston wndow s adjusted based less drastcally, so that the gap between the ncrement and

4 decrement functons s narrowed. Ths means, n equlbrum, both and STCP can tolerate larger loss probabltes than TCP Reno, thus achevng larger equlbrum wndows. However, nether solves the dynamcs problems at both the packet and the flow levels. In [], we show that the congeston wndows n Reno, and STCP all evolve accordng to: _w (t) =» (t) p (t) () u (t) where»(t) :=» (w (t);t (t)) and u (t) :=u (w (t);t (t)). Moreover, the dynamcs of FAST TCP also takes the same form; see below. They dffer only n the choce of the gan functon» (w ;T ), the margnal utlty functon u (w ;T ), and the end-to-end congeston measure p. Hence, at the flow level, there are only three desgn decsons: ffl» (w ;T ): the choce of the gan functon» determnes the dynamc propertes such as stablty and responsveness, but does not affect the equlbrum propertes. ffl u (w ;T ): the choce of the margnal utlty functon u manly determnes equlbrum propertes such as the ffl equlbrum rate allocaton and ts farness. p : n the absence of explct feedback, the choce of congeston measure p s lmted to loss probablty or queueng delay. The dynamcs of p (t) s determned at lnks. The desgn choces n Reno,, STCP and FAST are shown n Table I.» (w ;T ) u (w ;T ) p Reno =T :=w loss probablty :b(w )w : :=w : ( b(w ))T loss probablty STCP aw =T ρ=w loss probablty FAST flff ff =x queueng delay TABLE I COMMON DYNAMIC STRUCTURE: w IS SOURCE S WINDOW SIZE, T IS ITS ROUND-TRIP TIME, p IS CONGESTION MEASURE, x = w =T ; a; b(w );ρ;fl;ff ARE PROTOCOL PARAMETERS. These choces produce equlbrum characterzatons shown n Table II. Reno x = T x = T ff p : ff p : STCP x = ff T p FAST x = ff p TABLE II COMMON EQUILIBRIUM STRUCTURE. Ths common model () can be nterpreted as follows: the goal at the flow level s to equalze margnal utlty u (t) wth the end-to-end measure of congeston, p (t). Ths nterpretaton mmedately suggests an equaton-based packetlevel mplementaton where both the drecton and sze of the wndow adjustment _w (t) are based on the dfference between the rato p (t)=u (t) and the target of. Unlke the approach taken by Reno,, and STCP, ths approach elmnates packet-level oscllatons due to the bnary nature of congeston sgnal. It however requres the explct estmaton of the endto-end congeston measure p (t). Wthout explct feedback, p (t) can only be loss probablty, as used n TFRC [], or queueng delay, as used n TCP Vegas [] and FAST TCP. Queueng delay can be more accurately It s debatable whether TCP Vegas s equaton-based snce the sze of ts wndow adjustment does not depend on queueng delay. Ths s not mportant at low speed but crtcal at hgh speed. estmated than loss probablty both because packet losses n networks wth large bandwdth-delay products are rare events (probablty on the order or smaller), and because loss samples provde coarser nformaton than queueng delay samples. Indeed, each measurement of packet loss (whether a packet s lost) provdes one bt of nformaton for the flterng of nose, whereas each measurement of queueng delay provdes mult-bt nformaton. Ths allows an equatonbased mplementaton to stablze a network nto a steady state wth a target farness and hgh utlzaton. At the flow level, the dynamcs of the feedback system must be stable n the presence of delay, as the network capacty ncreases. Here, agan, queueng delay has an advantage over loss probablty as a congeston measure: the dynamcs of queueng delay seems to have the rght scalng wth respect to network capacty. Ths helps mantan stablty as network capacty grows [], [], []. B. Implementaton strategy The delay-based approach, wth proper flow and packet level desgns, can address the four dffcultes of Reno at large wndows. Frst, by explctly estmatng how far the current state p (t)=u (t) s from the equlbrum value of, our scheme can drve the system rapdly, yet n a far and stable manner, toward the equlbrum. The wndow adjustment s small when the current state s close to equlbrum and large otherwse, ndependent of where the equlbrum s, as llustrated n Fgure (b). Ths s n stark contrast to the approach taken by Reno,, and STCP, where wndow adjustment depends on just the current wndow sze and s ndependent of where the current state s wth respect to the target (compare Fgures (a) and (b)). Lke the equaton-based scheme n [], ths approach avods the problem of slow ncrease and drastc decrease n Reno, as the network scales up. Second, by choosng a mult-bt congeston measure, ths approach elmnates the packet-level oscllaton due to bnary feedback, avodng Reno s thrd problem. Thrd, usng queueng delay as the congeston measure p (t) allows the network to stablze n the regon below the overflowng pont, around pont F n Fgure (b), when the buffer sze s suffcently large. Stablzaton at ths operatng pont elmnates large queueng delay and unnecessary packet loss. More mportantly, t makes room for bufferng mce traffc. To avod the ond problem n Reno, where the requred equlbrum congeston measure (loss probablty for Reno, and queueng delay here) s too small to practcally estmate, the algorthm must adapt ts parameter ff wth capacty to mantan small but suffcent queueng delay. Fnally, to avod the fourth problem of Reno, the wndow control algorthm must be stable, n addton to beng far and effcent, at the flow level. The use of queueng delay as a congeston measure facltates the desgn as queueng delay naturally scales wth capacty [], [], []. The desgn of TCP congeston control algorthm can thus be conceptually dvded nto two levels: ffl ffl At the flow level, the goal s to desgn a class of functon pars, u (w ;T ) and»(w ;T ), so that the feedback system descrbed by (), together wth lnk dynamcs n p (t) and the nterconnecton, has an equlbrum that s far and effcent, and that the equlbrum s stable, n the presence of feedback delay. At the packet level, the desgn must deal wth ssues that are gnored by the flow-level model or modelng assumptons that are volated n practce, n order to acheve these

5 flow-level goals. These ssues nclude burstness control, loss recovery, and parameter estmaton. The mplementaton then proceeds n three steps: ) determne varous system components; ) translate the flow-level desgn nto packet-level algorthms; ) mplement the packet-level algorthms n a specfc operatng system. The actual process terates ntmately between flow and packet level desgns, between theory, mplementaton, and experments, and among the three mplementaton steps. The emergng theory of large-scale networks under end-toend control, e.g., [7], [], [], [], [7], [], [], [9], [], [], [], [9], [], [], [], [], [], [] (see also, e.g., [], [], [] for recent surveys), forms the foundaton of the flow-level desgn. The theory plays an mportant role by provdng a framework to understand ssues, clarfy deas, and suggest drectons, leadng to a robust and hgh performance mplementaton. We lay out the archtecture of FAST TCP next. IV. ARCHITECTURE AND ALGORITHMS We separate the congeston control mechansm of TCP nto four components n Fgure. These four components are functonally ndependent so that they can be desgned separately and upgraded asynchronously. In ths ton, we focus on the two parts that we have mplemented n the current prototype (see []). Data Control Wndow Control Estmaton Burstness Control TCP Protocol Processng Fg.. FAST TCP archtecture. The data control component determnes whch packets to transmt, wndow control determnes how many packets to transmt, and burstness control determnes when to transmt these packets. These decsons are made based on nformaton provded by the estmaton component. Wndow control regulates packet transmsson at the RTT tmescale, whle burstness control works at a smaller tmescale. In the followng subtons, we provde an overvew of estmaton and wndow control and the algorthms mplemented n our current prototype. An ntal prototype that ncluded the features dscussed here was demonstrated n November at the SuperComputng Conference, and the expermental results were reported n [7]. A. Estmaton Ths component provdes estmatons of varous nput parameters to the other three decson-makng components. It computes two peces of feedback nformaton for each data packet sent. When a postve acknowledgment s receved, t calculates the RTT for the correspondng data packet and updates the average queueng delay and the mnmum RTT. When a negatve acknowledgment (sgnaled by three duplcate acknowledgments or tmeout) s receved, t generates a loss ndcaton for ths data packet to the other components. The estmaton component generates both a mult-bt queueng delay sample and a one-bt loss-or-no-loss sample for each data packet. The queueng delay s smoothed by takng a movng average wth the weght (t) := mnf=w (t); =g that depends on the wndow w (t) at tme t, as follows. The k-th RTT sample T (k) updates the average RTT T (k) accordng to: T (k +) = ( (t k ))T (k) + (t k )T (k) where t k s the tme at whch the k-th RTT sample s receved. Takng d (k) to be the mnmum RTT observed so far, the average queueng delay s estmated as: ^q (k) = T (k) d (k) The weght (t) s usually much smaller than the weght (/) used n TCP Reno. The average RTT T (k) attempts to track the average over one congeston wndow. Durng each RTT, an entre wndow worth of RTT samples are receved f every packet s acknowledged. Otherwse, f delayed ack s used, the number of queueng delay samples s reduced so (t) should be adjusted accordngly. B. Wndow control The wndow control component determnes congeston wndow based on congeston nformaton queueng delay and packet loss, provded by the estmaton component. A key decson n our desgn that departs from tradtonal TCP desgn s that the same algorthm s used for congeston wndow computaton ndependent of the state of the sender. For example, n TCP Reno (wthout rate halvng), congeston wndow s ncreased by one packet every RTT when there s no loss, and ncreased by one for each duplcate ack durng loss recovery. In FAST TCP, we would lke to use the same algorthm for wndow computaton regardless of the sender state. Our congeston control mechansm reacts to both queueng delay and packet loss. Under normal network condtons, FAST perodcally updates the congeston wndow based on the average RTT and average queueng delay provded by the estmaton component, accordng to: w ψ mn n basertt w; ( fl)w + fl w + ff(w, qdelay) RTT o where fl (; ], basertt s the mnmum RTT observed so far, and qdelay s the end-to-end (average) queueng delay. In our current mplementaton, congeston wndow changes over two RTTs: t s updated n one RTT and frozen n the next. The update s spread out over the frst RTT n a way such that congeston wndow s no more than doubled n each RTT. In our current prototype, we choose the functon ff(w, qdelay) to be a constant at all tmes. Ths produces lnear convergence when the qdelay s zero. Alternatvely, we can use a constant ff only when qdelay s nonzero and an ff proportonal to wndow, ff(w, qdelay) = aw, when qdelay s zero. In ths case, when qdelay s zero, FAST performs multplcatve ncrease and grows exponentally at rate a to a neghborhood of qdelay >. Then ff(w, qdelay) swtches to a constant ff and, as we wll see n Theorem below, wndow converges exponentally to the equlbrum at a dfferent rate that depends on qdelay. The constant ff s the number of packets each flow attempts to mantan n the network buffer(s) at equlbrum, smlar to TCP Vegas []. All experments n Secton VI used lnear ncrease,.e., ff(w, qdelay) =ff for all qdelay. ()

6 Fg.. Flow-Level Algorthm x=f(p) p=g(x,p) Component Desgn Packet-Level Algorthm per Ack per RTT per Loss From flow-level desgn to mplementaton. OS-Specfc Implementaton functon functon... Although we would lke to use the same congeston control functon durng loss recovery, we have currently dsabled ths feature because of ambgutes assocated wth retransmtted packets. Currently when a packet loss s detected, FAST halves ts wndow and enters loss recovery. The goal s to back off packet transmsson quckly when severe congeston occurs, n order to brng the system back to a regme where relable RTT measurements are agan avalable for wndow adjustment () to work effectvely. A source does not react to delay untl t exts loss recovery. C. Packet-level mplementaton It s mportant to mantan an abstracton of the mplementaton as as the code evolves. Ths abstracton should descrbe the hgh-level operatons each component performs based on external nputs, and can serve as a road map for future TCP mplementatons as well as mprovements to the exstng mplementaton. Whenever a non-trval change s requred, one should frst update ths abstracton to ensure that the overall packet-level code would be bult on a sound underlyng foundaton. Snce TCP s an event-based protocol, our control actons should be trggered by the occurrence of varous events. Hence, we need to translate our flow-level algorthms nto event-based packet-level algorthms. There are four types of events that FAST TCP reacts to: on the recepton of an acknowledgment, after the transmsson of a packet, at the end of a RTT, and for each packet loss. For each acknowledgment receved, the estmaton component computes the average queueng delay, and the burstness control component determnes whether packets can be njected nto the network. For each packet transmtted, the estmaton component records a tme-stamp, and the burstness control component updates correspondng data structures for bookkeepng. At a constant tme nterval, whch we check on the arrval of each acknowledgment, wndow control calculates a new wndow sze. At the end of each RTT, burstness reducton calculates the target throughput usng the wndow and RTT measurements n the last RTT. Wndow pacng wll then schedule to break up a large ncrement n congeston wndow nto smaller ncrements over tme. Durng loss recovery, congeston wndow should be contnually updated based on congeston sgnals from the network. Upon the detecton of a packet loss event, a sender determnes whether to retransmt each unacknowledged packet rght away or hold off untl a more approprate tme. Fgure presents an approach to turn the hgh-level desgn of a congeston control algorthm nto an mplementaton. Frst, an algorthm s desgned at the flow-level and analyzed to ensure that t meets the hgh-level objectves such as farness and stablty. Based on that, one can determne the components necessary to mplement congeston control. The In the Lnux TCP mplementaton, congeston wndow was frequently reduced to one when there were heavy losses. In order to ensure a reasonable recovery tme, we mpose a mnmum wndow of packets durng loss recovery for connectons that use large wndows. Ths and other nterm measures wll be mproved n future FAST TCP releases. flow-level algorthm can then be translated nto a packet-level algorthm that conssts of a set of event-based tasks. The eventbased tasks should be ndependent of any specfc TCP or operatng system mplementaton, but yet detaled enough so the understandng of these tasks enables one to mplement FAST n any operatng system or protocol stack. V. EQUILIBRIUM AND STABILITY OF WINDOW CONTROL ALGORITHM In ths ton, we present a model of the wndow control algorthm. We show that, n equlbrum, the vectors of source wndows and lnk queueng delays are the unque solutons of a par of optmzaton problems (9) (). Ths completely characterzes the network equlbrum propertes such as throughput, farness, and delay. We also analyze the stablty of the wndow control algorthm. We prove n [] that, for a sngle lnk wth heterogeneous sources, the wndow control algorthm () s globally stable, assumng zero feedback delay, and converges exponentally to a unque equlbrum. Extensve experments n Secton VI llustrate ts stablty n the presence of feedback delay. Gven a network that conssts of a set of resources wth fnte capactes c l, e.g., transmsson lnks, processng unts, memory, etc., we refer to them n general as lnks n our model. The network s shared by a set of uncast flows, dentfed by ther sources. Let d denote the round-trp propagaton delay of source. Let R be the routng matrx where R l =f source uses lnk l, and otherwse. Let p P l (t) denote the queueng delay at lnk l at tme t. Let q (t) = l R lp l (t) be the roundtrp queueng delay, or n vector notaton, q(t) =R T p(t). Then the round trp tme of source s T (t) :=d + q (t). Each source adapts w (t) perodcally accordng to: dw (t) w (t +) = fl d + q + ff (w (t);q (t)) (t) + ( fl)w (t) () where fl (; ], at tme t, and ff (w ;q ) s defned by: ρ a w ff (w ;q f q = ) = ff otherwse (7) A key departure of our model from those n the lterature s that we assume that a source s send rate, defned as x (t) :=w (t)=t (t), cannot exceed the throughput t receves. Ths s justfed because of self-clockng: one round-trp tme after a congeston wndow s ncreased, packet transmsson wll be clocked at the same rate as the throughput the flow receves. See [] for detaled justfcaton and valdaton experments. A consequence of ths assumpton s that the lnk queueng delay vector, p(t), s determned mplctly by the nstantaneous wndow sze n a statc manner: gven w (t) =w for all, the lnk queueng delays p l (t) =p l for all l are gven by: X w R l d + q where agan q = P l R lp l. ρ = c l f p l >» c l f p l = Note that () can be rewrtten as (when ff (w ;q )=ff, constant) w (t +) = w (t) +fl (ff x (t)q (t)) From [], TCP Vegas updates ts wndow accordng to w (t +) = w (t) + T sgn(ff x (t)q (t)) (t) where sgn(z) = f z<, fz =, and f z>. Hence FAST can be thought of as a hgh-speed verson of Vegas. ()

7 7 The equlbrum values of wndows w Λ and delays p Λ of the network defned by () () can be characterzed as follows. Consder the utlty maxmzaton problem max x X and the followng (dual) problem: X ff log x s.t. Rx» c (9) X X mn c l p l ff log R l p l () p l l Theorem : Suppose R has full row rank. The unque equlbrum pont (w Λ ;p Λ ) of the network defned by () () exsts and s such that x Λ =(x Λ := w =(d + q Λ ); ) s the unque maxmzer of (9) and p Λ s the unque mnmzer of (). Ths mples n partcular that the equlbrum rate x Λ s ff -weghted proportonally far. Theorem mples that FAST TCP has the same equlbrum propertes as TCP Vegas [], []. Its throughput s gven by x = ff () q In partcular, t does not penalze sources wth large propagaton delays d. The relaton () also mples that, n equlbrum, source mantans ff packets n the buffers along ts path [], []. Hence, P the total amount of bufferng n the network must be at least ff packets n order to reach the equlbrum. We now turn to the stablty of the algorthm. Global stablty n a general network n the presence of feedback delay s an open problem (see [9], [] for stablty analyss for the sngle-lnk-sngle-source case). State-of-the-art results ether prove global stablty whle gnorng feedback delay, or local stablty n the presence of feedback delay. Our stablty result s restrcted to a sngle lnk n the absence of delay. Theorem : Suppose there s a sngle lnk wth capacty c. Then the network defned by () () s globally stable, and converges geometrcally to the unque equlbrum (w Λ ;p Λ ). The basc dea of the proof s to show that the teraton from w(t) to w(t +) defned by () () s a contracton mappng. Hence w(t) converges geometrcally to the unque equlbrum. Some propertes follow from the proof of Theorem. Corollary : ) Startng from any ntal pont (w();p()), the lnk s fully utlzed,.e., equalty holds n (), after a fnte tme. ) The queue length s lower and upper bounded after a fnte amount of tme. VI. PERFORMANCE We have conducted some prelmnary experments on our dummynet [] testbed comparng performance of varous new TCP algorthms as well as the Lnux TCP mplementaton. It s mportant to evaluate them not only n statc envronments, but also dynamc envronments where flows come and go; and not only n terms of end-to-end throughput, but also queue behavor n the network. In ths study, we compare performance among TCP connectons of the same protocol sharng a sngle bottleneck lnk. In summary, ) FAST TCP acheved the best overall performance n each of the four evaluaton crtera: throughput, farness, responsveness, and stablty. ) Both and STCP mproved throughput and responsveness of Lnux TCP, although both showed farness problems and oscllatons wth hgher frequences and larger magntudes. In the followng subtons, we wll descrbe n detal our expermental setup, evaluaton crtera, and results. A. Testbed and kernel nstrumentaton We constructed a testbed of a sender and a recever both runnng Lnux, and an emulated router runnng FreeBSD. Each testbed machne has dual Xeon. GHz, GB of man memory, and dual on-board Intel PRO/ Ggabt Ethernet nterfaces. We have tested these machnes to ensure each s able to acheve a peak throughput of 9 Mbps wth the standard Lnux TCP protocol usng perf. sender montor perf Sender ms ms ms ms queue montor Router perf Recever Fg.. Testbed and the expermental setup. Fgure shows the setup of the testbed. The testbed router supports paths of varous delays and a sngle bottleneck capacty wth a fxed buffer sze. It has montorng capablty at the sender and the router. The recever runs dfferent TCP traffc snks wth dfferent port numbers for connectons wth dfferent RTTs. We set up and run dfferent experments from the sender usng an automatc scrpt generator to start multple perf sessons to emulate multple TCP connectons. Our testbed router ran dummynet [] under FreeBSD. We confgured dummynet to create paths or ppes of dfferent delays,,,, and ms, usng dfferent destnaton port numbers on the recevng machne. We then created another ppe to emulate a bottleneck capacty of Mbps and a buffer sze of, packets, shared by all the delay ppes. Due to our need to emulate a hgh-speed bottleneck capacty, we ncreased the schedulng granularty of dummynet events. We recompled the FreeBSD kernel so the task scheduler ran every ms. We also ncreased the sze of the IP layer nterrupt queue (pntrq) to to accommodate large bursts of packets. We nstrumented both the sender and the dummynet router to capture relevant nformaton for protocol evaluaton. For each connecton on the sendng machne, the kernel montor captured the congeston wndow, the observed basertt, and the observed queueng delay. On the dummynet router, the kernel montor captured the throughput at the dummynet bottleneck, the number of lost packets, and the average queue sze every two onds. We retreved the measurement data after the completon of each experment n order to avod dsk I/O that may have nterfered wth the experment tself. We tested four TCP mplementatons: FAST,, STCP, and Reno (Lnux mplementaton). The FAST TCP s based on Lnux.. kernel, whle the rest of the TCP protocols are based on Lnux..9 kernel. We ran tests and dd not observe any apprecable dfference between the two plan Lnux kernels, and the TCP source codes of the two kernels are nearly dentcal. Lnux TCP mplementaton ncludes all of the latest RFCs such as New Reno, SACK, D-SACK, and TCP hgh performance extensons. There are two versons of [], []. We present the results of the mplementaton n [], but our tests show that the mplementaton n [] has comparable performance. In all of our experments, the bottleneck capacty s Mbps roughly packets/ms, and the maxmum buffer sze s packets.

8 FAST Scalable TCP 7 7 x propagaton delay(ms) x Fg.. propagaton delay(ms) FAST STCP flow number flow number HSCTP 7 7 x propagaton delay(ms) x propagaton delay(ms) Statc scenaro: aggregate throughput. Reno We now present our expermental results. We frst look at three cases n detal, comparng not only the throughput behavor seen at the source, but also the queue behavor nsde the network, by examnng trajectores of throughputs, wndows, nstantaneous queue, cumulatve losses, and lnk utlzaton. We then summarze the overall performance n a dverse set of experments n terms of quanttatve metrcs, defned below, on throughput, farness, stablty, and responsveness. B. Case study: statc scenaro We present expermental results on aggregate throughput n a smple statc envronment where, n each experment, all TCP flows had the same propagaton delay and started and termnated at the same tmes. Ths set of tests ncluded experments for dfferent parng of propagaton delays,,,, and ms, and the number of dentcal sources,,,,, and. We ran ths test sute under each of the four TCP protocols. We then constructed a -d plot, n Fgure, for each protocol wth the x-axs and y-axs beng the number of sources and propagaton delay, respectvely. The z-axs s the aggregate throughput. All four protocols performed well when the number of flows was large or the propagaton delay was small,.e., when the wndow sze was small. The performance of FAST TCP remaned consstent when these parameters changed. TCP Reno,, and STCP had varyng degrees of performance degradaton as the wndow sze ncreased, wth TCP Reno showng the most sgnfcant degradaton. Ths set of experments, nvolvng a statc envronment and dentcal flows, does not test the farness, stablty and responsveness of the protocols. We take a close look at these propertes next n a dynamc scenaro where network equlbrum changes as flows come and go. C. Case study: dynamc scenaro I In the frst dynamc test, the number of flows was small so that throughput per flow, and hence the wndow sze, was large. There were three TCP flows, wth propagaton delays of,, and ms, that started and termnated at dfferent tmes, as llustrated n Fgures 7(a). For each dynamc experment, we generated two sets of fgures, from the sender montor and the queue montor. From the sender montor, we obtaned the trajectores of ndvdual connecton throughput (n Kbps) and wndow sze (n packets) over tme. They are shown n Fgure. Lnux..9 TCP flow number flow number As new flows joned or old flows left, FAST TCP converged to the new equlbrum rate allocaton rapdly and stably. Reno s throughput was also relatvely smooth because of the slow (lnear) ncrease before packet losses. The lnk utlzaton was low at the end of the experment when t took mnutes for a flow to reclam the spare capacty due to the departure of another flow. and STCP, n an attempt to respond more quckly, went nto severe oscllaton. From the queue montor, we obtaned three trajectores: the average queue sze (packets), the number of cumulatve packet losses (packets), and the utlzaton of the bottleneck lnk (n packets/ms), shown n Fgure 9 from top to bottom. The queue under FAST TCP was qute small throughout the experment due to the small number of flows. and STCP exhbted strong oscllatons that flled the buffer. The lnk utlzatons of FAST TCP and Reno were qute steady, whereas those of and STCP showed fluctuatons. From the throughput trajectores of each protocol, we calculate Jan s farness ndex (see Secton VI-E for defnton) for the rate allocatons for each tme nterval that contans more than one flow (see Fgure 7(a)). The farness ndces are shown n Table III. FAST TCP obtaned the best farness, Tme #Sources FAST STCP Reno TABLE III DYNAMIC SCENARIO I: INTRA-PROTOCOL FAIRNESS. very close to, followed by, Reno, and then STCP. It confrms that FAST TCP does not penalze aganst flows wth large propagaton delays. Even though, STCP, and Reno all try to equalze congeston wndows among competng connectons nstead of equalzng rates, ths was not acheved as shown n Fgure. The unfarness s especally severe n the case of STCP, lkely due to MIMD as explaned n []. For FAST TCP, each source tres to mantan the same number of packets n the queue n equlbrum, and thus, n theory, each competng source should get an equal share of the bottleneck bandwdth. Even though FAST TCP acheved the best farness ndex, we dd not observe the expected equal sharng of bandwdth (see Fgure ). We found that connectons wth longer RTTs consstently observed hgher queueng delays than those wth shorter RTTs. For example, the connecton on the path of ms saw an average queueng delay of ms, whle the connecton on the path of ms saw an average queueng delay of 9 ms. Ths caused the connecton wth longer RTTs to mantan fewer packets n the queue n equlbrum, thus gettng a smaller share of the bandwdth. We have yet to uncover the source of ths problem, but the early conjecture s that when congeston wndow sze s large, t s much harder to break up bursts of packets. Wth bursty traffc arrvng at a queue, each packet would see a delay that ncludes the transmsson tmes of all precedng packets n the burst. However, f packets were spaced out smoothly, then each packet would have seen a smaller queueng delay at the queue. D. Case study: dynamc scenaro II Ths experment was smlar to dynamc scenaro I, except that there were a larger number () of flows, wth dfferent propagaton delays, whch joned and departed accordng to the schedule n Fgure 7(b). The qualtatve behavor n throughput, farness, stablty, and responsveness for each of

9 9 x ms x ms x ms x ms x ms x ms x ms 9 x ms x ms x ms (a) Dynamc scenaro I ( flows) (b) Dynamc scenaro II ( flows) Fg. 7. Dynamc scenaro: each colored block represents one or more connectons of certan propagaton delay. The left and the rght edges of each block represent the startng and endng tmes, respectvely, of the flow(s). Reno STCP Reno x qavg (pkt) x x Dynamc scenaro I: dummynet queue szes, losses, and lnk utlzaton. FAST STCP x cwnd (pkt) cwnd (pkt). Reno cwnd (pkt) Fg q 9 cumulatve loss (pkt) cumulatve loss (pkt) cumulatve loss (pkt) x q avg avg (pkt) (pkt) qavg (pkt) cumulatve loss (pkt) Dynamc scenaro I: throughput and cwnd trajectores. FAST x 9 x Fg.. 7 x x cwnd(pkt) cwnd(pkt) cwnd (pkt) STCP cwnd (pkt) cwnd (pkt) FAST... x.. Fg... x.... x. x. x x. x x. x... x.. x x Dynamc scenaro II: throughput and cwnd trajectores. the protocols s smlar to those n scenaro I, and n fact s amplfed as the number of flows ncreases. Specfcally, as the number of competng sources ncreases n a network, stablty becomes worse for the loss-based protocols. As shown n Fgures and, oscllatons n both congeston wndows and queue sze are more severe for all loss-base protocols. Packet loss s also more severe. The performance of FAST TCP dd not degrade n any sgnfcant way. Connectons sharng the lnk acheved very smlar rates. There was a reasonably stable queue at all tmes, wth lttle packet loss and hgh lnk utlzaton. Intra-protocol farness s shown n Table IV, wth lttle change for FAST TCP. E. Overall evaluaton We have conducted several other experments, wth dfferent delays, number of flows, and ther arrval and departure patterns. In all these experments, the bottleneck lnk capacty was Mbps and buffer sze packets. We present n ths subton a summary of protocol performance n terms of Tme Sources FAST STCP Reno TABLE IV FAIRNESS AMONG VARIOUS PROTOCOLS FOR EXPERIMENT II. some quanttatve measures on throughput, farness, stablty, and responsveness. We use the output of perf for our quanttatve evaluaton. Each perf sesson n our experments produced fve-ond averages of ts throughput. Ths s the data rate (.e., goodput) applcatons such as perf receves, and s slghtly less

10 FAST STCP Reno q avg (pkt) q avg (pkt) q avg (pkt) q avg (pkt) cumulatve loss (pkt).. x.. x x cumulatve loss (pkt).. x.. x x cumulatve loss (pkt).. x.. x x cumulatve loss (pkt).. x.. x x.. x.. x.. x.. x Fg.. Dynamc scenaro II: dummynet queue szes, losses, and lnk utlzaton. than the bottleneck bandwdth due to IP and Ethernet packet headers. Let x (k) be the average throughput of flow n the fveond perod k. Most tests nvolved dynamc scenaros where flows joned and departed. For the defntons below, suppose the composton of flows changes n perod k =, remans fxed over perod k =;:::;m, and changes agan n perod k = m +, so that [;m] s the maxmum-length nterval over whch the same equlbrum holds. Suppose there are n actve flows n ths nterval, ndexed by =;:::;n. Let mx x := x (k) m k= be the average throughput of flow over ths nterval. We now defne our performance metrcs for ths nterval [;m] usng these throughput measurements. ) Throughput: The average aggregate throughput for the nterval [;m] s defned as: 7 E := nx = ) Intra-protocol farness: Jan s farness ndex for the nterval [;m] s defned as []: x F = (P n = x ) n P n = x F (; ] and F =s deal (equal sharng). ) Stablty: The stablty ndex of flow s the sample standard devaton normalzed by the average throughput: S := x v uut m mx k= (x (k) x ) The smaller the stablty ndex, the less oscllaton a source experences. The stablty ndex for nterval [;m] s the average over the n actve sources: nx S := S n = ) Responsveness: The responsveness ndex measures the speed of convergence when network equlbrum changes at k =,.e., when flows jon or depart. Let x (k) be the runnng average by perod k» m: kx x (k) := x (k) k t= Then x (m) =x s the average over the entre nterval [;m]. 7 As mentoned above, ths s the throughput (or goodput) seen at the applcaton layer, not TCP layer. Responsveness ndex R measures how fast the runnng average x (k) of the slowest source converges to x : R := max max ρ f f f k : f x (k) x x f fff > : For each TCP protocol, we obtan one set of computed values for each evaluaton crteron for all of our experments. We plot the CDF (cumulatve dstrbuton functon) of each set of values. These are shown n Fgures. From Fgures, FAST has the best performance among all protocols under each evaluaton crteron. More mportantly, the varaton n each of the dstrbutons s smaller under FAST than under the other protocols, suggestng that FAST had farly consstent performance n our test scenaros. We also observe that both and STCP acheved hgher throughput and mproved responsveness compared wth TCP Reno. STCP had worse ntra-protocol farness compared wth TCP Reno, whle acheved comparable ntra-protocol farness to Reno (see Fgures, and ). Both and STCP showed ncreased oscllatons compared wth Reno (Fgures, and 9), and the oscllatons became worse as the number of sources ncreased (Fgures and ). From Fgure, FAST TCP acheved a much better responsveness ndex R (whch s based on worst case ndvdual throughput) than the other schemes. We cauton however that t can be hard to quantfy responsveness for protocols that do not stablze nto an equlbrum pont or a perodc lmt cycle, and hence the unresponsveness of Reno,, and STCP, as measured by ndex R, should be nterpreted wth care. VII. CONCLUSION We have descrbed an alternatve congeston control algorthm, FAST TCP, that addresses the four man problems of TCP Reno n networks wth hgh capactes and large latences. FAST TCP has a log utlty functon and acheves weghted proportonal farness. Its wndow adjustment s equatonbased, under whch the network moves rapdly toward equlbrum when the current state s far away and slows down when t approaches the equlbrum. FAST TCP uses queueng delay, n addton to packet loss, as a congeston sgnal. Queueng delay provdes a fner measure of congeston and scales naturally wth network capacty. We have presented expermental results of our frst Lnux prototype and compared ts performance wth TCP Reno,, and STCP. We have evaluated these algorthms not only n statc envronments, but also dynamc envronments where flows come and go, and not only n terms of end-to-end throughput, but also queue behavor n the network. In these The natural defnton of responsveness ndex as the earlest perod after whch the throughput x (k) (as opposed to the runnng average x (k) of the throughput) stays wthn % of ts equlbrum value s unsutable for TCP protocols that do not stablze nto an equlbrum value. Hence we defne t n terms of x (k) whch, by defnton, always converges to x by the end of the nterval k = m. Ths defnton captures the ntutve noton of responsveness f x (k) settles nto a perodc lmt cycle. ff