Bridging Router Performance and Queuing Theory

Transcription

1 Brdgng Router Performance and Queung Theory N. Hohn D. Vetch Department of Electrcal Engneerng Unversty of Melbourne, Australa {n.hohn, K. Papagannak C. Dot Intel Research Cambrdge, UK {dna.papagannak, ABSTRACT Ths paper provdes an authortatve knowledge of through-router packet delays and therefore a better understandng of data network performance. Thanks to a unque expermental setup, we capture all packets crossng a router for hours and present detaled statstcs of ther delays. These measurements allow us to buld the followng physcal model for router performance: each packet experences a mnmum router processng tme before enterng a flud output queue. Although smple, ths model reproduces the router behavour wth excellent accuracy and avods two common ptfalls. Frst we show that n-router packet processng tme accounts for a sgnfcant porton of the overall packet delay and should not be neglected. Second we pont out that one should fully understand both lnk and physcal layer characterstcs to use the approprate bandwdth value. Focusng drectly on router performance, we provde nsghts nto system busy perods and show precsely how queues buld up nsde a router. We explan why current practces for nferrng delays based on average utlzaton have fundamental problems, and propose an alternatve soluton to drectly report router delay nformaton based on busy perod statstcs. Categores and Subject Descrptors C.. [Computer-communcatons Networks]: Network Operatons Network ng General Terms Measurement, Theory Keywords Packet delay analyss, router model Ths work was done when Ncolas Hohn, Darryl Vetch and Konstantna Papagannak were wth the Sprnt Advanced Technology Laboratores, n Burlngame, CA, USA. Permsson to make dgtal or hard copes of all or part of ths work for personal or classroom use s granted wthout fee provded that copes are not made or dstrbuted for proft or commercal advantage and that copes bear ths notce and the full ctaton on the frst page. To copy otherwse, to republsh, to post on servers or to redstrbute to lsts, requres pror specfc permsson and/or a fee. SIGMETRICS/Performance, June,, New York, NY, USA. Copyrght ACM -8--//...$... INTRODUCTION End-to-end packet delay s an mportant metrc to measure n networks, both from the network operator and applcaton performance ponts of vew. An mportant component of ths delay s the tme for packets to traverse the dfferent forwardng elements along the path. Ths s partcularly mportant for network provders, who may have Servce Level Agreements (SLAs) specfyng allowable values of delay statstcs across the domans they control. A fundamental buldng block of the path delay experenced by packets n Internet Protocol (IP) networks s the delay ncurred when passng through a sngle IP router. Examnng such through-router delays s the man topc of ths paper. Although there have been many studes examnng delay statstcs measured at the edges of the network, very few have been able to report wth any degree of authorty on what actually occurs at swtchng elements. In [8] an analyss of sngle hop delay on an IP backbone network was presented, and dfferent delay components were solated. However, snce the measurements were lmted to a subset of the router nterfaces, only samples of the delays experenced by packets, on some lnks, were dentfed. In [] sngle hop delays were also obtaned for a router. However snce the router only had one nput and one output lnk, whch were of the same speed, the nternal queueng was extremely lmted. Ths s not a typcal operatng scenaro, and n partcular t led to the through-router delays beng extremely low. In ths paper we work from a data set recordng all IP packets traversng a Ter- access router over a hour perod. All nput and output lnks were ed, allowng a complete pcture of through-router delays to be obtaned. The frst am of ths paper s to explot the unque certanty provded by the data set by reportng n detal on the actual magntudes, and temporal structure, of delays on a subset of lnks whch experenced sgnfcant congeston: mean utlsaton levels on the target output lnk ranged from ρ =. to ρ =.7. Hgh utlsaton scenaros wth sgnfcant delays are of the most nterest, and yet are rare n today s backbone IP networks. From a measurement pont of vew, ths paper provdes the most comprehensve pcture of endto-end router delay performance that we are aware of. We base all our analyss on emprcal results and do not make any assumptons on traffc statstcs or router functonaltes. Our second am s to use the completeness of the data as a tool to nvestgate how packet delays occur nsde the router, n other words to provde a physcal model of the router delay performance. For ths purpose we frst poston ourselves n the context of the popular store & forward router archtectures wth Vrtual Output Queues (VOQs) at the nput lnks []. We are able to confrm n a wth one neglgble excepton.

2 detaled way the prevalng assumpton that the bottleneck of such an archtecture s n the output queues, and justfy the commonly used flud output queue model for the router. We go further to provde two refnements to the smple queue dea whch lead to a model wth excellent accuracy, close to the lmts of tmestampng precson. We explan why the model should be robust to many detals of the archtecture. The model focuses on datapath functons, performed at the hardware level for every IP datagram. It only mperfectly takes account of the much rarer control functons, performed n software on a very small subset of packets. The thrd contrbuton of the paper s to combne the nsghts from the data, and smplcatons from the model, to address the queston of how delay statstcs can be most effectvely summarsed and reported. Currently, the exstng Smple Network Management Protocol (SNMP) focuses on reportng utlsaton statstcs rather than delay. Although t s possble to gan nsght nto the duraton and ampltude of congeston epsodes through a mult-scale approach to utlsaton reportng [7], the connecton between the two s complex and strongly dependent on the structure of traffc arrvng to the router. We explan why tryng to nfer delay from utlsaton s n fact fundamentally flawed, and propose a new approach based on drect reportng of queue level statstcs. Ths s practcally feasble as buffer levels are already made avalable to actve queue management schemes mplemented n modern routers (note however that actve management was swtched off n the router under study). We propose a computatonally feasble way of recordng the structure of congeston epsodes, and reportng them back va SNMP. The statstcs we select are rch enough to allow detaled metrcs of congeston behavour to be estmated wth reasonable accuracy. A key advantage s that a genercally rch descrpton s reported, wthout the need for any traffc assumptons. The paper s organzed as follows. The router measurements are presented n secton, and analyzed n secton, where the methodology and sources of error are descrbed n detal. In secton we construct and justfy the router model, measure ts accuracy and dscuss the nature of resdual errors. In secton we defne congeston epsodes and show how mportant detals of ther structure can be captured n a smple way. We then descrbe how to report the statstcs wth low bandwdth requrements, and llustrate how such measurements can be exploted.. FULL ROUTER MONITORING In ths secton we descrbe the hardware nvolved n the passve measurements, present our experment setup to a full router, and detal how packets from dfferent traces are matched.. Hardware consderatons We frst gve the most pertnent features of the archtecture of the router we, and then recall relevant physcal consderatons of the SONET lnk layer, before descrbng our passve measurement nfrastructure... Router archtecture As mentoned n the ntroducton, our router s of the store & forward type, and mplements Vrtual Output Queues (VOQ). Detals of such an archtecture can be found n []. The router s essentally composed of a swtchng fabrc controlled by a centralzed scheduler, and nterfaces or lnecards. Each lnecard controls two lnks: one nput and one output. A typcal datapath followed by a packet crossng the router s as follows.when a packet arrves at the nput lnk of a lnecard, ts destnaton address s looked up n the forwardng table. Ths does not occur however untl the packet completely leaves the nput lnk and fully arrves n the lnecard s memory,.e. the store part of store & forward. Vrtual Output Queung means that each nput nterface has a separate Frst In Frst Out (FIFO) queue dedcated to each output nterface. The packet s stored n the approprate queue of the nput nterface where t s decomposed nto fxed length cells. When the packet reaches the head of lne t s transmtted through the swtchng fabrc cell by cell (possbly nterleaved wth competng cells from VOQ s at other nput nterfaces dedcated to the same output nterface) to ts output nterface, and reassembled before beng handed to the output lnk scheduler,.e. the forward part of store & forward. The packet mght then experence queung before beng seralsed wthout nterrupton onto the output lnk. In queung termnology t s served at a rate equal to the bandwdth of the output lnk, and the output process s of flud type because the packet flows out gradually nstead of leavng n an nstant. In the above descrpton the packet mght be queued both at the nput nterface and the output lnk scheduler. However n practce the swtch fabrc s overprovsoned and therefore very lttle queueng should be expected at the nput queues... Layer overheads Each nterface on the router uses the Hgh Level Data Lnk Control (HDLC) protocol as a transport layer to carry IP datagrams over a Synchronous Optcal NETwork (SONET) physcal layer. Packet over SONET (PoS) s a popular choce to carry IP packets n hgh speed networks because t provdes a more effcent lnk layer than IP over ATM, and faster falure detecton than broadcast technologes. We now detal the calculaton of the bandwdth avalable to IP datagrams encapsulated wth HDLC over SONET. The frst level of encapsulaton s the SONET framng mechansm. A basc SONET OC- frame contans 8 bytes and s repeated wth a 8kHz frequency. Ths yelds a nomnal bandwdth of.8mbps. Snce each SONET frame s dvded nto a transport overhead of 7 bytes, a path overhead of bytes and an effectve payload of 78 bytes, the bandwdth accessble to the transport protocol, also called the IP bandwdth, s n fact 9.9 Mbps. OC-n bandwdth (wth n {,, 8, 9}) s acheved by mergng n basc frames nto a sngle larger frame, and sendng t at the same 8kHz rate. In ths case the IP bandwdth s (9.9 n) Mbps. For nstance the IP bandwdth of an OC- lnk s exactly 9.7 Mbps. The second level of encapsulaton s the HDLC transport layer. Ths protocol adds bytes before and bytes after each IP datagram, rrespectve of the SONET nterface speed []. These layer overheads mean that n terms of queung behavour, an IP datagram of sze b bytes carred over an OC- lnk should be consdered as a b + 9 byte packet transmtted at 9.7 Mbps. The mportance of these seemngly techncal ponts wll be demonstrated n secton... Tmestampng of PoS packets All measurements are made usng hgh performance passve ng DAG cards []. We use DAG. cards to OC-c and OC-c lnks, and DAG. cards to OC-8 lnks. The cards use dfferent technologes to tmestamp PoS packets. DAG. cards are based on a desgn dedcated to ATM measurement and therefore operate wth byte chunks correspondng to the length of an ATM cell. The PoS tmestampng functonalty was added at a later stage wthout alterng the orgnal byte processng scheme. However, snce PoS frames are not algned wth the byte dvsons of the PoS stream operated by the DAG card, sgnfcant tmestampng errors occur. In fact, a tmestamp s generated when a new SONET frame s detected wthn a byte chunk. Ths mechansm can cause errors of up to.µs on an OC- lnk [].

3 Set Lnk # packets Average rate Matched packets Duplcate packets Router traffc (Mbps) (% total traffc) (% total traffc) (% total traffc) BB n %.. out %.. BB n %.8.9 out %.8.8 C out %.. n 9 99.%.9. C out %.. n %.. C out %.. n. N/A N/A N/A C out %.9.8 n 99.7%.9.8 Table : Trace detals: Each was collected on Aug., between : : UTC. n BB out n BB out OC8 OC8 OC OC OC OC out C n out n out n out n C C C GPS clock sgnal Fgure : Expermental setup: gateway router wth synchronzed DAG cards. DAG. cards are dedcated to PoS measurement and do not suffer from the above lmtatons. They look past the PoS encapsulaton (n ths case HDLC) to consstently tmestamp each IP datagram after the frst ( bt) word has arrved. As a drect consequence of the characterstcs of the measurement cards, tmestamps on OC- lnks have a worst case precson of.µs. Addng errors due to potental GPS synchronzaton problems between dfferent DAG cards leads to a worst case error of µs []. Ths number should be kept n mnd when we assess our router model performance.. Expermental setup The data analyzed n ths paper was collected n August at a gateway router of the Sprnt IP backbone network. Sx nterfaces of the router were ed, accountng for more than 99.9% of all traffc flowng through t. The expermental setup s llustrated n fgure. Two of the nterfaces are OC-8 lnecards connectng to two backbone routers (BB and BB), whle the other four connect customer lnks: two trans-pacfc OC- lnecards to Asa (C and C), one OC- (C) and one OC- (C) lnecard to domestc customers. A small lnk carryng less than packets per second was not ed for techncal reasons. Each DAG card s synchronzed wth the same GPS sgnal and outputs a fxed length byte record for each packet on the ed lnk. The detals of the record depend on the lnk type (ATM, SONET or Ethernet). In our case all the IP packets are PoS packets, and each byte record conssts of 8 bytes for the tmestamp, bytes for control and PoS headers, bytes for the IP header and the frst bytes of the IP payload. We captured hours of mutually synchronzed traces, representng more than 7. bllon IP packets or Tera Bytes of traffc. The DAG cards are located physcally close enough to the router so that the tme taken by packets to go between them can be neglected.. Packet matchng The next step after the trace collecton s the packet matchng procedure. It conssts n dentfyng, across all the traces, the records correspondng to the same packet appearng at dfferent nterfaces at dfferent tmes. In our case the records all relate to a sngle router, but the packet matchng program can also accommodate mult-hop stuatons. We descrbe below the matchng procedure, and llustrate t n the specfc case of the customer lnk C-out. Our methodology follows [8]. We match dentcal packets comng n and out of the router by usng a hash table. The hash functon s based on the CRC algorthm and uses the IP source and destnaton addresses, the IP header dentfcaton number, and n most cases the full byte IP header data part. In fact when a packet sze s less than bytes, the DAG card uses a paddng technque to extend the record length to bytes. Snce dfferent models of DAG cards use dfferent paddng content, the padded bytes are not ncluded n the hash functon. Our matchng algorthm uses a sldng wndow over all the synchronzed traces n parallel to match packets hashng to the same key. When two packets from two dfferent lnks are matched, a record of the nput and output tmestamps as well as the byte PoS payload s produced. Sometmes two packets from the same lnk hash to the same key because they are dentcal: these packets are duplcate packets generated by the physcal layer []. They can create ambgutes n the matchng process and are therefore dscarded, however ther frequency s ed. Matchng packets s computatonally ntensve and demandng n terms of storage: the total sze of the result fles rvals that of the raw data. For each output lnk of the router, the packet matchng program creates one fle of matched packets per contrbutng nput lnk. For nstance, for output lnk C-out four fles are created, correspondng to the packets comng respectvely from BB-n, BB- n, C-n and C-n (the nput lnk C-n has vrtually no traffc and s dscarded by the matchng algorthm). All the packets on a lnk for whch no match could be found were carefully analyzed. Apart from duplcate packets, unmatched packets comprse packets gong to or comng from the small uned lnk, or wth source or destnaton at the router nterfaces themselves. There could also be unmatched packets due to packet drops at the router. Snce the router dd not drop a sngle packet over the hours, no such packets were found. Assume that the matchng algorthm has determned that the m th

4 9 (a) Total output C out nput BB n to C out nput BB n to C out Total nput 8 (b) Total output C out nput BB n to C out nput BB n to C out Total nput Lnk Utlzaton (Mbps) 8 7 Lnk Utlzaton (kpps) 8 : 9: : : Tme of day (HH:MM UTC) : 9: : : Tme of day (HH:MM UTC) Fgure : Utlzaton for lnk C-out n (a): Megabt per second (Mbps) and (b): klo packet per second (kpps). Set Lnk # Matched packets % traffc on C-out C n 987.% C n 77.% BB n 79 7.% BB n % C out % Table : Breakdown of packet matchng for output lnk C-out. packet of output lnk Λ j corresponds to the n th packet of nput lnk λ. Ths can be formalzed by a matchng functon M, obeyng M(Λ j, m) = (λ, n). () The matchng procedure effectvely defnes ths functon for all packets over all output lnks. Packets that can not be matched are not consdered part of the doman of defnton of M. Table summarzes the results of the matchng procedure. The percentage of matched packets s at least 99.% on each lnk, and as hgh as 99.98%, showng convncngly that almost all packets are matched. In fact, even f there were no duplcate packets and f absolutely all packets were ed, % could not be attaned because of router generated packets, whch represents roughly.% of all traffc. The packet matchng results for the customer lnk C-out are detaled n table. For ths lnk, 99.9% of the packets can be successfully traced back to packets enterng the router. In fact, C-out receves most of ts packets from the two OC-8 backbone lnks BB-n and BB-n. Ths s llustrated n fgure where the utlzaton of C-out across the full hours s plotted. The breakdown of traffc accordng to packet orgn shows that the contrbutons of the two ncomng backbone lnks are roughly smlar. Ths s the result of the Equal Cost Mult Path polcy deployed n the network when packets may follow more than one path to the same destnaton. Whle the utlzaton n Mbps n fgure (a) gves an dea of how congested the lnk mght be, the utlzaton n packets per second s mportant from a packet trackng perspectve. Snce the matchng procedure s a per packet mechansm, fgure (b) llustrates the fact that roughly all packets are matched: the sum of the nput traffc s almost ndstngushable from the output packet count. In the remander of the paper we focus on lnk C-out because t s the most hghly utlzed lnk, and s fed by two hgher capacty lnks. It s therefore the best canddate for observng queung behavour wthn the router.. PRELIMINARY DELAY ANALYSIS In ths secton we analyze the data obtaned from the packet matchng procedure. We start by carefully defnng the system under study, and then present the statstcs of the delays experenced by packets crossng t. The pont of vew s that of lookng from the outsde of the router, seen largely as a black box, and we concentrate on smple statstcs. In the next secton we begn to look nsde the router, and examne delays n greater detal.. System defnton Recall the notaton from equaton (): the m th packet of output lnk Λ j corresponds to the n th packet of nput lnk λ. The DAG tmestamps an IP packet on the ncomng nterface sde as t(λ, n), and later on the outgong nterface at tme t(λ j, m). As the DAG cards are physcally close to the router, one mght thnk to defne the through-router delay as t(λ j, m) t(λ, n). However, ths would amount to defnng the router system n a somewhat arbtrary way, because, as we showed n secton.., packets are tmestamped dfferently dependng on the measurement hardware nvolved. Furthermore there are several other dsavantages to such a defnton, leadng us to suggest the followng alternatve. For self-consstency and extensblty to a mult-hop scenaro, where we would lke ndvdual router delays to add, arrval and departure tmes of a packet should be measured consstently usng the same bt. It s natural to focus on the end of the (IP) packet for two reasons: () as a store & forward router, the output queue s the most mportant component to descrbe. It s therefore approprate to consder that the packet has left the router when t completes ts servce at the output queue, that s when t has completely exted the router. () Agan as a store and forward router, no acton (for example the forwardng decson) s performed untl the packet has fully entered the router. Thus the nput buffer can be consdered as part of the nput lnk, and packet arrval to occur after the arrval of the last bt. The arrval and departure nstants n fact defne the system, whch s the part of the router whch we study, and s not exactly the same as the physcal router as t excses the nput buffer. Ths buffer, beng a component whch s already understood, does not have to be modelled or measured. Defnng the system n ths way can be compared wth choosng the most practcal coordnate system to solve a gven problem. We now establsh the precse relatonshps between the DAG tmestamps defned earler and the tme nstants τ(λ, n) of arrval and τ(λ j, m) of departure of a gven packet to the system as just

5 λ, θ Λ Θ j, j t(λ,n) (a) tme.. mn mean max BB n to C out λ, θ Λ Θ j, j τ(λ,n) (b) tme log Delay (us).. λ, θ (c). Λ Θ j, j t(λ,m) j tme :: 9:: :: :: Tme of day (HH:MM UTC) λ, θ (d) Fgure : Packet delays from BB-n to C-out. above ms are due to opton packets. All delays Λ Θ j, j τ(λ,m) j Fgure : Four snapshots of a packet crossng the router. defned. Denote by l n = L m the sze of the packet n bytes when ndexed on lnks λ and Λ j respectvely, and let θ and Θ j be the correspondng lnk bandwdths n bts per second. We denote by H the functon gvng the depth of bytes nto the IP packet where the DAG tmestamps t. H s a functon of the lnk speed, but not the lnk drecton. For a gven lnk λ, H s defned as H(λ ) = f λ s an OC-8 lnk, tme = b f λ s an OC- or OC- lnk, where we take b to be a unformly dstrbuted nteger between and mn(l n, ) to account for the ATM based dscretsaton descrbed earler. We can now derve the desred system arrval and departure event tmes as: τ(λ, n) = t(λ, n) + 8(l n H(λ ))/θ () τ(λ j, m) = t(λ j, m) + 8(L m H(Λ j))/θ j These defntons are dsplayed schematcally n fgure. The snapshots are: (a): the packet s tmestamped by the DAG card ng the nput nterface at tme t(λ, n), at whch pont t has already entered the router, but not yet the system, (b): t has fnshed enterng the router (arrves at the system) at tme τ(λ, n), and (c): s tmestamped by the DAG at the output nterface at tme t(λ j, m). Fnally (d): t fully exts the router (and system) at tme τ(λ j, m). Wth the above notatons, the through-system delay experenced by packet m on lnk Λ j s defned as d λ,λ j (m) = τ(λ j, m) τ(λ, n). () To smplfy notatons we shorten ths to d(m) n what follows.. Delay statstcs A thorough analyss of sngle hop delays was presented n [8]. Here we follow a smlar methodology and obtan comparable results, but wth the added certanty ganed from not needng to address the samplng ssues caused by unobservable packets on the nput sde. Fgure shows the mnmum, mean and maxmum delay experenced by packets gong from nput lnk BB-n to output lnk C-out over consecutve mnute ntervals. As observed n [8], there s a constant mnmum delay across tme, up to tmestampng precson. The fluctuatons n the mean delay follow roughly the changes n the lnk utlzaton presented n fgure. The maxmum delay value has a nosy component wth smlar varatons to the mean, as well as a spky component. All the spkes above ms have been ndvdually studed. The analyss revealed that they are caused by IP packets carryng optons, representng less than.% of all packets. Opton packets take dfferent paths through the router snce they are processed through software, whle all other packets are processed wth dedcated hardware on the socalled fast path. Ths explans why they take sgnfcantly longer to cross the router. In any router archtecture t s lkely that many components of delay wll be proportonal to packet sze. Ths s certanly the case for store & forward routers, as dscussed n []. To nvestgate ths here we compute the excess mnmum delay experenced by packets of dfferent szes, that s not ncludng ther transmsson tme on the output lnk, a packet sze dependent component whch s already understood. Formally, for every packet sze L we compute λ,λ j (L) = mn m {d λ,λ j (m) 8l m/θ j l m = L}. () Note that our defnton of arrval tme to the system convenently excludes another packet sze dependent component, namely the tme nterval between begnnng and completng entry to the router at the nput nterface. Fgure shows the values of λ,λ j (L) for packets gong from BB-n to C-out. The IP packet szes observed vared between 8 and bytes. We assume (for each sze) that the mnmum value found across hours corresponds to the true mnmum,.e. that at least one packet encountered no contenton on ts way to the output queue and no packet n the output queue when t arrved there. In other words, we assume that the system was empty from the pont of vew of ths nput-output par. Ths means that the excess mnmum delay corresponds to the tme taken to make a forwardng decson (not packet sze dependent), to dvde the packet nto cells, transmt t across the swtch fabrc and reassemble t (each beng packet sze dependent operatons), and fnally to delver t to the approprate output queue. The step lke curve means that there exst ranges of packet szes wth the same mnmum transt tme. Ths s consstent wth the fact that each packet s dvded nto fxed length cells, transmtted through the backplane cell by cell, and reassembled. A gven number of cells can therefore correspond to a contguous range of packet szes wth the same mnmum transt tme.

6 (a) Mnmum Router Transt Tme ( µs ) N nputs (b) N nputs Δ Δ N Δ packet sze (bytes) Fgure : Measured mnmum excess system transt tmes from BB-n to C-out.. MODELLING We are now n a poston to explot the completeness of the data set to look nsde the system. Ths enables us to fnd a physcally meanngful model whch can be used both to understand and predct the end-to-end system delay very accurately.. The flud queue We frst recall some basc propertes of FIFO queues that wll be central n what follows. Consder a FIFO queue wth a sngle server of determnstc servce rate µ, and let t be the arrval tme to the system of packet of sze l bytes. We assume that the entre packet arrves nstantaneously (whch models a fast transfer across the swtch), but t leaves progressvely as t s served (modellng the output seralsaton). Thus t s a flud queue at the output but not at the nput. Nonetheless we wll for convenence refer to t as the flud queue. Let W be the length of tme packet wats before beng served. The servce tme of packet s smply l /µ, so the system tme, that s the total amount of tme spent n the system, s S = W + l µ. () The watng tme of the next packet ( + ) to enter the system can be expressed by the followng recurson: W + = [W + l µ (t+ t)]+, () where [x] + = max(x, ). The servce tme of packet + reads S + = [S (t + t )] + + l+ µ. (7) We denote by U(t) the amount of unfnshed work at tme t, that s the tme t would take, wth no further nputs, for the system to completely dran. The unfnshed work at the nstant followng the arrval of packet s nothng other than the end-to-end delay that that packet wll experence across the queung system. It s therefore the natural mathematcal quantty to consder when studyng delay. Note that t s defned at all real tmes t.. A smple router model The delay analyss of secton revealed two man features of the system delay whch should be taken nto account n a model: the mnmum delay experenced by a packet, whch s sze, nterface, and archtecture dependent, and the delay correspondng to the tme spent n the output buffer, whch s a functon of the rate of the output nterface and the occupancy of the queue. The delay across Fgure : Router mechansms: (a) Smple conceptual pcture ncludng VOQs. (b) Actual model wth a sngle common mnmum delay. the output buffer could by tself be modelled by the flud queue as descrbed above, however t s not mmedately obvous how to ncorporate the mnmum delay property n a sensble way. Assume for nstance that the router has N nput lnks λ,..., λ N contrbutng to a gven output lnk Λ j and that a packet of sze l arrvng on lnk λ experences at least the mnmum possble delay λ,λ j (l) before beng transferred to the output buffer. A representaton of ths stuaton s gven n fgure (a). Our frst problem s that gven dfferent technologes on dfferent nterfaces, the functons λ,λ j,..., λn,λ j are not necessarly dentcal. The second s that we do not know how to measure, nor to take nto account, the potentally complex nteractons between packets whch do not experence the mnmum excess delay but some larger value due to contenton n the router arsng from cross traffc. We address ths by n fact smplfyng the pcture stll further, n two ways. Frst we assume that the mnmum delays are dentcal across all nput nterfaces: a packet of sze l arrvng on lnk λ and leavng the router on lnk Λ j now experences an excess mnmum delay Λj (l) = mn { λ,λ j (l)}. (8) In the followng we drop the subscrpt Λ j to ease the notaton. Second, we assume that the multplexng of the dfferent nput streams takes place before the packets experence ther mnmum delay. By ths we mean that we preserve the order of ther arrval tmes and consder them to enter a sngle FIFO nput buffer. In dong so, we effectvely gnore all complex nteractons between the nput streams. Our hghly smplfed pcture, whch s n fact the model we propose, s shown n fgure (b). We wll justfy these smplfcatons a posteror n secton. where the comparson wth measurement shows that the model s remarkably accurate. We now explan why we can expect ths accuracy to be robust. Suppose that a packet of sze l enters the system at tme t + and that the amount of unfnshed work n the system at tme t was U(t ) > (l). The followng two scenaros produce the same total delay: () the packet experences a delay (l), then reaches the output queue and wats U(t) (l) > before beng served, or () the packet reaches the output queue straght away and has to wat U(t) before beng served. In other words, as long as there s more than an amount (l) of work n the queue when a packet of sze l enters the system, the fact that the packet should wat (l) before reachng the output queue can be neglected. Once the system s busy, t behaves exactly

7 queue sze ( µs ) 8 data model queue sze ( µs ) data model Fgure 7: Comparsons of measured and predcted delays on lnk C-out: Grey lne: unfnshed work U(t) n the system accordng to the model, Black dots: measured delay value for each packet. lke a smple flud queue. Ths mples that no matter how complcated the front end of the router s, one can smply neglect t when the output queue s suffcently busy. The errors made through ths approxmaton wll be strongly concentrated on packets wth very small delays, whereas the more mportant medum to large delays wll be fathfully reproduced. Apart from ts smplcty, ths robustness s the man motvaton for the model. A system equaton for our two stage model can be derved as follows. Assume that the system s empty at tme t and that packet k of sze l enters the system at tme t +. It wats (l) before reachng the empty output queue where t mmedately starts beng served. Its servce tme s l /µ and therefore ts total system tme s S = (l ) + l µ. (9) Suppose a second packet enters the system at tme t and reaches the output queue before the frst packet has fnshed beng served,.e. t + (l ) < t + S. It wll start beng served when packet k leaves the system,.e at t + S. Its system tme wll therefore be: S = S (t t ) + l µ. The same recurson holds for successve packets k and k + as long as the amount of unfnshed work n the queue remans above (l k+ ) when packet k + enters the system: t k+ + (l k+ ) < t k + S k. () Therefore, as long as equaton () s verfed, the system tmes of successve packets are obtaned by the same recurson as for the case of a busy flud queue: S k+ = S k (t k+ t k ) + l k+ µ. () Suppose now that packet k + of sze l k+ enters the system at tme t + k+ and that the amount of unfnshed work n the system at tme t k+ s such that < U(t k+ ) < (l k+). In ths case, the output buffer wll be empty by the tme packet k + reaches t after havng wated (l k+ ) n the frst stage of the model. The servce tme of packet k + therefore reads S k+ = (l k+ ) + l k+ µ. () A crucal pont to note here s that n ths stuaton, the output queue can be empty but the system stll busy wth a packet watng n the front end. Ths s also true of the actual router. Once the queue has draned, the system s dle untl the arrval of the next packet. The tme between the arrval of a packet to the empty system and the tme when the system becomes empty agan defnes a system busy perod. In ths bref analyss, we have assumed an nfnte buffer sze. It s a reasonable assumpton snce t s qute common for a lne card to be able to accommodate up to ms worth of traffc.. Evaluaton We now evaluate our model and compare ts results wth emprcal delay measurements. The model delays are obtaned by multplexng the traffc streams BB-n to C-out and BB-n to C-out and feedng the resultng packet tran to the model n an exact trace drven smulaton. Fgure 7 shows two sample paths of the unfnshed work U(t) correspondng to two fragments of real traffc destned to C-out. The process U(t) s a rght contnuous jump process where each jump marks the arrval tme of a new packet. The resultant new local maxmum s the tme taken by the newly arrved packet to cross the system, that s ts delay. The black dots represent the actual measured delays for the correspondng nput packets. In practce the queue state can only be measured when a packet enters the system. Thus the black dots can be thought of samples of U(t) obtaned from measurements, and agreement between the two seems very good. In order to see the lmtatons of our model, we focus on a set of busy perods on lnk C-out nvolvng packets all together. The top plot of fgure 8 shows the system tmes experenced by ncomng packets, both from the model and from measurements. The largest busy perod on the fgure has a duraton of roughly ms and an ampltude of more than ms. Once agan, the model reproduces the measured delays very well. The lower plot n fgure 8 shows the error of our model, that s the dfference between measured and modeled delays at each packet arrval tme, plotted on the same tme axs as the upper plot. There are three man ponts one can make about the model accuracy. Frst, the absolute error s wthn µs of the measured delays for almost all packets. Second, the error s much larger for a few packets, as shown by the spky behavour of the error plot. These spkes are due to a local reorderng of packets nsde the router that s not captured by our model. Recall from fgure (b) that we made the smplfyng assumpton that the multplexng of the nput streams takes place before the packets experence ther mnmum delay. Ths means that packets ext our system n the exact same order as they entered t. However n practce local reorderng can happen when a large packet arrves at the system on one nterface just before a small packet on another nterface. Gven that the mnmum transt tme of a packet depends lnearly on ts sze (see

8 delay ( µs ) error ( µs ) measured delays model Fgure 8: Measured delays and model predctons (top), Absolute error between data and model (bottom). fgure ), the small packet can overtake the large one and reach the output buffer frst. Once the two packets have reached the output buffer, the amount of work n the system s the same, rrespectvely of ther arrval order. Thus these local errors do not accumulate. Intutvely, local reorderng requres that two packets arrve almost at the same tme on two dfferent nterfaces. Ths s much more lkely to happen when the lnks are busy. Ths s n agreement wth fgure 8 whch shows that spkes always happen when the queung delays are ncreasng, a sgn of hgh local lnk utlzaton. The last pont worth notcng s the systematc lnear drft of the error across a busy perod duraton. Ths s due to the fact that our queung model drans slghtly faster than the real queue. We could not confrm any physcal reason why the IP bandwdth of the lnk C-out s smaller than what was predcted n secton... However, the mportant observaton s that ths phenomenon s only notceable for very large busy perods, and s lost n measurement nose for most busy perods. The model presented above has some lmtatons. Frst t does not take nto account the fact that a small number of opton packets wll take a slow software path through the router nstead of beng entrely processed at the hardware level. As a result, opton packets experence a much larger delay before reachng the output buffer, but as far as the model s concerned, transt tmes through the router only depend on packet szes. Second, the output queue stores not only the packets crossng the router, but also the unmatched packets generated by the router tself, as well as control PoS packets. These packets are not accounted for n the model. Despte ts smplcty, our model s consderably more accurate than other sngle-hop delay models. Fgure 9(a) compares the errors made on the packet delays from the OC- lnk C-out presented n fgure 8 wth three dfferent models: our two stage model, a flud queue wth OC- nomnal bandwdth, and a flud queue wth OC- IP bandwdth. As expected, wth a smple flud model,.e. when one does not take nto account the mnmum transt tme, all the delays are systematcally underestmated. If moreover one chooses the nomnal lnk bandwdth (. Mbps) for the queue nstead of a carefully justfed IP bandwdth (9.7 Mbps), the errors nsde a busy perod buld up very quckly because the queue drans too fast. There s n fact only a % dfference between the nomnal and effectve bandwdths, but ths s enough to create errors up 8µs nsde a moderately large busy perod. Fgure 9(b) shows the cumulatve dstrbuton functon of the delay error for a mnute wndow of C-out traffc. Of the delays nferred by our model, 9% are wthn µs of the measured ones. Gven the tmestampng precson ssues descrbed n secton.., these results are very satsfactory. We now evaluate the performance of our model over the entre hours of traffc on C-out as follows. We dvde the perod nto ntervals of mnutes. For each nterval, we plot the average relatve delay error aganst the average lnk utlzaton. The results are presented n fgure 9. The absolute relatve error s less than.% for the whole trace, whch confrms the excellent match between the model and the measurements. For large utlsaton levels, the relatve error grows due to the fact that large busy perods are more frequent. The packet delays therefore tend to be underestmated more often due to the unexplaned bandwdth msmatch occurrng nsde large busy perods. Overall, our model performs very well for a large range of lnk utlzatons.. Router model summary Based on the observatons and analyss presented above, we propose the followng smple approach for modelng store and forward routers. For each output lnk Λ j: () measure the mnmum excess (.e. excludng servce tme) packet transt tme λ,λ j between each nput λ and the gven output Λ j, as defned n equaton (). These depend only on the hardware nvolved, not the type of traffc, and could potentally be tabulated. Defne the overall mnmum packet transt tme Λj as the mnmum over all nput lnks λ, as descrbed n equaton (8). () calculate the IP bandwdth of the output lnk by takng nto account the dfferent levels of packet encapsulaton, as descrbed n secton... () obtan packet delays by aggregatng the nput traffc correspondng to the gven output lnk, and feedng t to a smple two stage model, llustrated n fgure (b), where packets are frst delayed by an amount Λj before enterng a FIFO queue. System equatons are gven n secton.. A model of a full router can be obtaned by puttng together the models obtaned for each output lnk Λ j. Although very smple, ths model performed remarkably well for our data set, where the router was lghtly loaded and the output buffer was clearly the bottleneck. As explaned above, we expect the model to contnue to perform well even under heaver load where nteractons n the front end become more pronounced, but not domnant. The accuracy would drop off under loads heavy enough to shft the bottleneck to the swtchng fabrc, when detals of the schedulng algorthm could no longer be neglected.. DELAY PERFORMANCE: UNDERSTANDING AND REPORTING. Motvaton From the prevous secton, our router model can accurately predct delays when the nput traffc s fully characterzed. However n practce the traffc s unknown, whch s why network operators rely on avalable smple statstcs, such as curves gvng upper bounds on delay as a functon of lnk utlzaton, when they want to nfer packet delays through ther networks. The problem s that

9 (a) (b) (c).8. error ( µs ).. Relatve error (%). 8 model flud queue wth OC effectve bandwdth flud queue wth OC nomnal bandwdth. error (µs) Lnk utlzaton (Mbps) Fgure 9: (a) Comparson of error n delay predctons from dfferent models of the sample path from fgure 8. (b) Cumulatve dstrbuton functon of model error over a mnute wndow on lnk C-out. (c) Relatve mean error between delay measurements and model on lnk C-out vs lnk utlzaton. these curves are not unque snce packet delays depend not only on the mean traffc rate, but also on more detaled traffc statstcs. In fact, lnk utlzaton alone can be very msleadng as a way of nferrng packet delays. Suppose for nstance that there s a group of back to back packets on lnk C-out. Ths means that packets follow each other on the lnk wthout gaps,.e. the local lnk utlzaton s %. However ths does not mply that these packets have experenced large delays nsde the router. They could very well be comng back to back from the nput lnk C-n wth the same bandwdth as C-out. In ths case they would actually cross the router wth mnmum delay n the absence of cross traffc. Inferrng average packet delays from lnk utlzaton only s therefore fundamentally flawed. Instead, we propose to study performance related questons by gong back to the source of large delays: queue buld-ups n the output buffer. In ths secton we use our understandng of the router mechansms obtaned from our measurements and modellng work of the prevous sectons to frst descrbe the statstcs and causes of busy perods, and second to propose a smple mechansm that could be used to report useful delay nformaton about a router.. Busy perods.. Defnton Recall from secton that we defned busy perods as the tme between the arrval of a packet n the empty system and the tme when the system goes back to ts empty state. The equvalent defnton n terms of measurements s as follows: a busy perod starts when a packet of sze l bytes crosses the system wth a delay (l) + l/µ, and t ends wth the last packet before the start of another busy perod. Ths defnton, whch makes full use of our measurements, s a lot more robust than an alternate defnton based solely on packet nter-arrval tmes at the output lnk. For nstance, f one were to detect busy perods by usng tmestamps and packet szes to group together back-to-back packets, the followng two problems would occur. Frst, tmestampng errors could lead to wrong busy perods separatons. Second and more mportantly, accordng to our system defnton from secton., packets belongng to the same busy perod are not necessarly back to back on the output lnk (see equaton )... Statstcs To descrbe busy perods, we begn by collectng per busy perod statstcs, such as duraton, number of packets and bytes, and ampltude (maxmum delay experenced by a packet nsde the busy perod). The cumulatve dstrbuton functons (CDF) of busy pe (a) 8 Ampltude (µs) busy perod ampltude (ms) (c) 7 8 busy perod duraton (ms).8... (b)... Duraton (ms) busy perod ampltude (ms) (d).... medan delay (ms) Fgure : (a) CDF of busy perod ampltudes. (b) CDF of busy perod duratons. (c) Busy perod ampltudes as a functon of busy perod duratons. (d) Busy perod ampltudes as a functon of medan packet delay. rod ampltudes and duratons are plotted n fgures (a) and (b) for a mnute traffc wndow. For ths traffc wndow, 9% of busy perods have an ampltude smaller than µs, and 8% last less than µs. Fgure (c) shows a scatter plot of busy perod ampltudes aganst busy perod duratons for ampltudes larger than ms on lnk C-out (busy perods contanng opton packets are not shown). There does not seem to be any clear pattern lnkng ampltude and duraton of a busy perod n ths data set, although roughly speakng the longer the busy perod the larger ts ampltude. A scatter plot of busy perod ampltudes aganst the medan delay experenced by packets nsde the busy perod s presented n fgure (d). One can see a lnear, albet nosy, relatonshp between maxmum and medan delay experenced by packets nsde a busy perod. Ths means ntutvely that busy perods have a regular shape,.e. busy perods where most of the packets experence small delays and only a few packets experence much larger delays are unlkely... Orgns Our full router measurements allow us to go further n the char-

10 (a) (b) (c) C out BB n to C out BB n to C out C out BB n to C out BB n to C out C out BB n to C out BB n to C out delay ( ms ) delay ( ms ) delay ( ms ) (d) (e) (f) delay ( ms ) delay ( ms ) delay ( ms ) Fgure : (a) (b) (c) Illustraton of the multplexng effect leadng to a busy perod on the output lnk C-out. (d) (e) (f) Collecton of largest busy perods n each mn nterval on the output lnk C-out. acterzaton of busy perods. In partcular, we can use our knowledge about the nput packet streams on each nterface to understand the mechansms that create the busy perods observed for our router output lnks. It s clear that, by defnton, busy perods are created by a local aggregate arrval rate whch exceeds the output lnk servce rate. Ths can be acheved by a sngle nput stream, the multplexng of dfferent nput streams, or a combnaton of both phenomena. A detaled analyss can be found n [9]. We restrct ourselves n ths secton to an llustraton of these dfferent mechansms. To create the busy perods shown n fgure, we store the ndvdual packet streams BB-n to C-out and BB-n to C-out, feed them ndvdually to our model and obtan vrtual busy perods. The delays obtaned are plotted on fgure (a), together wth the true delays measured on lnk C-out for the same tme wndow as n fgure 8. In the absence of cross traffc, the maxmum delay experenced by packets from each ndvdual nput stream s around ms. However, the largest delay for the multplexed nputs s around ms. The large busy perod s therefore due to the fact that the delays of the two ndvdual packet streams peak at the same tme. Ths non lnear phenomenon s the cause of all the large busy perods observed n our traces. A more surprsng example s llustrated n fgure (b) that shows one nput stream creatng at most a ms packet delay by tself and the other a successon of µs delays. The resultng congeston epsode for the multplexed nputs s agan much larger than the ndvdual epsodes. A dfferent stuaton s shown on fgure (c), where one lnk contrbutes almost all the traffc of the output lnk for a short tme perod. In ths case, the measured delays are almost the same as the vrtual ones caused by the busy nput lnk. It s nterestng to notce that the three large busy perods plotted n fgures (a), (b) and (c) all have a roughly trangular shape. Fgures (d), (e) and (f) that show that ths s not due to a partcular choce of busy perods. They were obtaned as follows. For each mn nterval, we detect the largest packet delay, store the correspondng packet arrval tme t, and plot the delays experenced by packets n a wndow ms before and ms after t. The resultng sets of busy perods are grouped accordng to the largest packet delay observed: fgure (d) when the largest ampltude s between ms and ms, fgure (e) between ms and ms, and fgure (f) between ms and ms. Other ampltude ranges were omtted for space reasons. For each of the plots (d), (e) and (f), the black lne hghlghts the busy perod detaled n the plot drectly above t. The strkng pont s that most busy perods have a roughly trangular shape. The largest busy perods have slghtly less regular shapes, but a trangular assumpton can stll hold. These results are remnscent of the theory of large devatons, whch states that rare events happen n the most lkely way. Some hnts on the shape of large busy perods n (Gaussan) queues can be found n [] where t s shown that, n the lmt of large ampltude, busy perods tend to be antsymmetrc about ther mdway pont, n agreement wth what we see here.. Modellng busy perod shape Although a trangular approxmaton may seem very crude at frst, we now study how useful such a model could be. To do so, we frst llustrate n fgure a basc prncple: any busy perod of duraton D seconds s bounded above by the busy perod obtaned n the case where the D seconds worth of work arrve n the system at maxmum nput lnk speed. The amount of work then decreases wth slope f no more packets enter the system. In the case of the OC- lnk C-out fed by the two OC-8 lnks BB and BB (each lnk beng tmes faster than C-out), t takes at least D/ seconds for the load to enter the system. From our measurements, busy perods are qute dfferent from ther theoretcal bound. The busy perod shown n fgures 8 and (a) s agan plotted n fgure for comparson. One can see that ts ampltude A s much lower than the theoretcal maxmum, n agreement wth the scatter

11 D measured busy perod theoretcal bound modelled busy perod.8. delay A L D tme Fgure : Modellng of busy perod shape wth a trangle. plot of fgure (c). In the rest of the paper we model the shape of a busy perod of duraton D and ampltude A by a trangle wth base D, heght A and same apex poston as the busy perod. Ths s llustrated n fgure by the trangle superposed over the measured busy perod. Ths very rough approxmaton can gve surprsngly valuable nsght nto packet delays. We defne our performance metrc as follows. Let L be the delay experenced by a packet crossng the router. A network operator mght be nterested n knowng how long a congeston level larger than L wll last, because ths gves a drect ndcaton of the performance of the router. Let d L,A,D be the length of tme the workload of the system remans above L durng a busy perod of duraton D and ampltude A, as obtaned from our delay analyss. Let d (T ) L,A,D be the approxmated duraton obtaned from the shape model. Both d L,A,D and are plotted wth a dashed lne n fgure. From basc geometry one can show that d (T ) L,A,D j D( L,A,D = L ) f A L A otherwse. d (T ) () In other words, d (T ) L,A,D s a functon of L, A and D only. For the metrc consdered, the two parameters (A, D) are therefore enough to descrbe busy perods, the knowledge of the apex poston does not mprove our estmate of d L,A,D. Denote by Π A,D the random process governng {A, D} pars for successve busy perods over tme. The mean length of tme durng whch packet delays are larger than L reads Z T L = d L,A,D dπ A,D. () T L can be approxmated by our busy perod model wth Z T (T ) L = d (T ) L,A,DdΠA,D. () We use equaton () to approxmate T L on the lnk C-out. The results are plotted on fgure for two mnute wndows of traffc wth dfferent average utlzatons. For both utlzaton levels, the measured duratons (sold lne) and the results from the trangular approxmaton (dashed lne) are farly smlar. Ths shows that our very smple trangular shape approxmaton captures enough nformaton about busy perods to answer questons about duraton of congeston epsodes of a certan level. The small dscrepancy between data and model can be consdered nsgnfcant n the context of Internet applcatons because a servce provder wll be realstcally only nterested n the order of magntude (ms, ms, ms) Mean duraton (ms) Data.7 utlzaton Equaton () Equaton (7) Data. utlzaton Equaton () Equaton (7).... L (ms) Fgure : Average duraton of a congeston epsode above L ms defned by equaton (), for two dfferent utlzaton levels (. and.7) on lnk C-out. Sold lnes: data, dashed lnes: equaton (), dots: equaton (7). of a congeston epsode greater than L. Our smple approach therefore fulflls that role very well. Let us now qualtatvely descrbe the behavours observed on fgure. For a small congeston level L, the mean duraton of the congeston epsode s also small. Ths s due to the fact that, although a large number of busy perods have an ampltude larger than L, as seen for nstance from the ampltude CDF n fgure (a), most busy perods do not exceed L by a large amount, so the mean duraton s small. It s also worth notcng that the results are very smlar for the two dfferent lnk utlzatons. Ths means that busy perods wth small ampltude are roughly smlar at ths tme scale, and do not depend on average utlzaton. As the threshold L ncreases, the (condtonal on L) mean duraton frst ncreases as there are stll a large number of busy perods wth ampltude greater than L on the lnk, and of these, most are consderably larger than L. Wth an even larger values of L however, fewer and fewer busy perods qualfy. The ones that do cross the threshold L do so for a smaller and smaller amount of tme, up to the pont where there are no busy perods larger than L n the trace.. Reportng busy perod statstcs The study presented above shows that one can get useful nformaton about delays by jontly usng the ampltude and duraton of busy perods. Now we look nto ways n whch such statstcs could be concsely reported usng SNMP. We start by formng busy perods from the queue sze values and collectng (A, D) pars durng mnutes ntervals. Ths s feasble n practce snce the queue sze s already accessed by other software such as actve queue management schemes. Measurng A and D s easly performed on-lne. In prncple we need to report the par (A, D) for each busy perod n order to recreate the process Π A,D and evaluate equaton (). Snce ths represents a very large amount of data n practce, we nstead assume that busy perods are ndependent and therefore that the full process Π A,D can be descrbed by the jont margnal dstrbuton F A,D of A and D. Thus, for each busy perod we need smply update a sparse -D hstogram. The bn szes should be as fne as possble consstent wth avalable computng power and memory. We do not consder these detals here. They are not crtcal snce at the end of the mnute nterval a much coarser dscretsaton s performed n order to lmt the volume of data fnally exported va SNMP. We control ths drectly by choosng N bns for each of the ampltude and the duraton dmensons. As we do not know a pror what delay values are common, the