Advances in Internet Quality of Service

Transcription

1 049.PDF 1 Advances n Internet Qualty of Servce Vctor Frou, Jean-Yves Le Boudec, Don Towsley, and Zh-L Zhang Abstract We descrbe recent advances n theores and archtecture that support performance guarantees needed for qualty of servce networks. We start wth determnstc computatons and gve applcatons to ntegrated servces, dfferentated servces, and playback delays. We revew the methods used for obtanng a scalable ntegrated servces support, based on the concept of a stateless core. New probablstc results that can be used for a statstcal dmensonng of dfferentated servces are explaned; some are based on classcal queung theory, whle others captalze on the determnstc results. Then we dscuss performance guarantees n a best effort context; we revew: methods to provde some qualty of servce n a pure best effort envronment; methods to provde some qualty of servce dfferentaton wthout access control, and methods that allow an applcaton to control the performance t receves, n the absence of network support. Keywords Qualty of Servce, Performance Guarantees, Network Calculus, Elastc Servces, Dfferentated Servces, Integrated Servces, Scalablty I. INTRODUCTION The problem of Internet QoS provsonng has been an extremely actve area of research for many years. From the earler Integrated Servces (IntServ) archtecture [1] to the more recent Dfferentated Servces (DffServ) archtecture [2], many QoS control mechansms, especally n the areas of packet schedulng and queue management algorthms, have been proposed. Elegant theores such as network calculus and effectve bandwdths have also been developed. Several books have been wrtten on the subject, some focus more on archtectural and other practcal ssues [3], [4], whle others on theoretcal aspects of QoS provsonng [5], [6]. To provde a more focused overvew, The authors afflatons and contact nformaton are: Vctor Frou, Nortel Networks, 600 Technology Park Drve, Blerca, MA USA, vfrou@nortelnetworks.com; Jean-Yves Le Boudec, EPFL-ICA, INN(Ecublens), CH-1015 Lausanne, Swtzerland, jean-yves.leboudec@epfl.ch; Don Towsley, Department of Computer Scence, Unversty of Massachussetts, Amherst, MA 01003, towsley@cs.umn.edu; and Zh-L Zhang, Department of Computer Scence and Engneerng, Unversty of Mnnesota, 200 Unon Street SE, Mnneapols, MN 55455, zhzhang@cs.umn.edu. Ths work was supported n part by the Natonal Scence Foundaton under the grants EIA , ITR , EIA , ANI , ITR , and CAREER Award NCR Any opnons, fndngs, and conclusons or recommendatons expressed n ths paper are those of the authors and do not necessarly reflect the vews of the Natonal Scence Foundaton. n ths paper we survey a number of recent advances n Internet QoS provsonng, wth emphass on theoretcal developments. The objectve s two-fold: 1) to provde the reader wth the state-of-the-art knowledge n a few selectve areas n Internet QoS provsonng, wth ponters for further readngs; and 2) to hghlght the ssues and challenges stll facng the development of scalable Internet QoS provsonng solutons. The selected areas we wll survey are: theory of network calculus for determnstc QoS guarantees; and archtectures and solutons for scalable QoS support; newly developed theores for provdng stochastc servces; servce dfferentaton wthn best effort; archtectures and control algorthms for elastc servces and adaptve applcaton QoS control. Before we start our survey n these areas, we frst ntroduce a few mportant notons and ssues n Internet QoS provsonng. They wll lay the background for our dscusson later. Network QoS can be defned n a varety of ways and nclude a dverse set of servce requrements such as performance, avalablty, relablty, securty, etc. All these servce requrements are mportant aspects of a comprehensve network QoS servce offerng. However, n ths paper we wll take a more performance-centrc vew of network QoS and focus prmarly on the ssues n provdng performance guarantees. Typcal performance metrcs used n defnng network QoS are bandwdth, delay/delay jtter, and packet loss rate. Usng these performance metrcs, network performance guarantees can be specfed n varous forms, such as absolute (or determnstc), e.g., a network connecton s guaranteed wth 10 Mbps bandwdth all the tme; probablstc (or stochastc), e.g., network delay s guaranteed to be no more than 100 ms for 95% of the packets; tme average, e.g., packet loss rate s less than 10 5 measured over a month. The guarantee feature of network QoS s what dfferentates t from the best-effort network servces. The exact form of performance guarantee wll be part of the servce level agreement (SLA) between the network servce provder and ts customers. There are lkely two major drvers for network servces wth QoS guarantees. One comes from applcatons wth strngent QoS requrements. Two possble examples of such applcatons are IP telephony and vdeo-on-demand (VoD) over the Internet. In IP telephony two end users send packetzed voce and the qualty of rendered sound

2 049.PDF 2 depends on low delay and small loss rate of end-end packet transmsson. Lkewse, streamng vdeos over the Internet requres adequate bandwdth and packet loss guarantees from the network to ensure TV-broadcast qualty. The other major drver for network QoS s the need for servce dfferentaton due to compettve nature of the marketplace. For example, one network servce provder may support a vrtual prvate network (VPN) servce over ts network wth only securty guarantee but no performance guarantee. Whereas, another network servce provder may support a vrtual leased lne (a form of VPN) servce over ts network that, n addton to securty guarantee, has bandwdth, delay and loss guarantees comparable to a physcal leased lne. The frst network servce provder may be forced to enhance ts VPN servce also wth performance guarantees or to lose ts customers who demand performance guarantees to ts compettor. Hence guaranteed QoS performance can serve as a servce dfferentatng feature for network servces. Apart from overprovsonng network resources, provdng QoS guarantees requres deployment of approprate QoS control mechansms n the operatons and management of a network. A vast varety of QoS control mechansms have been proposed and developed n last decade or so, wth varyng degree of complexty and cost. To help understand these mechansms and ther assocated complexty and cost, we consder several mportant aspects of QoS controls. A key aspect of QoS controls s the tme scale at whch a control mechansm operates. We can roughly dvde the tme scales of QoS controls nto a few levels. The fastest tme scale s at the packet level (ο1-100s μs), whch s the smallest unt a network can exert control. QoS control mechansms operatng at ths tme scale nclude traffc condtonng devces (e.g., traffc classfers, markers, polcers, and shapers), packet schedulers, and actve queue management. The next fastest tme scale s round-trp-tme (ο1-100s ms), at whch scale feedback-based QoS control mechansms such as congeston and flow control operate. Slower than packet tme and round-trp-tme s the sesson tme scale (seconds, mnutes or longer). Ths s the tme scale user sessons (defned n whatever meanngful way) typcally last, and at whch QoS control mechansms such as admsson control and QoS routng operate. Beyond the sesson tme scale, a varety of long-term QoS control mechansms operate at tme scales rangng from mnutes, hours, to days, weeks, or months. Examples nclude traffc engneerng, tme-of-day servce prcng, resource provsonng and capacty plannng. Another key aspect of QoS control s the granularty of control nformaton (.e., control state) used by a QoS mechansm n makng control decsons and exertng control. The fnest granularty of control s the per-flow state nformaton (e.g., as dentfed by the 5-tuple the IP source-destnaton addresses, port numbers and protocol feld carred n the IP header) whch can be used to enforce QoS for ndvdual user flows. Coarser-gran controls use nformaton that s specfed and mantaned for an aggregate of user flows: the granularty of coarse-gran QoS controls vares dependng on the level of flow aggregaton, such as per host, per network prefx, per ngressegress par, per servce class, etc. Closely related to the granularty of control are two other mportant aspects of QoS control the carrer of control state,.e., where the control state s stored, whether n routers or n the packet header only; and the locaton of control,.e., where a control mechansm operates, whether at the end-hosts, the network edge or boundares between ether users and network or network domans, or nsde the network core. We can vew the granularty of control, carrer of control state and locaton of control as formng the space dmenson of QoS control, whereas the tme scale the tme dmenson of QoS control. These two dmensons together defne a broad desgn space from whch QoS provsonng archtectures can be bult, reflectng varous trade-offs n QoS servce performance, operatons and management complexty and mplementaton cost. For example, control granularty has a drect mpact on the operatons and management complexty of network data plane (.e., the network elements such as routers that are drectly nvolved n data packet forwardng) and per-packet processng cost of network elements. It also affects the QoS servce performance ndvdual users wll experence. Tme scale of control determnes how frequently control nformaton must be conveyed to network elements, thus affectng ther processng, memory and communcaton overheads. Both the tme and space dmensons of QoS controls have enormous mplcatons n the desgn, operatons and management of network control plane (whch conssts of network control enttes such as routng processors, resource managers, servce confguraton modules that are not drectly nvolved n user data forwardng, but are essental to the operatons of a network). For example, a QoS provsonng archtecture that employs per-flow QoS control and stores QoS state at every router requres a sgnalng protocol that conveys QoS states to every router on a per-flow bass. Such an archtecture mandates a sophstcated control plane at every router, complcatng ts operatons and management and thus lmtng ts scalablty. Hence to desgn a scalable and costeffectve QoS provsonng archtecture, t s mperatve to make judcous desgn choces along the tme and space dmensons and carefully evaluate ther trade-offs and mp-

3 049.PDF 3 catons n both network data and control planes. The remander of ths paper s structured as follows. A recent seres of development has shown that determnstc computatons can be made more powerful wth the use of a few smple theores, based on mn and max calculus. Secton II ntroduces the reader to these development, and gves applcatons to ntegrated servces, dfferentated servces, and playback delays. Secton III explans the methods used for obtanng a scalable ntegrated servces support, based on the concept of a stateless core. Secton IV descrbes probablstc results that can be used for a statstcal dmensonng of dfferentated servces; some are based on classcal queung theory, whle others captalze on the determnstc results n Secton II to obtan stochastc bounds. In a best-effort context, QoS dfferentaton and guarantees can be provded based on queue management, traffc condtonng and engneerng, but need a consderable amount of network control nformaton, and guarantees are average, approxmate. Secton V descrbes the recent theores and the conclusons that can be drawn. Secton VI descrbes methods to provde some qualty of servce n a pure best-effort envronment, wthout any access control. Secton VII descrbes methods that allow an applcaton to control the QoS t receves, n the absence of network support. Secton VIII concludes the paper wth a short lst of challenges for the future. II. NETWORK CALCULUS, ATHEORY FOR THE DETERMINISTIC SETTING Determnstc bounds on quanttes such as loss and delay can be expressed f we combne constrants on traffc flows and servce guarantees. The bounds depend on the nature of the schedulers, and may be very complex to derve [7], [8], [9], [10], [11], [12]; see also [13] for a revew of packet schedulng. Many of these results can be cast nto a common framework coned network calculus, whch we explan n ths secton. In short, network calculus can be vewed as the applcaton of mn and max algebra to flow problems. It was poneered by Chang [14] and Cruz [15], [16], and found ts fnal form n subsequent work by the same authors and by Agrawal, Le Boudec and Rajan [17], [18], [19]. A comprehensve treatment can be found n two textbooks [5], [6]. We frst ntroduce network calculs on an example, then we revew applcatons to ntegrated servces, dfferentated servces, and the computaton of mnmum playback delay for vdeo sequences. A. Introductory Example: The Shaper Arrval Curves. Dfferentated and ntegrated servces assume that ndvdual traffc flows are lmted, for example usng the concept of token (or leaky ) bucket. More generally, gven some wde-sense ncreasng functon ff(t), we say that a flow s ff-smooth f the amount of data that can be observed on the flow over any tme wndow of duraton t s» ff(t). We also say that ff s an arrval curve for the flow. A token bucket, wth rate r and burst b corresponds to ff(t) = rt + b; ths s a common constrant mposed n traffc contracts between network and customer. Arrval curve constrants may also arse from physcal lmtatons. Consder a flow that s known to arrve on a lnk of bt rate equal to C bts/second; f the flow s observed bt by bt, then we can say that t s ff-smooth, wth ff(t) = Ct. Consder the same flow, but now observed at the lnk layer recever that termnates the lnk of bt rate C; here we observe entre packets nstead of bts. If the packet sze s M or less, then the flow s ff-smooth, wth ff(t) = Ct + M. Combnng a token bucket constrant, mposed as part of a traffc contract, wth a physcal lmtaton, gves an arrval curve of the form ff(t) =mn(ct+m;rt+b), whch s commonly used n the context of ntegrated servces ( T-SPEC [20]). Shapers. Traffc generated by sources cannot be expected to naturally satsfy some a pror arrval curve constrant; a shaper s used to force a flow to satsfy some arrval curve constrant. Gven some functon ff(t), a shaper stores ncomng bts n a buffer and delvers them n such a way that the resultng output s ff-smooth. A shaper s greedy f t delvers the data as soon as possble. If ff(t) = rt + b, the greedy shaper can be mplemented as a leaky bucket regulator, whch smply montors the level of a fcttous token bucket, represented by a sngle counter[15]. The spacer-controller used n ATM s also an example of shaper [21], [22]. Greedy shapers have a number of smple, physcal propertes; we focus here on one, the preservaton of arrval constrants. Consder a flow, ntally known to be ffsmooth, whch s passed nto a shaper n order to be made ff-smooth. Ths example s commonplace; for example, ff s a token bucket constrant, and ff s a constrant mposed by physcal lmtatons or by an upstream shaper (Fgure 1). A property of greedy shapers s that the shaper output stll satsfes the orgnal arrval curve constrant ff, n other words [23] what s done by shapng cannot be undone by shapng. Note that systems other than greedy shapers do not generally have ths property. The preservaton property was ntally obtaned by Cruz n [15] by an ad-hoc (complex) computaton, vald for the specfc case of leaky bucket controllers. In the sequel, we gve a general result and show how t s obtaned. Mn-Plus Convoluton. We now ntroduce a network calculus formalsm. We consder n ths secton only wde-

4 049.PDF 4 B H A I D J H = B B E?? I J H = E > O = I 4 4 H A I D = F J H = B B E? Fg. 1. Shapers preserve arrval constrants. sense ncreasng functons of tme that are 0 for t» 0. For any two such functons f (t) and g(t) we defne a thrd one (f Ω g)(t), called mn-plus convoluton, by (f Ω g)(t) = nf (f (s) +g(t s)) (1) 0»s»t Ths operaton s the analog of standard convoluton, f we replace the two standard operatons + and by mn and +; mn-plus algebra s the name of the calculus obtaned wth ths mappng (see [24], [5] or [6] for a general presentaton). The analogy bears some frut many propertes of standard convoluton, such as assocatvty and commutatvty, are also true here: (f Ω g) Ω h = f Ω (g Ω h) = f Ω g Ω h and f Ω g = g Ω f. We characterze a flow wth ts cumulatve functon R(t), defned as the number of bts observed from an arbtrary tme orgn up to tme t. Then, sayng that the flow s ff-smooth s equvalent to R» R Ω ff, whch s also equvalent to R = R Ω ff. Ths can be seen mmedately by applyng the defnton of mn-plus convoluton. Consder now a shaper, whch forces the flow nto an arrval curve ff. We assume that ff s sub-addtve, n other words, ff(s + t)» ff(s)+ff(t). Ths s not a restrcton, as any arrval curve constrant can be expressed wth a sub-addtve functon [14]. In addton, all concave arrval curves (such as the arrval curves presented above) are sub-addtve. I/O Characterzaton of Shapers. Call R Λ the output of the shaper. It must satsfy the constrants ρ RΛ» R (2) R Λ» R Λ Ω ff The former nequalty expresses that the output derves from the nput after bufferng; the latter expresses that t s ff-smooth. Any wde-sense ncreasng functon R Λ (t) that satsfes (2) s the output of some shaper, not necessarly greedy. It turns out that the system (2) s a classcal mnplus problem [25] and has one maxmum soluton, gven by R Λ = R Ω ff (3) Ths statement can be proven n a general mn-plus settng, but n ths partcular case, a drect proof s possble and holds n a few lnes ([6], Secton 1.5). The greedy shaper output s necessarly the maxmum soluton, whch establshes that (3) s true for the shaper output. The frst proof appeared n [17] and uses a dfferent network calculus method than presented here. Consequences. Ths establshes that shapers are mnplus lnear systems. We show now how ths mples the preservaton property mentoned above. The assocatvty of mn-plus convoluton can be used: R Λ Ω ff =(R Ω ff) Ω ff =(R Ω ff) Ω ff = R Ω ff = R Λ The last but one equalty s because the nput s ff-smooth and thus R Ω ff = R. Thus, ths establshes that R Λ = R Λ Ω ff as well, whch means that the output of the shaper s ff-smooth, as requred (of course t s also ff-smooth as well). Another consequence of the mn-plus representaton of shapers n Equaton (3) s that a concatenaton of I shapers n sequence wth curves ff ; = 1; :::; I s equvalent to a global shaper wth curve ff = ff 1 Ω ::: Ω ff I. If the curves ff are concave, then ff =mn 1»»I ff. Ths s commonly used to mplement shapers for concave pecewse lnear functons as the concatenaton of leaky bucket controllers. A strkng fact s that the order of the concatenaton does not play a role here. Packetzaton Effects. The theory presented n ths secton gnores packetzaton constrants, whch play a role when packets of a flow are of dfferent szes. Packetzaton effects are modeled wth the concept of packetzer, ntroduced by C.S. Chang [5]. A packetzer can be thought of as a devce that collects bts untl entre packets can be delvered. The results mentoned earler reman vald, as long as the arrval curves are concave and have a jump at the orgn at least as large as one maxmum packet sze [22]. Else, the nserton of a shaper weakens the arrval curve by one maxmum packet sze. B. IntServ and Servce Curves The Prncple of Reservatons. The IETF Integrated Servces (IntServ) archtecture supports dfferent reservaton prncples; we focus here on the guaranteed servce [20], whch provdes determnstc guarantees (statstcal guarantees are dscussed n Secton IV). IntServ uses admsson control, whch operates as follows. ffl In order to receve the guaranteed or controlled load servce, a flow must frst perform a reservaton durng a flow setup phase. ffl A flow must conform to an arrval curve of the form ff(t) =mn(m + Ct;rt + b) (T-SPEC). ffl All routers along the path accept f they are able to provde a servce guarantee and enough buffer for loss-free operaton. The servce guarantee s expressed durng the reservaton phase, usng the concept of servce curve, as explaned below.

5 049.PDF 5 Servce Curves, a Mn-Plus Approach. The reservaton phase assumes that all routers can export ther characterstcs usng a very smple model. The problem s that routers may mplement very dfferent schedulng strateges. Ths s solved wth the concept of servce curve. It was ntroduced by Parekh and Gallager [7] and Cruz [23] n a restrcted sense, then ndependently n ts fnal form by Agrawal, Chang, Cruz, Le Boudec, Okno and Rajan [19], [18], [17]. It s defned as follows. Consder a system S and a flow through S wth nput and output functon R and R Λ. Let f(t) be a non-negatve wde-sense ncreasng functon We say that S offers to the flow a servce curve f f and only f R Λ R Ω f (4) In practcal terms, t means that for any tme t, there exsts a tme s» t such that R Λ (t) R(s) +f(t s) (5) Ths defnton may seem obscure, but t turns out to be the rght abstracton. Frst, t captures well the classcal queung systems, but also apples to complex systems. Consder a queue that serves a flow wth a rate at least equal to c (for example, a generalzed processor sharng node [7]); such a node offers a servce curve equal to f(t) =ct (for t 0). More generally, a node that guarantees to serve at least f(t) bts durng any nterval of duraton t nsde a busy perod guarantees a servce curve equal to the functon f(t); n that case we say that we have a strct servce curve. In practce though, the concept of strct servce curve does not mean much for a complex system, because there are delay elements. Consder for example a system about whch we only know that the delay s bounded by some value T ; assume that the nput s a small but steady flow of data, at a rate ffl; the system s always n a busy perod; however, the output rate ffl can be arbtrarly small, thus the only strct servce curve we could express would be 0. In contrast, wth the defnton of servce curve gven above, ths system offers a servce curve f = ff T, defned by ff T (t) =+1 f t>tand ff T (t) =0f t» T. In some sense, the servce curve concept replaces the analyss by busy perod whch s commonplace n queung theory, but does not apply to complex systems. Second, the defnton supports concatenaton. Consder a tandem of two nodes, offerng servce curves f 1 and f 2, wth the output of the frst feedng the nput of the second. It follows mmedately from (4) and the assocatvty of mn-plus convoluton that the tandem, vewed as a sngle system, offers the servce curve f = f 1 Ω f 2. Thus, t s very easy to compute a servce curve for complex nodes. For example, the mn-plus convoluton of 1 f(t) = ct + and ff T (t) s equal to the so-called rate-latency functon f(t) = c(t T ) +, thus the concatenaton of a node wth guaranteed rate c and a node wth maxmum delay T offers a rate-latency servce curve. IntServ requres that all routers can be abstracted wth such a servce curve [26] (or equvalently, as a guaranteed rate node, see below). Thrd, the combnaton of arrval curve and servce curve supports the dervaton of the followng tght bounds. Let a system offer a servce curve f to a flow that s constraned by some arrval curve ff. Then the backlog for ths flow s bounded by the vertcal devaton v(ff; f) := sup [ff(s) f(s)] (6) s 0 If the node serves the bts of ths flow n FIFO order (an assumpton that s true n the IntServ context), then the delay s bounded by the horzontal devaton (Fgure 2) h(ff; f) := sup [nf fd 0such that f (t)» g(t + d)g] t 0 (7) For a flow wth arrval curve ff(t) = mn(ct+ M;rt+ b) L = > = D = > Fg. 2. Bounds on backlog and delay derved from arrval and servce curves. and a rate-latency servce curve f(t) = R(t T ) +, ths gves the backlog bound [12], [27] ;T b M v(ff; f) =b + r max C r and the delay bound h(ff; M f) = + b M (C R)+ C r + T (8) R End-to-end bounds. The above results can be combned to obtan the worst case end-to-end delay across an IntServ network. A flow that goes through a sequence of routers = 1; :::; I, each wth servce curve f (t) = R (t T ) +, sees the network as a system offerng the servce curve f = f 1 Ω ::: Ω f I. A drect computaton shows 1 We use the notaton m + =max(m; 0). >

6 049.PDF 6 P that f(t) =R(t T ) + wth R =mn R and T = T. Together wth the delay bound (8), ths s used by routers durng the reservaton setup phase, n order to determne f a reservaton should be accepted [6]. By computng the end-to-end servce curve as the mnplus convoluton of the servce curves of all nodes, t can also be establshed that the worst case delay over a concatenaton of nodes s less than the sum of the worst case delay at every node. A smlar statement s known under the term pay bursts only once, whch says that the mpact of the burstness parameter b n the arrval curve ff(t) =mn(ct + M;rt + b) of a flow does not accumulate over the number of nodes traversed by the flow, but, n contrast, occurs only once. Ths s a drect applcaton of the results above ([6] Secton 1.4.3). Re-shapng s for Free. Another property whch can be establshed wth ths abstract settng s re-shapng s for free. Re-shapng s often ntroduced nsde a network, or at network boundares, n order to control the accumulaton of burstness that may otherwse occur. Assume now that a flow, constraned by an arrval curve ff, s nput to a tandem of networks, each offerng servce curves f 1, f 2 (Fgure 3). Assume a greedy shaper, wth curve ff ff s nserted between the two systems. The condton means that the re-shaper enforces some or all of the ntal curve constrant. It follows drectly from (3) B H A I D J H = B B E? I D = F A H > I > 4 4 Fg. 3. Reshapng example. that the re-shaper offers a servce curve equal to ff. Thus, the worst case delay for the combnaton wth re-shaper s d 0 = h(ff; f 1 Ω ff Ω f 2 ) whereas for the orgnal combnaton t s d = h(ff; f 1 Ω f 2 ). Now mn-plus convoluton s assocatve and commutatve, thus d 0 = h(ff; ff Ωf 1 Ωf 2 ); we nterprete ths as the worst case delay for a new combnaton where the re-shaper s put mmedately before the frst network, nstead of between the two. But n that case, the nput traffc s ff-smooth, thus also ff-smooth, and the re-shaper never delays any bt of data. Thus we can remove the re-shaper from the new combnaton and d 0 = d. We have shown n these few lnes that the delay bound for the system wthout shaper s also vald for the system wth shaper. In other words, nodes may re-shape flows wthout exportng that nformaton. Other Aspects. The concepts of servce and arrval curves have been used by Cruz and Sarowan [28], [29], Georgads, Guérn, Pers and Rajan [12] to desgn schedulers that optmze the combnaton of delay guarantees, buffer and bt rate requrements, and go beyond the ntal desgn deas of Kalmanek, Kanaka and Restrck [10] and H. Zhang and Ferrar [11]. Some of these schedulers are desgned to have have servce curves that are not ratelatency, therefore, ther propertes are not well exploted wthn the IntServ framework. All computatons so far were done wth the assumpton that the systems are empty at tme 0. Ths s vald for statc reservatons, but not for dynamc reservatons, whch are supported by IntServ and ATM-ABR. The modfcatons to the calculus presented above were found by Gordano et al n [30]. Delay and Delay Jtter. For playback operatons, only the varable part of delay, called delay jtter, s mportant (Secton II-D). In contrast, for nteractve servces, the total delay s also of mportance. Thus, both delays must be accounted for; ths can be done as follows. If the latency terms of servce curves do not ncorporate constant delays, then delay bounds such as (8) gve the delay jtter; a bound on total delay s then obtaned by addng to t the sum of all constant delays. Guaranteed Rate Servers, a Max-Plus Approach. The servce curve concept defned above can be approached from the dual pont of vew, whch conssts n studyng the packet arrval and departure tmes nstead of the functons R(t) (whch count the bts arrved up to tme t). Ths latter approach leads to max-plus algebra (whch has the same propertes as mn-plus), s often more approprate to account for detals due to varable packet szes, but works well only when the servce curves are of the ratelatency type. It s used n Secton III wth the core stateles approach to obtan detaled relatons between packet departure tmes across a network. It also useful when nodes cannot be assumed to be FIFO per flow, as may be the case wth DffServ (Secton II-C). We now descrbe ths approach here and how t relates to the mn-plus approach. A node s sad to be of the Guaranteed Rate (GR) type [9] (also called Rate-Latency server), wth rate r and latency e, f the departure tme d n of the nth packet, counted n order of arrval, satsfes d n» f n + e (9) where f n (vrtual fnsh tme) s gven by the recurson (a n s the arrval tme, l n the length n bts, of packet n): ρ f0 =0 f n = max [a n ;f n 1 ]+ ln r for n 1 (10) GR s an alternatve way of descrbng the rate-latency servce curve property. More precsely, a GR node wth

7 049.PDF 7 rate r and latency e can be decomposed as a node offerng the rate-latency servce curve f(t) =r(t e) +, followed by a packetzer [6]. Note that addng a packetzer weakens the servce curve property by one maxmum packet sze, but does not ncrease the packet delay. Conversely, but only for a FIFO node, the rate-latency servce curve f(t) = r(t e) + mples GR wth rate r and latency e. It follows from ths equvalence that the delay bounds n Equaton (8) hold for a FIFO GR node; t s shown n [31] that t also holds for non FIFO nodes. Specfcally, the packet delay for a flow that s ff-smooth s bounded by» ff(t) sup t>0 r t + e (11) For GR nodes that are FIFO per flow, the concatenaton result obtaned wth the servce curve approach apples. Specfcally, the concatenaton of I GR nodes (that are FIFO per flow) wth rates r and latences P e s GR wth rate r = mn r and latency e = e +(I 1) Lmax, r where L max s the maxmum packet sze for the flow. The term (I 1) Lmax s due to packetzers. For GR nodes that r are not FIFO per flow, ths result s no longer true [31]. The recurson n (10) can be solved easly, usng the propertes of max-plus algebra. We obtan that GR s equvalent to sayng that for all n there s some k 2 fj +1; :::; ng such that d n» a l k + ::: + l n k + + e (12) r whch s the dual of (5) wth f(t) =r(t e) + [5]. C. DffServ, Aggregate Schedulng and Adaptve Servce Curves The IETF Dfferentated Servces (DffServ) archtecture dffers from the IntServ archtecture n that flows are treated n an aggregate manner nsde a network. DffServ s a framework whch supports many servces; we focus here on Expedted Forwardng (EF) 2. Roughly speakng, EF can be thought of as a prorty servce. Packets marked as EF (namely, wth the PHB feld n the IP header set to EF ) receve a low delay and practcally loss-free servce. Ths s typcally used for crcut emulaton or hgh qualty vdeoconferencng. Expedted Forwardng and Intuton Behnd. At every router, all EF packets are vewed as one sngle aggregate. In contrast, at network access ponts, ndvdual flows of EF packets (called mcroflows ) are shaped 2 DffServ makes a dstncton between servce and Per-Hop Behavour (PHB). In classcal, OSI, termnology, the former means the servce provded by a network, and the latter s the servce provded by a network element. one-by-one, accordng to an arrval curve smlar to the T- SPEC defned n Secton II-B. As wth IntServ, the arrval curves constrants put on mcro-flows are expected to support hard end-to-end qualty of servce guarantees. Unlke IntServ though, mcroflows are not scheduled separately. The ntuton s that, as long as the ntensty of EF traffc s small, EF queues reman empty and delays are small delays reman small. More precsely, the orgnal descrpton of EF n [32] was mplctly assumng that sources are, n the worst case, perodc (ths s now dropped from the formal defnton of EF). Then, f the EF traffc ntensty s small, t s plausble that the delay varaton for packets nsde one mcroflow s less than the perod of the source. As a result, packets from the same mcroflow would never catch up and the servce would be smple to analyze and use. Chlamtac et al [33], [34], [35] have shown that ths ntuton does hold, but n an ATM context, under the assumpton that sources satsfy the source rate condtons, whch requre that the perod of a source (n tme slots) be at least as large as ts route nterference number. The route nterference number s the number of tmes when the path of a gven source merges wth that of other sources. However, t s dffcult to transpose ths result from ATM to Internet, frst because of varable packet szes, and second because the FIFO assumpton may be too strct. PSRG, Formal Defnton of EF. Thus, the current defnton of EF s not based on ths result. In contrast, t s based on an abstract node model, nspred by GPS [7], called Packet Scale Rate Guarantee, from whch a delay bound can be obtaned. Ths s analog to IntServ assumng that every router can be modeled a GR node, but wth some dfferences, to whch we come back later n ths secton. A node s sad to offer to a flow of packets (here: the EF aggregate) the packet scale rate guarantee [36] wth rate r and latency e f the departure tme d n of the nth packet, counted n order of arrval, satsfes (9) where f n s gven by the followng recurson: ρ f0 =0 f n = max [a n ; mn (d n 1 ;f n 1 )] + ln r for n 1 (13) (a n s the arrval tme, l n the length n bts, of packet n). A non-preemptve prorty scheduler wth rate r satsfes the defnton, wth e r equal to the maxmum sze of low prorty packets; as explaned later PSRG apples to more complex nodes, possbly non-fifo. PSRG dffers from GR defned n Secton II-B by the d n 1 term n (13). It follows that PSRG s stronger than GR,.e., any PSRG node satsfes the GR property wth the same parameters. We wll use these propertes now to obtan an end-to-end

8 049.PDF 8 delay bound. End-to-end Delay Bound for EF. Charny and Le Boudec have obtaned n [37] a bound on delay varaton that s vald for EF, as follows. Assume that mcroflow s constraned by the arrval curve ρ t + ff at the network access. Insde the network, EF mcroflows are not shaped. At node m, the EF aggregate s served accordng to the packet scale rate guarantee, wth rate r m and latency e m (Fgure 4)). Call H a bound on the number of hops used H I E? H B M E H E I E - = C C H A C = J A = A Fg. 4. Model of EF network H = J A H = J A? O A by any flow (ths s typcally 10 or less, and s much less than the number of nodes). Call D a bound on the queung delay undergone by a flow at any sngle node (assumng a fnte bound exsts, whch s shown n [37]), and consder some arbtrary node m. The data that feeds node m has undergone a varable delay n the range [0; (H 1)D], thus an arrval curve for the EF aggregate at node m s νr m (t +(H 1)D) +r m f, where ν (maxmum utlzaton factor) s a bound 3 on 1 r m P3m ρ and f (maxmum packet delay varaton) s a bound on 1 r m P3m ff ). By applcaton of (11), the delay seen by any packet s bounded by D» e + f +(H 1)Dν; thus f the utlzaton factor ν s less than 1, we have the followng bound on delay H 1 at one hop e + f D» (14) 1 (H 1)ν The bound can be mproved f we have more nformaton about the peak rate at whch the EF aggregate may arrve at the node [37]. The bound s vald only for small utlzaton factors; t explodes at ν> 1, whch does not mean that the worst H 1 case delay does grow to nfnty [38]. As far as we know, ths ssue s stll unresolved ([6] Secton 6.3). However, t s shown n [36] that any better bound must make more assumptons about the network than s sutable n the EF framework. See also Secton IV-B for statstcal bounds that are vald under the same settng. PSRG versus Servce Curve Delay from Backlog. Why do we need for EF a defnton such as Packet Scale 3 the notaton 3 m means that node m s on the path of mcroflow Rate Guarantee, nstead of usng for example the servce curve or GR characterzaton of IntServ? Indeed, a GR defnton mght be a vald node abstracton for EF, snce the delay bound mentoned above used only the GR property. The reason for choosng PSRG nstead s based on the desre to have a delay-from-backlog bound, whch s used n cases wth statstcal multplexng. Indeed, GR (and servce curve guarantee) may gve brth to the lazy scheduler syndrom, whch conssts n that t s perfectly vald for a GR scheduler to serve the frst k packets of a flow faster than necessary, and then take advantage of ths advance to delay subsequent packets for an arbtrarly long amount of tme [13]. As a result, t s not possble to derve a bound on the delay undergone by a packet from the backlog t sees upon arrval, unlke the case of an deal GPS scheduler [6]. In contrast, wth PSRG, the effect of the d n 1 term s that, f a packet s served earler than ts deadlne, then the deadlne of all subsequent packets s reduced accordngly. The followng delay-from-backlog bound s shown n [31]: for a packet served n a PSRG node, that sees a backlog equal to Q upon arrval, the delay s bounded by Q r + e. Wth IntServ, t s natural to assume that a node serves the packets nsde a flow n FIFO order. Ths per-flow FIFO assumpton cannot usually be made wth DffServ. Indeed, wth DffServ, a scheduler sees an entre EF aggregate as one flow. Snce the EF aggregate usually enters a router va more than one nput ports, the delay though the router nternal may vary a lot across packets, and as a result, the node may not be globally FIFO. All bounds on delay mentoned above are true for PSRG nodes, even non FIFO [31]. For the specal case of FIFO nodes, PSRG s equvalent to the adaptve servce curve property, a varant of the servce curve property defned by Agrawal, Cruz, Okno and Rajan [39]. Concatenaton rules based on mn-plus convoluton apply here also, but they do not extend to non-fifo nodes. Mn-Max Algebra. We have explaned n Secton II-B how servce curves and the IntServ framework are based on mn-plus and max-plus algebras. For DffServ, mnmax algebra has to be nvoked also to derve propertes of PSRG, as we explan now. The teratve defnton of f n n (13) can be re-wrtten as a mn-max equaton: f n =» a n + l n r _» d n 1 + l n r ^ f n 1 + l n r (15) Now mn-max algebra enjoys the same propertes as mnplus algebra; ths s used n [31] to show that PSRG s equvalent to sayng that for all n and all 0» j» n 1

9 049.PDF 9 ether d n» e + d l j+1 + ::: + l n j + r or there s some k 2fj +1; :::; ng such that (16) d n» e + a l k + ::: + l n k + (17) r Equatons (16) and (17) consttute a characterzaton of PSRG wthout the vrtual fnsh tmes; they are the key relatons from whch all propertes of PSRG mentoned earler n ths secton [31] are derved. Low Jtter Alternatves to EF. A number of research proposals am to obtan better bounds than (14), at the expense of more elaborate schedulers, whle preservng aggregate schedulng. A frst proposal uses the concept of damper [40], [41], whch has the effect of compensatng delay varaton at one node n the next downstream node. Wth dampers appled to the EF aggregate, the end-to-end delay bound becomes much smaller and s fnte for all utlzaton factors less than 1 [6]. A smpler and more powerful alternatve s proposed by Z.-L. Zhang et al under the name of Statc Earlest Tme Frst (SETF) [42]. Assume that packets are stamped wth ther tme of arrval at the network access, and that they are served wthn the EF aggregate at one node n order of tme stamps. More precsely, we assume that nodes offer a GR guarantee to the EF aggregate, as defned by (10) or (12), but that packets are numbered n order of ther arrval at the network access (not at ths node). Then the analyss that led to the end-to-end delay bound (14) can be modfed as follows. Call D h a bound on the end-to-end delay after h hops, h» H 1. Consder a tagged packet, wth label n, and call d h ts delay n h hops. Consder the node m that s the hth hop for ths packet. Apply (12): there s some label k» n such that d n» e + a l k + ::: + l n k + (18) r where a j and d j are the arrval and departure tmes at node m of the packet labeled j, and l j ts length n bts. Now packets k to n must have arrved at the network access before a n d h and after a m D H 1. Thus l k + ::: + l n» ff(a n a m d h + D H 1 ) where ff s an arrval curve at network access for the traffc that wll flow through node m. Wehaveff(t)» r m (νt + f ). By (11), the delay d n a n for our tagged packet s bounded by e+sup t 0» ff(t dh + D H 1 ) r m t = e+f +ν(d H 1 d h ) thus d h+1» d h + e + f + ν(d H 1 d h ) The above nequaton can be solved teratvely for d h as a functon of D H 1 ; then take h = H 1 and assume the tagged packet s one that acheves the worst case h-hop delay, thus D H 1 = d H 1 whch gves an nequalty for D H 1 ; last, take h = H and obtan the end-to-end delay bound 1 (1 ν)h D H» (e + f ) (19) ν(1 ν) H 1 The bound s fnte for all values of the utlzaton factor ν < 1, unlke the end-to-end bound n (14). Note that for small values of ν, the two bounds are equvalent. We have assumed here nfnte precson about the arrval tme stamped n every packet. In practce, the tmestamp s wrtten wth some fnte precson; n that case, Zhang [42] fnds a bound whch les between (14) and (19) (at the lmt, wth null precson, the bound s exactly (14)). D. Playback Delay for Pre-Recorded Vdeo Consder a clent readng a pre-recorded vdeo fle from a server across a network. Assume the network guarantees a bound on varable delay T but requres the flow to be ff-smooth (these assumptons correspond to sendng the vdeo over EF; a smlar example s studed n [43] but wth IntServ nstead of DffServ). On the clent sde, the flow s processed wth hgh prorty before beng sent to the dsplay; ths s modeled by assumng that the flow receves a rate-latency servce curve, wth a rate equal to the processng rate, and a latency accountng for the maxmum nterrupton [44]. It follows that the combnaton of network delay and processor delay at the clent sde can be modeled wth a servce curve, say f(t). Before beng sent nto the network, the flow s processed by a smoother n order to be made conformant to the arrval curve constrant ff; the smoother s smlar to the shaper descrbed n Secton II-A, except that snce the fle s pre-recorded, t may send bts n advance of ther natural readng tme (n other words, t does not have to be causal). Once processed at destnaton, the flow s played back nto a decodng buffer whch has to re-create the orgnal tmng of the flow. We assume that ths s done by delayng the frst packet of data for some amount D called the playback delay. If the arrval curve constrant s very large, then there s no need for smoothng and the decodng buffer need only compensate for the delay jtter due to network and clent processor; here, t s necessary and suffcent for D to be an upper bound on delay jtter. Otherwse, n the general case, smoothng s necessary and the decodng buffer needs to compensate for both delay jt-

10 049.PDF 10 8 E@ A I A H L A H 5 J D A H A J M H 2 H? A I I H 2 = O > =? * K B B A H 8 E@ EI F = O 4 J 4 \J 4 J, I J > J 4 J Fg. 5. Playng a vdeo fle over a network. ter and the tmng dfference due to smoothng (Fgure 5). Call R(t) the number of bts of the orgnal flow, when t s read n real tme (the rate of whch s not assumed to be constant). A lazy smoother wll smply delay R(t), lke a shaper would do; a more aggressve smoother may antcpate bursts n R(t) and thus obtan a smaller playback delay (e.g., usng prefetch smoothng [45]). We are nterested n fndng the mnmum playback delay D that can be acheved, gven ff and f, among all smoothng strateges. Rexford and Towsley [46] fnd the soluton when the network servce s constant bt rate, whch corresponds to ff(t) = f(t) = rt for some r. Le Boudec and Verscheure fnd the soluton n the general case, by modelng the problem wth a set of nequaltes and apply the same method mentoned wth Equaton (2) n Secton (II-A). They fnd n [43] that the mnmum playback delay s gven by the horzontal devaton: D = h(r; ff Ω f) (20) Fgure 6 llustrates the formula. [43] also fnd an optmal smoother output (one that acheves the mnmum playback delay D) and obtan an explct representaton usng mn-plus operatons. A number of applcatons follow from ths representaton. Frst, the optmal smoother s not a shaper; ndeed, a shaper smoothes out bursts n R(t) once they occur, whereas n most cases, the optmal smoother has to pre-fetch the bursts. Second, the strategy that would consst n equalzng delay jtter before presentng data to the decoder buffer s not optmal because of a pay bursts only once syndrom. Thrd, the optmal smoother output s ant-causal, n other words, the optmal tme at whch frame n should be sent depends only on the szes of frames m n. Thus the producton of small playback delays s based on the ablty to look-ahead n the stored vdeo fle. Ths s used n [47] to construct the encodng R(t) whch mnmzes dstorton, gven ff;f and a target playback delay D. Extenson of optmal vdeo smoothng to a multcast envronment (wth applcaton-level QoS mechansms, see Secton VII) can be found n [48]. & $ " % $ # "!! " 4 J I > J, "! # I! " Fg. 6. Computaton of mnmum playback delay for an MPEG sequence. The top left box shows r(t), the number of bytes for the tth frame. R(t) = Pt s=1 r(t) s the correspondng cumulatve functon. The mnmum playback delay, shown by the arrow, s the horzontal devaton between R(t) and (ff Ω f)(t). III. ARCHITECTURES FOR SCALABLE QOS SUPPORT Scalablty s a key ssue n the desgn of Internet QoS provsonng archtectures, n both data plane and control plane. In network data plane approprate control state nformaton s needed for per-packet processng such as schedulng and queue management at core routers so as to support dfferentated packet treatment and provde QoS guarantees. Granularty of such control state nformaton and how t s obtaned and mantaned determne the complexty of QoS state management n data plane, and thus ts scalablty. Lkewse, approprate control state nformaton s also needed n network control plane for resource reservaton and QoS provsonng. Complexty and scalablty of control plane operatons depend crtcally on the granularty and tme scale of such control state nformaton. In addressng the scalablty ssues n data plane, classbased aggregate schedulng s an mportant approach, as s adopted n DffServ. However, as we have seen earler, ths ncreased scalablty s acheved at the expense of reduced performance, at least n terms of worst-case end-toend delay performance. Another attractve approach s the dynamc packet state approach [49], where control state nformaton necessary for packet schedulng s carred n packet headers; core routers perform smple per-packet

11 049.PDF 11 Edge condtoner Core router Packet state Edge condtoner Unregulated traffc Regulated traffc of flow j of flow j Arrval tmes of unregulated traffc r j Traffc after edge condtoner Network core j r tme (a) A conceptual network model. (b) Edge condtoner and ts effect Fg. 7. Edge condtonng n the vrtual tme reference system. state update. As a result, usng the dynamc packet state approach, per-flow end-to-end QoS guarantees smlar to those provded by IntServ can be supported wthout perflow management at core routers. In secton III-A we wll provde an overvew of the vrtual tme reference system a unfyng schedulng framework to provde scalable support for guaranteed servces based on the dynamc packet state approach [50]. To reduce the complexty and thus enhance the scalablty of control plane operatons, a number of new approaches have been developed. They can be roughly categorzed nto three general approaches: lghtweght sgnalng, end-pont/edge admsson control and centralzed bandwdth broker. In secton III-B we wll brefly descrbe some representatve examples of these three dfferent approaches. A. Dynamc Packet State and Vrtual Tme Reference System The noton of dynamc packet state was frst proposed by Stoca and Zhang [51], [52], [49], where control state nformaton s carred n data packets and updated at core routers for schedulng purposes. In [52] Stoca and Zhang demonstrated that a core stateless verson of Jtter Vrtual Clock (Jtter-VC) can be mplemented usng the dynamc packet state technque to attan the same end-to-end delay bound wthout per-flow management. Ther scheme was generalzed by Zhang et al n [50], where usng the dynamc packet state approach, a general core stateless framework the vrtual tme reference system (VTRS) was developed to provde scalable support for guaranteed servces. The key construct n the vrtual tme reference system s the noton of packet vrtual tme stamps, whch are referenced and updated as packets traverse each core router. As we wll see shortly, the vrtual tme stamps assocated wth packets of a flow form the thread that weaves together the per-hop behavors of core routers along the path of the flow to provde QoS guarantees for the flow. A key property of packet vrtual tme stamps s that they can be computed usng solely the packet state carred by packets (plus a couple of fxed parameters assocated wth core routers). In ths sense, the vrtual tme reference system s core stateless, as no per-flow state s needed at core routers for computng packet vrtual tme stamps. Conceptually, the vrtual tme reference system conssts of three logcal components: packet state carred by packets, edge traffc condtonng at the network edge (see Fgure 7), and per-hop vrtual tme reference/update mechansm at core routers (see Fgure 8). These three components are brefly descrbed below. Edge Traffc Condtonng. Edge traffc condtonng plays a key role n the VTRS, as t ensures that packets ofaflow 4 wll never be njected nto the network core at a rate exceedng ts reserved rate (see Fgure 7(b)). Formally, for a flow j wth a reserved rate r j, the nter-arrval tme of two consecutve packets of the flow at the frst hop core router s such that ^a j;k+1 1 ^a j;k 1 Lj;k+1, where ^a j;k 1 denotes the arrval tme of the kth packet p j;k of flow j at the network core, L j;k the sze of packet p j;k, and r j the reserved rate of flow j. Ths s equvalent to passng the flow through a shaper wth ff(t) = r j t, followed by a packetzer. Packet State. After gong through the edge condtoner at the network edge, packets enterng the network core carry, n ther packet headers, certan packet state nformaton that s ntalzed and nserted at the network edge. The packet state carred by the kth packet p j;k ofaflowj contans three types of nformaton: 1) QoS reservaton (a rate-delay parameter par hr j ;d j ) of the flow; 2) the vr- 4 Here a flow can be ether an ndvdual user flow, or an aggregate traffc flow of multple user flows, defned n any approprate fashon. r j