PAPER Providing Scalable Support for Multiple QoS Guarantees: Architecture and Mechanisms

Transcription

1 2830 IEICE TRANS. COMMUN., VOL.E84 B, NO.10 OCTOBER 2001 PAPER Provdng Scalable Support for Multple QoS Guarantees: Archtecture and Mechansms Ywe Thomas HOU, Regular Member, Zhenha DUAN, Zh-L ZHANG, Nonmembers, and Takafum CHUJO, Regular Member SUMMARY The IETF Dfferentated Servces (DffServ) framework acheves scalablty by (1) aggregatng traffc flows wth coarse gran QoS on the data plane, and (2) allocatng network resources wth a bandwdth broker (BB) on the control plane. However, there are many ssues that need to be addressed under such framework. Frst, t has been shown that the concatenaton of strct prorty (SP) scheduler of class-based queues (CBQ) can cause delay jtter unbounded under certan utlzaton, whch s not acceptable to support the premum servce (PS). Furthermore, t s not clear how such a DffServ network can support traffc flows requrng the guaranteed servce (GS), whch s a desrable feature of the future Internet. Ths paper presents archtecture and mechansms to support multple QoS under the DffServ paradgm. On the data plane, we present a node archtecture based on the vrtual tme reference system (VTRS). The key buldng block of our node archtecture s the core-stateless vrtual clock (CSVC) schedulng algorthm, whch, n terms of provdng delay guarantee, has the same expressve power as a stateful weghted far queueng (WFQ) scheduler. Wth the CSVC scheduler as our buldng block, we desgn a node archtecture that s capable of supportng ntegrated transport of the GS, the PS, the assured servce (AS), and the tradtonal best effort (BE) servce. On the control plane, we present a BB archtecture to provde flexble resource allocaton and QoS provsonng. Smulaton results demonstrate that our archtecture and mechansms can provde scalable and flexble transport of ntegrated traffc of the GS, the PS, the AS, and the BE servces. key words: qualty of servce, dfferentated servces, schedulng, bandwdth broker, scalablty 1. Introducton The ablty to provde end-to-end guaranteed servces (GS) [22] (e.g., guaranteed delay or bandwdth) for networked applcatonssa desrable feature of the future Internet. To enable such servces, Qualty-of-Servce (QoS) support from both the network data plane (e.g., packet schedulng) and the control plane (e.g., admsson control and resource reservaton) s needed. For example, under the IETF Integrated Servces(IntServ) archtecture, schedulng algorthms such as weghted far queueng (WFQ) [10], [19], [20] were developed to support the GS. Furthermore, a sgnalng and reservaton protocol, RSVP [6], [26], for settng up end-to-end Manuscrpt receved September 28, Manuscrpt revsed March 12, The authors are wth Fujtsu Laboratores of Amerca, Sunnyvale, CA, USA. The authors are wth Unverstyof Mnnesota, Department of Computer Scence and Engneerng, Mnneapols, MN, USA. QoS reservaton along a flow s path was also proposed and standardzed. However, due to ts need for performng per-flow management at core routers, the scalablty of the IntServ archtecture hasbeen questoned. To address the ssue of scalablty, the IETF has ntroduced the Dfferentated Servces(DffServ) model [3], whch acheves scalablty by offerng servces for an aggregate traffc rather than on a per-flow bass. Furthermore, the DffServ model pushes much complexty out of the core of the network nto edge routers, whch processes smaller volume of traffc and fewer number of flows. On the data plane, smple per-hop behavors (PHBs) (e.g., expedted forwardng (EF) [16] and assured forwardng (AF) [14]) have been defned to treat traffc aggregate and to provde dfferentaton n packet forwardng. On the control plane, a centralzed bandwdth broker (BB) [18] wasntroduced to perform resource management and allocaton wthn a DffServ doman (ntra-doman) and to mantan servce level agreement (SLA) between DffServ domans (nter-doman). Under such DffServ paradgm, two new servces, namely, the premum servce (PS) and the assured servce (AS) have been proposed [18] to provde coarsegraned end-to-end QoS guaranteesover the Internet. The PS sexpected to offer a guaranteed rate, low delay jtter packet delvery, whle the AS sexpected to offer a sustanable rate guarantee for a traffc flow. It has been suggested [18] that a network of routers employng a smple class-based strct prorty (SP) schedulng between the PS and the AS queues(wth FIFO for each queue) may acheve end-to-end support for the PS and the AS through a DffServ network doman. Several ssues have been rased regardng such class-based SP node archtecture n a DffServ network. Frst, t has been shown recently [2], [8] that the delay jtter under a concatenaton of class-based SP schedulerscan be unbounded over a certan utlzaton, whch meansthat the QoS requrement for the PS cannot always be supported under such node archtecture. Second, t s not clear how such class-based SP node archtecture can support end-to-end (ether per-flow or Here a flow can be ether an ndvdual user flow, or an aggregate traffc flow of multple user flows, defned n any approprate fashon.

2 HOU et al.: PROVIDING SCALABLE SUPPORT FOR MULTIPLE QOS GUARANTEES 2831 aggregate) GS [22], whch sa desrable feature of the future Internet. The purpose of ths paper s to desgn a node archtecture under the DffServ paradgm to provde scalable and ntegrated transport of the GS, the PS, the AS, and the tradtonal best effort (BE) servces. More specfcally, we want to acheve the followng three desgn objectves. Objectve 1: Multple QoS Guarantees. Our network should be capable of smultaneously transportng the GS, the PS, the AS, and the BE servces whle meetng the QoS requrementsof each servce. More specfcally, (1) For the GS, each flow specfes ts traffc behavor through a traffc profle (e.g., (σ, ρ, P, L max )) and a servce requrement (e.g., delay requrement) [22]. The network decdeswhether to admt or reject the call through an admsson control procedure. If the GS flow sadmtted nto the network, then the end-to-end delay bound must never be volated and no packet shall be lost for thsflow, aslong asthe source straffc conforms to tstraffc profle. (2) For the PS, each flow specfes tstraffc profle and servce profle through a peak rate requrement [18]. The network decdeswhether to admt or reject the call through an admsson control procedure. At the network edge, each admtted PS flow s shaped accordng to ts peak rate requrement. An admtted PS flow should experence low delay jtter and low loss when t traverses the network. (3) For the AS, each flow specfes ts traffc profle and servce profle through a sustanable rate requrement. The network decdeswhether to admt or reject the call through the admsson control procedure. At the network edge, an admtted AS flow s marked accordng to ts sustanable rate. Packets spaced accordng to ts sustanable rate wll have ther AS bt marked; packetsexceedng the sustanable rate wll be marked as the BE servce. Unlke the PS, edge shapng s not employed for the AS. (4) For the BE servce, there s no specfc traffc profle and QoS servce requrement for each flow. Each flow can enter the network freely and share any remanng network resources. Objectve 2: Scalablty n Network Core (Core- Stateless). In consstent wth the IETF DffServ model, we dentfy all the routerswthn a DffServ doman and dstngush them between edge and core routers. To support the GS, the PS, and the AS, we allow edge routers to mantan per-flow state. That s, edge routers perform per-flow traffc classfcaton and condtonng (shapng and markng). Snce each edge router typcally processes smaller volume of traffc and fewer number of flowsascompared wth a core node, scalablty s not a concern at an edge router. On the other hand, we requre that core routersmantan no per-flow state, e.g., per-flow reservaton state or perflow schedulng state. Such scalablty requrement on the core nodes s also called core-stateless approach [15], [23]. Objectve 3: Decouple QoS Control from Core Routers for Flexble Resource Allocaton and QoS Provsonng. We am to decouple the QoS control plane from the core routersand use a centralzed BB to control and manage doman-wde QoS provsonng. As we shall see n later sectons of ths paper, there are many sgnfcant advantages for decouplng QoS control plane from the data plane. One advantage that senabled by such decouplng sthat new network management polcescan be easly mplemented, elmnatng the addtonal complexty (hardware/software upgrade) on the core routersasotherwse would ncur. That s, all the polcy decson and ts enforcement can be performed on the control plane usng the BB wthout any hardware/software change on the data plane. Thspaper presentsan archtecture to meet the above three desgn objectves. Our node archtecture sbased on the vrtual tme reference system (VTRS), whch hasbeen recently ntroduced by Zhang et al. [27] asa unfyng schedulng framework for scalable support for the GS. Under the VTRS, a core-stateless scheduler, called core-stateless vrtual clock (CSVC) hasbeen ntroduced. The CSVC scheduler has the same expressve power n provdng delay and rate guarantee as a stateful WFQ scheduler, albet t does not mantan any reservaton or schedulng states n the router. The archtecture and mechansms presented n ths paper, whch coversboth data plane and control plane, bulds upon the VTRS/CSVC and amsto acheve the three desgn objectves lsted earler. In ths sense, ths paper sa sequel to [27]. The remander of thspaper sorganzed asfollows. In Sect. 2, we frst gve an overvew of the VTRS and the CSVC scheduler. Then we present our node archtecture (edge and core) on the data plane for ntegrated transport of the GS, the PS, the AS, and the BE servces. Secton 3 presents a BB archtecture and admsson control algorthms on the control plane. In Sect. 4, we dscuss the performance and complexty of our archtecture. Secton 5 presents smulaton results to demonstrate the performance of our node archtecture n provdng QoS guaranteesto each type of servces. In Sect. 6, we dscuss related work. Secton 7 concludesthspaper. 2. A Core-Stateless Archtecture wth Multple QoS Support Ths secton presents our core-stateless node archtecture on the data plane to support the GS, the PS, the AS, and the BE servces. We organze ths secton as follows. Secton 2.1 presents the essental background on the VTRS and ntroducesthe CSVC scheduler. In Sect. 2.2, we propose a node archtecture whch bulds upon the VTRS/CSVC, for ntegrated transport of the GS, the PS, the AS, and the BE servces n a DffServ network doman.

3 2832 IEICE TRANS. COMMUN., VOL.E84 B, NO.10 OCTOBER 2001 Fg. 2 Edge condtonng and ts effect. Fg. 1 A network doman where the VTRS s deployed. 2.1 Vrtual Tme Reference System: A Background The VTRS [27] wasdeveloped asa unfyng schedulng framework to provde scalable support of the GS. The key construct n the VTRS s the noton of packet vrtual tme stamps, whch, aspart of the packet state, are referenced and updated aspacketstraverse each core router. As we wll see shortly, the vrtual tme stamps assocated wth the packets of a flow form a thread whch weaves together the per-hop behavors of core routersalong the path of the flow to provde the QoS guaranteesfor the flow. A key property of packet vrtual tme stamps s that they can be computed usng solely the packet state carred by packets (plus a couple of fxed parameters assocated wth core routers). In ths sense, the VTRS s core-stateless, asno per-flow state s needed at core routers for computng packet vrtual tme stamps. The dea of vrtual tme has been used extensvely n the desgn of packet schedulng algorthms such as the VC [25] and the WFQ [10], [19], [20] schedulers. The noton of vrtual tme defned n these contexts s used to emulate an deal schedulng system and s defned local to each scheduler. Computaton of the vrtual tme functon requres per-flow nformaton to be mantaned at each node. In contrast, under the VTRS, the noton of vrtual tme emboded by packet vrtual tme stamps can be vewed, n some sense, as global to an entre doman (see Fg. 1). Its computaton s core-stateless, relyng only on the packet state carred by packets. Conceptually, the VTRS conssts of three logcal components: packet state carred by packets, edge traffc condtonng at the network edge (see Fg. 2), and per-hop vrtual tme reference/update mechansm at core routers (see Fg. 3). These three components are brefly descrbed below. 1. Edge Traffc Condtonng. Edge traffc condtonng playsa key role n the VTRS, ast ensures that the packetsof a flow wll never be njected nto the network core at a rate exceedng ts reserved rate (see Fg. 2). Formally, for a flow j wth a reserved rate r j, Fg. 3 Per-hop behavor and operatons at a core node under the VTRS. the nter-arrval tme of two consecutve packets of the flow at the frst hop core router s such that â j,k+1 1 â j,k 1 Lj,k+1 r j, (1) where â j,k 1 denotesthe arrval tme of the kth packet p j,k of flow j at the frst network core router, L j,k the sze of packet p j,k, and r j the reserved rate of flow j. 2. Packet State. After gong through the edge condtoner at the network edge, packetsenterng the network core carry n ther packet headerscertan packet state nformaton that s ntalzed and nserted at the network edge. The packet state carred by the kth packet p j,k of a flow j contansthree typesof nformaton: (1) QoS reservaton (.e, the reserved rate r j )of Note that n order to model non-preemptve, non-cutthrough network system, throughout the paper we adopt the followng conventon: a packet s consdered to have arrved at a server onlywhen ts last bt has been receved, and t to have departed the server onlywhen ts last bt has been servced. In addton, we assume that the edge condtoner and the frst-hop router (.e., the frst core router) are co-located, and thus the propagaton delayfrom the edge condtoner to the frst core router s neglgble.

4 HOU et al.: PROVIDING SCALABLE SUPPORT FOR MULTIPLE QOS GUARANTEES 2833 the flow ; (2) the vrtual tme stamp ω j,k of the packet that s assocated wth the router currently beng traversed; and (3) the vrtual tme adjustment term δ j,k of the packet. The rate parameter (r j ) sdetermned by the BB (durng call set up tme and at the network edge) based on flow j s QoS requrements (see Sect. 3.4 for detals), and snserted nto every packet of the flow. For the kth packet of flow j, tsvrtual tme stamp ω j,k 1 sntalzed to â j,k 1, the actual tme t leavesthe edge condtoner and entersthe frst core router along the flow spath. That s, ω j,k 1 =â j,k 1. (2) The vrtual tme adjustment term δ j,k for packet p j,k sset to δ j,k = j,k h, (3) where h sthe number of hopsalong the flow spath, and j,k scomputed at the network edge usng the followng recursve formula: j,1 = 0 (4) j,k = max {0, j,k 1 + h Lj,k 1 L j,k r j +â j,k 1 1 } â j,k 1 + Lj,k r j, for k =2, 3,... (5) The physcal meanng of j,k sthat t representsthe cumulatve delay experenced by packet p j,k n an deal dedcated per-flow system [27], where packetsof flow j are servced by h tandem servers wth capacty r j. 3. Vrtual Tme Reference/Update Mechansm and Per-Hop Core Router Behavor Characterzaton. In the conceptual framework of the VTRS, each core router sequpped wth a per-hop vrtual tme reference/update mechansm to mantan the contnual progresson of the (global) vrtual tme emboded by the packet vrtual tme stamps. Ths vrtual tme stamp represents the arrval tme of the kth packet p j,k of flow j at the th core router n the (global) vrtual tme, and thust salso referred to asthe vrtual arrval tme of the packet at the core router. The vrtual ω j,k tme stamps ω j,k s assocated wth packets of flow j satsfy the followng two mportant propertes: (1) vrtual spacng property: ω j,k+1 ω j,k L j,k+1 /r j, and (2) the realty check property: â j,k ω j,k, where â j,k denotesthe actual arrval tme of packet p j,k at router. The vrtual spacng property says that, when measured accordng to ths(global) vrtual tme, the traffc flow preserves ts reserved rate. These two propertes are mportant n ensurng that the end-to-end delay experenced by packetsof a flow acrossthe network core s bounded. In order to ensure that these two propertes are satsfed, the vrtual tme stamps must be approprately referenced or updated aspacketsenter or leave a core router (see Fg. 3). The referencng/updatng rule dependson the schedulng algorthm (or scheduler) employed by a core router and ts characterstcs. Snce we only consder rate-based schedulers n ths paper, then, for scheduler S at the th router, packet p j,k s assocated wth the vrtual delay parameter, d j,k, and d j,k = Lj,k r j + δ j,k, (6) and ts vrtual fnsh tme, ν j,k, sdefned as ν j,k = ω j,k + d j,k = ω j,k + Lj,k r j + δ j,k. (7) The per-hop behavor of a core router (or rather, ts scheduler) s characterzed by an error term, whch s defned wth respect to the vrtual fnsh tme and actual j,k fnsh tme of packets at the router. Let ˆf denote the actual tme packet p j,k departsthe scheduler S.Wesay that S can guarantee flow j tsreserved rate r j wth an j,k error term Ψ, f for any k, ˆf ν j,k +Ψ. In other words, each packet of flow j sguaranteed to depart scheduler S by the tme ν j,k +Ψ = ω j,k j,k + d +Ψ. Gven the error term Ψ of the scheduler S, the vrtual tme stamp of packet p j,k after t hastraversed S supdated usng the followng reference/update rule: ω j,k +1 = νj,k +Ψ + π = ω j,k j,k + d +Ψ + π, (8) where π denotesthe propagaton delay from the th router to the next-hop router along the flow spath. It has been shown [27] that by usng the reference/update rule n (8), the vrtual spacng and realty check propertes of vrtual tme stamps are satsfed at every core router End-to-End Delay Bound and QoS Abstracton of Data Plane An mportant consequence of the VTRS outlned above sthat the end-to-end delay bound on the delay experenced by packetsof a flow acrossthe network core can be expressed n terms of the reserved rate of a flow and the error termsof the routersalong the flow spath. Suppose there are total h hopsalong the path of flow j, all of whch are rate-based schedulers. Then for each packet p j,k of flow j, wehave ˆf j,k h âj,k 1 h Lj,max r j + h =1 h 1 Ψ + π, (9) =1 where L j,max sthe maxmum packet sze of flow j. For the purpose of ths work, we wll onlyconsder ratebased schedulers under the VTRS. A more general treatment of the VTRS that ncludes delay-based schedulers can be found n [27].

5 2834 IEICE TRANS. COMMUN., VOL.E84 B, NO.10 OCTOBER 2001 The VTRS does not mandate any specfc schedulng mechansms to be mplemented n a network doman aslong asther abltesto provde delay guaranteescan be characterzed usng the noton of error term. In fact, t has been shown [27] that almost all known schedulng algorthmscan be characterzed, be they stateless (e.g., FIFO) or stateful (e.g., WFQ) Core-Stateless Vrtual Clock Schedulng Algorthm The VTRS leads to the desgn of a set of new corestateless schedulng algorthms. One of the most mportant one sthe core-stateless vrtual clock (CSVC) scheduler, whch wll be the key buldng block n our node archtecture n thspaper. The CSVC sa work-conservng counterpart of the core-jtter vrtual clock (CJVC) schedulng algorthm [23]. The CSVC scheduler servces packets n the order of ther vrtual fnsh tmes (recall that the vrtual fnsh tme of packet p j,k sgven by ν j,k = ω j,k + L j,k /r j + δ j,k ). It hasbeen shown [27] that aslong asthe total reserved rate of flowstraversng a CSVC scheduler does not exceed ts capacty (.e., j rj C), then the CSVC scheduler can guarantee each flow tsreserved rate r j wth the mnmum error term Ψ = L,max /C, where L,max sthe largest packet sze among all flows traversng the CSVC scheduler. Theorem 1: The end-to-end delay bound experenced by a flow j wth reserved rate r j n a network of the CSVC schedulers s the same as that under a network of the WFQ schedulers. The theorem sproved [27] by showng that the WFQ scheduler has the same error term as the CSVC scheduler under the VTRS, whch s Ψ = L,max /C. Theorem 1 shows that the CSVC scheduler has the same expressve power, n terms of provdng delay and rate guarantees, as a stateful WFQ scheduler, albet t does not mantan any reservaton or schedulng state n the router. Due to paper length lmtaton, we can only gve some hghlghts of VTRS here. We strongly encourage readersto [27] for more detalson the archtectural nsghts and theoretcal underpnnng for VTRS. 2.2 A Node Archtecture wth Scalable Support of Multple QoS Edge Condtonng Edge traffc condtonng playsa key role n our archtecture. In the followng, we show how traffc condtonng sperformed for the GS, the PS, the AS, and the BE flows, respectvely. Edge Shapng forthe GS Flows. Before enterng the traffc shaper, suppose the traffc profle of an GS flow j s specfed usng the standard dual token bucket regulator (see Fg. 4) (σ j,ρ j,p j,l j,max ) where σ j L j,max s the maxmum burst sze of flow j, ρ j s the sustaned rate of flow j, P j sthe peak rate of flow j, and L j,max sthe maxmum packet sze of flow j. Then, under the VTRS, we must ensure that packets of thsflow wll never be njected nto the network core at a rate exceedng tsreserved rate r j (see Fg. 2). That s, the shaper n Fg. 5 ensures that, for a flow j wth a reserved rate r j, the nter-arrval tme of two consecutve packetsof the flow at the frst hop core router satsfes Eq. (1). Note that the edge shaper ntroduces an addtonal edge delay (due to shapng) to the flow. Such edge delay should be accounted for when measurng the end-toend delay bound for the GS flow. Denote d j edge the maxmum delay that packetsof flow j experenced at the edge shaper. Assume that the flow has a reserved Fg. 4 A dual token bucket regulator for an GS flow j. Fg. 5 Edge node shapng and markng for an GS flow. In ths secton, we present a node archtecture, whch bulds upon the VTRS/CSVC, for scalable support of ntegrated traffc of the GS [22], the PS [18], the AS [18], and the BE servces. In Sect , we present the edge condtonng functons. Secton shows the bufferng and schedulng mechansms for both edge and core nodes. Fg. 6 Edge delay due to edge shapng for an GS flow.

6 HOU et al.: PROVIDING SCALABLE SUPPORT FOR MULTIPLE QOS GUARANTEES 2835 rate r j (see Sect. 3.4 for r j calculaton durng admsson control procedure). Then from Fg. 6, we have P j r j r σj L j,max j P j ρ + Lj,max j r j d j edge = f ρ j r j <P j, (10) L j,max r f ρ j P j r j. j Denote T j as T j = σj L j,max P j ρ j (11) f ρ j r j <P j. Then T j sthe maxmum duraton that flow j can nject traffc at tspeak rate (P j )nto the edge shaper f ρ j r j <P j. After the shaper, the packet s marked as an GS packet, wth ts VTRS states set n the packet header as follows (also see Sect. 2.1): (1) r j := r j, whose calculaton sgven n Sect. 3.4; (2) ω j,k 1 := â j,k 1 ; and (3) δ j,k, whch scomputed at edge accordng to Eq. (3). Edge Shapng/Markng for the PS Flows. For an PS flow j, t only hasa peak rate requrement P j as tstraffc profle. Accordng to [18], an PS flow should experence low delay jtter and low packet loss when t traverses the network doman. Under our archtecture, we wll offer the same treatment to an PS flow asthat for an GS flow (albet t doesnot have a delay bound requrement). More specfcally, we wll provde an PS flow j a reserved rate equal to ts peak rate,.e., r j = P j, and let t share the CSVC scheduler wth the GS flowsn the network core (see Sect ). Gven that we wll treat an PS flow the same as f t were an GS flow, the edge traffc condtonng functon for an PS flow svery much smlar to that for an GS flow, except that traffc s shaped usng the traffc s peak rate P j. The shaper n Fg. 7 ensures that, for an PS flow j wth a peak rate P j, the nter-arrval tme of two consecutve packets of the flow at the frst hop core router ssuch that â j,k+1 1 â j,k 1 Lj,k+1 P j. (12) Smlar to the GS, the edge shaper ntroduces an addtonal edge delay (due to shapng) to the flow, whch should be accounted for when measurng the endto-end delay bound for the PS flow. Denote d j edge the maxmum delay that packetsof PS flow j experenced at the edge shaper. Assume that the flow has a reserved rate r j = P j. Then we have d j edge = Lj,max r j. (13) After the shaper, the packet s then marked as an PS packet, wth VTRS states set nto the packet as follows: (1) r j := P j ; (2) ω j,k 1 := â j,k 1 ; and (3) δj,k, whch scomputed at edge accordng to Eq. (3). Edge Markng for the AS Flows. For an AS flow, t Fg. 7 Edge node shapng and markng for an PS flow. Fg. 8 Edge node markng for an AS flow. only has a sustanable rate requrement ρ j aststraffc profle and any packet exceedng such sustanable rate wll be marked asan BE packet [18]. The edge traffc condtonng sshown n Fg. 8. Unlke for an GS or an PS flow, edge shapng s not performed on an AS flow [18]. The functon of the edge condtoner sto examne (test) whether consecutve packets are properly spaced accordng to the sustanable rate ρ j. That s, â j,k+1 1 â j,k 1 Lj,k+1 ρ j. (14) It scrucal that proper care needsto be taken when mplementng (14). In partcular, we must ensure that, n the mplementaton, the ndex, k (representng the kth packet) and arrval tme â j,k 1 for the kth packet supdated n such a way that all the packetsmarked as AS satsfy (14). In partcular, f the arrval tme of a packet wth current ndex (k + 1)th satsfes ths spacng condton, we wll mark such packet as an AS packet and update the varables k wth (k + 1) and â j,k 1 wth â j,k+1 1. Otherwse, we wll mark ths packet as an BE packet and do not update the varables k and â j,k 1, both of whch are mantaned at the edge condtoner (along wth the flow s requested sustanable rate ρ j ). Such update mechansm wll ensure that consecutvely marked AS packetsof flow j satsfy (14) when they enter the core network. Edge Markng for the BE Flows. Snce there sno traffc profle and QoS requrementsfor an BE flow, the BE packetscan enter the network wthout shapng. The edge condtoner only needsto set the BE bt pattern n the packet header and let t drectly enter the network core Bufferng and Schedulng at Edge and Core Nodes The key buldng block n our node archtecture sthe CSVC scheduler, whch s dscussed n Sect Recall that CSVC s a work-conservng schedulng algorthm whch servces packets n the order of ther vrtual fnsh tmes. The vrtual fnsh tme of packet p j,k sgven by ν j,k = ω j,k +L j,k /r j +δ j,k. Aslong asthe total reserved rate of all flows traversng a CSVC scheduler does not

7 2836 IEICE TRANS. COMMUN., VOL.E84 B, NO.10 OCTOBER 2001 Fg. 9 A stateless node archtecture for ntegrated support of the GS, the PS, the AS, and the BE servces. exceed tscapacty (.e., j rj C), the CSVC scheduler can guarantee each flow tsreserved rate r j wth a mnmum error term Ψ = L,max /C, where L,max s the largest packet sze among all the flows traversng the CSVC scheduler. Fgure 9 shows the schematc dagram of our node archtecture. We mantan two separate buffers: one for the GS and the PS flows, and the other for the AS and the BE flows. Both the GS and the PS traffc s servced by the CSVC scheduler, whle the AS and the BE traffc s servced by the FIFO scheduler. A strct prorty (SP) scheduler s employed between the CSVC and the FIFO queues. Schedulng forthe GS Traffc. Recall that the kth packet p j,k of an GS flow j arrvng at the th core router carresa vrtual tme stamp ω j,k, whch represents the arrval tme of ths packet at the th core router n the vrtual tme. Upon arrvng at the CSVC queue, the vrtual fnsh tme of ths packet s calculated by (7). Then thspacket snserted nto the CSVC queue n such a way that the vrtual fnsh tme of all packetsare n ascendng order the packet wth the smallest vrtual fnsh tme s placed at the front of the queue and s servced frst. Upon departure from the CSVC queue of the th node, the vrtual tme stamp of packet p j,k supdated accordng to (8). Note that the above reference (n (7)) (for schedulng) and update (n (8)) requre the feld for vrtual tme stamp to be touched twce once for read out,.e., calculatng vrtual fnsh tme, and once for wrtten nto,.e., updatng vrtual arrval tme for the next hop (also see Fg. 3). A closer look reveals that these two separate operatons are only necessary for conceptual purpose (under VTRS) and can be, for all practcal purpose, combned (and thus smplfed) nto just one operaton (by usng (8)), whch s partcularly sgnfcant n actual hardware mplementaton. More specfcally, snce the term (Ψ + π )sacommon term added nto vrtual tme stamp for all packetsat the th node (ndependent of the partcular flow), we can use the fnal updated vrtual arrval tme for the next hop (.e., ω j,k +1 ) drectly for nserton (.e., sortng) and schedulng n the CSVC queue thus elmnatng the step of calculatng the vrtual fnsh tme ν j,k, whch s used earler for nserton and schedulng. In lght of the above observaton, the followng s a smplfed verson of the CSVC schedulng algorthm n our mplementaton. Upon the arrval of the kth packet p j,k of flow j arrvng at the CSVC queue of the th core router, update the packet svrtual tme stamp ω j,k wth ω j,k +1 usng (8). Then nsert the packet nto ts poston n the CSVC queue such that all packets are n ncreasng order of ther vrtual tme stamp (.e., ω j,k +1 ) the packet carryng the smallest vrtual tme stamp beng placed at the front of the queue and wll be served frst. Asfar asthe GS traffc sconcerned, the lower prorty queue (for the AS and the BE traffc) doesnot nterfere (or hasno effect on) the lnk accessfor the CSVC queue due to the strct prorty schedulng between the CSVC queue and the FIFO queue, and the fact that we have consdered the error term of the CSVC scheduler. Denote d j core the maxmum delay of all packetsn GS flow j traversng the network core. Then d j core sgven by (9),.e., d j core = h Lj,max r j + Denote D P tot as D P tot = h =1 h =1 L,max h 1 + π. (15) C =1 L,max h 1 + π. (16) C =1 Note that Dtot P sa path parameter and sndependent of the partcular GS flow j traversng ths path. Combnng (10) and (15), the maxmum end-to-end delay, d j e2e, for all packetsn flow j sthen gven by d j e2e = dj edge + dj core = P j r j r j σj L j,max P j ρ j (h +1) Lj,max r j +(h +1) Lj,max r j f ρ j r j <P j, + D P tot f ρ j P j r j. + D P tot (17) Observe that the end-to-end delay formula n (17) s precsely the same as that specfed n the IETF IntServ We assume that the buffer sze for the GS and the PS are properlyprovsoned so that no GS or PS packet wll be lost due to buffer overflow. Note that under the DffServ model, the buffer for the GS and the PS traffc maybe mapped to the EF PHB [16] whle the buffer for the AS and the BE maybe mapped to the AF PHB [14]. Snce the specfc mplementatons of EF and AF PHBs are left to the equpment vendors, node archtecture and mechansms other than the one proposed n ths paper mayalso be employed.

8 HOU et al.: PROVIDING SCALABLE SUPPORT FOR MULTIPLE QOS GUARANTEES 2837 GS [22] usng the WFQ as the reference model! In ths sense, the CSVC scheduler provdes the same expressve power, n terms of supportng end-to-end delay guarantee, asa stateful WFQ scheduler, albet t does not mantan any reservaton or schedulng state n the router. Schedulng forthe PS Traffc. Recall that under our archtecture, we treat an PS flow the same as an GS flow (albet t doesnot have a delay bound requrement). That s, we wll provde an PS flow j a reserved rate equal to tspeak rate,.e., r j = P j, and let t share the CSVC scheduler wth the GS flows n the network core. Snce the th CSVC s a rate-based scheduler wth an error term L,max /C, t wll guarantee flow j tsreserved rate P j wth an error term L,max /C. Asan extra beneft for usng the CSVC for the PS traffc, each packet of an PS flow j hasa guaranteed delay bound n the network core,.e., d j core = h Lj,max P j + h =1 L,max h 1 + C =1 = h Lj,max P j + Dtot. P (18) Combnng (13) and (18), the maxmum end-toend delay, d j e2e, for all packetsn PS flow j sthen gven by d j e2e = dj edge + dj core =(h +1) Lj,max r j + Dtot P. (19) We pont out that such an end-to-end delay bound offered by the CSVC scheduler effectvely crcumvents the problem assocated wth (FIFO) class-based SP schedulng, where t has been shown [2], [8] that the delay jtter can be unbounded over a certan utlzaton. Schedulng and BufferManagement forthe AS and the BE Traffc. For the AS and the BE servce, we employ a smple FIFO queue, whch s smlar to that n [18]. The FIFO queue s servced wth a strctly lower prorty than the CSVC queue (see Fg. 9). To dfferentate packetsn the FIFO queue, RED wth In and Out (RIO) [9] semployed asthe buffer management mechansm. RIO retansall the attractve featuresof RED [11] and hasaddtonal capablty of droppng out-of-profle packets(correspondng to the BE packetsunder our archtecture) durng congeston. A schematc dagram for the RIO mechansm s depcted n Fg. 10. RIO employstwo RED algorthmsfor droppng packets, one for the n-profle packets (corresponds to the AS) and one for the out-of-profle packets (corresponds to the BE). By choosng the parameters for the respectve RED algorthms dfferently, RIO s able to preferentally drop the BE packetsand support the sustanable rates of the AS flows [9]. To see how ths can be done, we frst note that RIO starts to drop all BE packetswhen the average queue sze π Fg. 10 Buffer management for the AS and the BE traffc wth RIO mechansm. exceeds Max Out, whle the AS packetscan stll enter the buffer whle the buffer sbeng draned by the FIFO scheduler. If (1) we have provsoned enough buffer sze and set the average buffer threshold Mn In and Max In as large as possble; and (2) the lnk capacty sproperly provsoned for the aggregate AS flows (see Sect. 3.4 on admsson control for the AS flows), we expect that few AS packet wll be dropped and the sustanable rate ρ j for an AS flow j wll be guaranteed (wthn a properly chosen tme scale). 3. Control Plane Operatons The prevous secton presents the network archtecture on the data plane based on the VTRS/CSVC. An attractve feature of the VTRS/CSVC sthat t enablesthe decouplng of QoS control functonsfrom core routers, whch helps to facltate the desgn of a centralzed bandwdth broker (BB) to control and manage resources n a network doman. In ths secton, we present control plane operatons (n partcular, the admsson control procedure) n a BB. We organze thssecton as follows. Secton 3.1 dscusses the advantages of decouplng QoS control from core routers, whch s made possble by the VTRS/CSVC. In Sect. 3.2, we gve an overvew of a BB. Secton 3.3 presents an archtectural desgn for a BB, n partcular, those detals pertnent to admsson control procedure. In Sect. 3.4, we present the admsson control procedure for the GS, the PS, and the AS flows. 3.1 Decouplng QoS Control from Data Plane In the IETF DffServ framework, a centralzed model based on the noton of BB [18] has been proposed for the control and management of QoS provsonng. Under thscentralzed model, each network doman hasa BB (a specal network server) that s responsble for mantanng the network QoS states and performng varousqos control and management functonssuch as admsson control, resource reservaton and provsonng for the entre network doman. Issues n desgnng We refer nterested readers to [13], [15] for more background on varous buffer management mechansms for the Internet.

9 2838 IEICE TRANS. COMMUN., VOL.E84 B, NO.10 OCTOBER 2001 and buldng such a centralzed BB archtecture have been nvestgated n several recent studes [1], [24]. In ths secton, we present the control plane operatonsperformed by a BB for the ntegrated transport of the GS, the PS, the AS and the BE servces. ThsBB archtecture releson the VTRS to provde an QoS abstracton of the data plane. Each router n the network doman employsour node archtecture (Fg. 9) at each of tsoutput port, where the CSVC scheduler can be characterzed by an error term (Ψ = L,max /C) under the VTRS. The novelty of our BB lesn that all QoS reservaton and other QoS control state nformaton (e.g., the amount of bandwdth reserved at a core router) s removed from core routers, and s solely mantaned at and managed by the BB. In supportng the GS, the PS, the AS, and the BE servces n a network doman, core routersperform no QoS control and management functons such as admsson control, but only data plane functons such as packet schedulng and forwardng. In other words, the data plane of the network doman s decoupled from the QoS control plane. Despte the fact that all the QoS reservaton states are removed from core routers and mantaned solely at the BB, the proposed BB archtecture s capable of supportng the GS wth the same QoS granularty and expressve power as the IntServ/GS model. More mportant, thssacheved wthout the potental complexty and scalablty problems of the IntServ model. Some Advantages of Decouplng. Because of ths decouplng of data plane and QoS control plane, our BB archtecture s appealng n several aspects. Frst of all, by mantanng QoS reservaton states only n the BB, core routersare releved of QoS control functons such as admsson control, makng them potentally more effcent. Second, and perhapsmore mportant, an QoS control plane that sdecoupled from the data plane allowsa network servce provder to ntroduce new servces wthout necessarly requrng software/hardware upgrades at core routers. In other words, t enables network servces to evolve (more or less) ndependently from the underlyng network nfrastructure (data plane). Thrd, wth QoS reservaton states mantaned by a BB, t can perform sophstcated (and thus powerful) QoS provsonng and admsson control algorthmsto optmze network utlzaton n a network-wde fashon. Such network-wde optmzaton s dffcult, f not mpossble, under the conventonal hop-by-hop reservaton set-up approach, where each core router makes ts own admsson control decsons based on local QoS nformaton. Furthermore, because the network QoS states are centrally managed by the BB, the problemsof unrelable or nconsstent control states [23] are effectvely crcumvented. Ths s n contrast to the IETF IntServ QoS control model based on RSVP [5], [26], where every router partcpatesn hopby-hop sgnalng for reservng resources and mantans ts own QoS state database. Last but not the least, under our approach, the relablty, robustness and scalablty ssues of QoS control plane (.e., the BB archtecture) can be addressed separately from, and wthout ncurrng addtonal complexty to, the data plane. Asan example to llustrate the flexblty that s made possble through the decouplng of QoS control from core routers. We show how varous lnk sharng polcy [12] can be easly mplemented wthn a network doman, wthout ncurrng any hardware/software upgradesat core routers only the polcy software module n the BB needsto be changed/upgraded. We consder the followng lnk sharng polces for the GS, the PS, the AS, and the BE servces, one of whch (.e., Polcy A) wll be used n our smulaton study n Sect. 5. Polcy A: Statc Lnk Sharng Under thspolcy, the GS, the PS, and the AS each slmted to a fxed maxmum percentage of lnk capacty, e.g., 10% for the GS, 20% for the PS, and 30% for the AS. That s, the GS, the PS, the AS cannot exceed tsmaxmum capacty allocaton on a lnk; any remanng capacty can be used by the BE traffc. Polcy B: Dynamc Lnk Sharng Under thspolcy, none of the servces s lmted to a fxed percentage of lnk capacty. Instead, the lnk capacty s completely shared by all four servces, as long asthe aggregate reserved ratesfor the GS, the PS, and the AS flowsdo not exceed the lnk scapacty. Polcy C: Hybrd Lnk Sharng Thspolcy falls between polcy A and polcy B, both of whch may be consdered extreme cases of lnk sharng for the GS, the PS, the AS, and the BE servces. Under polcy C, the GS, the PS, and the AS each sguaranteed a certan percentage to share lnk capacty (smlar to that under polcy A), e.g., 10% for the GS, 10% for the PS, and 20% for the AS. The remanng capacty can be completely shared by all four types of servces (smlar to that under polcy B), aslong asthe aggregate requred ratesfor the GS, the PS, and the AS flowsdo not exceed the lnk sremanng shared capacty pool. We do not want to make any comment here on whch lnk sharng polcy s better than the other such matter s a network management polcy ssue and should be left to the network provder. The pont that we want to stress s that, due to the decouplng of QoS control from core routers, polcy ssues such as lnk sharng can be easly set up and changed (f nec- Snce the BE servce does not have anyspecfc QoS requrements, such flows do not go through a BB for call processng. Instead, the BE traffc can freelyenter the network, just as the case under today s Internet. For smplcty, we assume that there s a sngle centralzed BB for a network doman. In practce, there can be multple BBs for a network doman to mprove relablty and scalablty[28]. In [28], we address the ssue of scalable desgn of BBs.

10 HOU et al.: PROVIDING SCALABLE SUPPORT FOR MULTIPLE QOS GUARANTEES 2839 Fg. 11 Illustraton of a bandwdth broker (BB) and ts operaton n a VTRS network doman. essary) solely at the BB, and wthout ncurrng any hardware/software upgrade on any core router on the data plane. Thssn contrast wth the hop-by-hop approach, where any lnk sharng polcy change would requre physcal re-confguraton of hardware/software on all relevant core routers(on the data plane), e.g., the weghts n class-based (CBQ) WFQ schedulers. 3.2 Bandwdth Broker: An Archtectural Overvew As the bass for our study, we frst gve an overvew of the basc centralzed BB archtectural model and descrbe how admsson control s performed under such a model, thus, settng the stage for the rest of ths secton. The basc centralzed BB model s schematcally depcted n Fg. 11. In thsarchtectural model, the BB centrally managesand mantansa number of management nformaton (data)bases (MIBs) regardng the network doman. Asshown n Fg. 11, the BB conssts of several modules such as admsson control, QoS routng, and polcy control. In thspaper, we wll focusprmarly on the admsson control module. The BB also mantans a number of management nformaton (data)bases (MIB) for the purpose of QoS control and management of the network doman. For example, the topology nformaton base contanstopology nformaton that the BB uses for route selecton and other management and operaton purposes; the polcy nformaton base contanspolces(e.g., lnk sharng polcy) and other admnstratve regulatons of the network doman. Among them, the network topology database and network QoS state databases are most relevant to admsson control. The network topology database and network QoS state databases together provde a logcal representaton (.e., an QoS abstracton) of the network doman and ts entre states. Wth ths QoS abstracton of the network doman, the BB performsqos control functons by managng and updatng these databases. In ths sense, the QoS control plane of the network doman sdecoupled from tsdata plane. The core routers of the network doman are removed from the QoS control plane: core routersdo not mantan any QoS reservaton state, whether per-flow or aggregate, and do not perform any QoS control functon such as admsson control. In our BB model, the network QoS states are represented at two levels: lnk-level and path-level. The lnk QoS state database mantans nformaton regardng the QoS states of each lnk n the network doman, such as the total reserved bandwdth or the avalable bandwdth of the lnk. The path QoS state database mantansthe QoS state nformaton regardng each path of the network doman, whch s extracted and summarzed from the lnk QoS states of the lnks of the path. An example of the path QoS state s the avalable bandwdth along a path, whch sthe mnmal avalable bandwdth among all tslnks. By mantanng a separate path-level QoS state, the BB can conduct fast admssblty test for flows routed along the path. Furthermore, path-wse resource optmzaton can also be performed based on the (summarzed) path QoS state. Lastly, we note that both the lnk QoS states and path QoS states are aggregate QoS states regardng the lnks and paths. No per-flow QoS states are mantaned n ether of the two QoS databases the QoS and other control state nformaton regardng each flow such as ts QoS requrement and reserved bandwdth smantaned n a separate flow nformaton database managed by the BB. We brefly dscuss the basc operatons of the BB, n partcular, those pertnent to the admsson control module. The detals of the admsson control procedure wll be gven n later part of ths secton. For llustraton purpose, we use an GS flow, whch s the most sophstcated among all the servces. When a new GS flow j wth traffc profle (σ j,ρ j,p j,l j,max ) and end-to-end delay requrement D j,req arrves at an ngress router. The ngress router sends a new flow servce request message to the BB (usng, e.g., COPS [4]). Upon recevng the servce request, the BB frst checks for polcy and other admnstratve nformaton bases to determne whether the new flow s admssble. If not, the request s mmedately rejected. Otherwse, the BB selects a path (from the ngress to an approprate egress router n the network doman) for the new flow, based on the network topology nformaton and the current network QoS state nformaton, n If necessary, a new path may be set up dynamcally. The problem of QoS routng,.e., fndng an optmal path for the flow can be handled relatvelyeaslyunder our BB archtecture. Snce t s beyond the scope of ths paper, we wll not dscuss t here. As an asde, our BB archtecture can also accommodate advance QoS reservaton n a farly straghtforward fashon, thanks to decouplng of the QoS control plane and data plane and the centralzed network QoS state nformaton bases.

11 2840 IEICE TRANS. COMMUN., VOL.E84 B, NO.10 OCTOBER 2001 addton to other relevant nformaton (such as polcy constrants applcable to ths flow). Once the path s selected, the BB wll nvoke the admsson control module to determne f the new flow can be admtted. The detals of admsson control procedure for each type of servces wll be gven n Sect Generally speakng, the admsson control procedure conssts of two phases: (1) admsson control test phase durng whch t s determned whether the new flow servce request can be accommodated and how much network resources must be reserved f t can be accommodated; and (2) bookkeepng phase durng whch the relevant nformaton bases such as the flow nformaton base, path QoS state nformaton base and node QoS state nformaton base wll be updated, f the flow sadmtted. If the admsson control test fals, the new flow servce request wll be rejected, no nformaton bases wll be updated. In ether case, the BB wll nform the ngress of the decson. In the case that the new flow servce request s granted, the BB wll also pass the QoS reservaton nformaton (e.g., reserved rate r j for the GS flow) to the ngress router so that t can set up a new or re-confgure an exstng edge condtoner (whch s assumed to be colocated at the ngress router) for the new flow. The edge condtoner wll approprately ntalze and nsert the packet states nto packets of the new flow once t starts to send packets nto the network. When an exstng flow departs the network, relevant QoS control states and MIBs should also be updated. 3.3 Management Informaton Bases In ths subsecton, we descrbe n detal the MIBs that wll be used by the admsson control module, and set the stage for our dscusson on admsson control procedure n the subsequent subsecton. Flow Informaton Base. ThsMIB contansnformaton regardng ndvdual flows such as servce type (e.g., GS, PS, and AS), flow d., traffc profle (e.g., (σ j,ρ j,p j,l j,max ) for GS, P j for PS, ρ j for AS), servce profle (e.g., end-to-end delay requrement D j,req for the GS, peak rate requrement P j for the PS, and sustanable rate requrement ρ j for the AS), route d. (whch dentfesthe path that a flow j traverses n the network doman) and QoS reservaton (r j n the case of the GS, P j for the PS, and ρ j for the AS) assocated wth each flow. Other admnstratve (e.g., accountng and bllng) nformaton pertnent to a flow may also be mantaned here. Network QoS State Informaton Bases. These MIBsmantan the QoS statesof the network doman, and thusare the key to the QoS control and management of the network doman. Under our BB archtecture, the network QoS state nformaton s represented n two-levels usng two separate MIBs: path QoS state nformaton base and lnk QoS state nformaton base. These two MIBs are presented n detal below. Path QoS state nformaton base mantansa set of paths(each wth a route d.) between varous ngress and egress routers of the network doman. These paths can be pre-confgured or dynamcally set up. Assocated wth each path are certan statc parameterscharacterzng the path and dynamc QoS state nformaton regardng the path. Examples of statc parameters assocated wth a path P are the number of hops h on P, sum of the router error termsof all the CSVC schedulersand propagaton delay along P, Dtot P (see (16)), and the maxmum permssble packet sze (.e., MTU) L P,max. The dynamc QoS state nformaton assocated wth P nclude, among others, the set of flowstraversng P (.e., the GS, the PS, and the AS flows) and a number of QoS state parameters regardng the (current) QoS reservaton status of P such as the mnmal remanng (resdual) bandwdth along P for each type of servces,.e., CP,res GS for the GS, CP,res PS for the PS, and CAS P,res for the AS. Lnk QoS state nformaton base mantansnformaton regardng the router lnksn the network doman. Assocated wth each router lnk s a set of statc parameters characterzng the router and a set of dynamc parameters representng the router s current QoS states. Examples of statc parameters assocated wth a router lnk are ts error term(s) Ψ = L,max /C, propagaton delaysto ts next-hop routers π s, confgured total bandwdth and buffer sze. The dynamc router QoS state parameter ncludesthe current resdual bandwdth and buffer sze at the lnk for each type of servces. 3.4 Admsson Control We present a path-orented approach to perform effcent admsson control test and resource allocaton. Unlke the conventonal hop-by-hop approach whch performs admsson control ndvdually based on the local QoS state at each router along a path, thspathorented approach examnes the resource constrants along the entre path smultaneously, and makesadmsson control decson accordngly. As a result, we can sgnfcantly reduce the tme of conductng admsson control test. Clearly, such a path-orented approach s Note that durng the process of a path set-up, no admsson control test s admnstered. The major functon of the path set-up process s to confgure forwardng tables of the routers along the path, and f necessary, provson certan schedulng/queue management parameters at the routers, dependng on the schedulng and queue management mechansms deployed. Hence we refer to such a path a traffc engneered (TE) path. Set-up of such an TE path can be done byusng a path set-up sgnalng protocol, say, MPLS [7], [21], or a smplfed verson (mnus resource reservaton) of RSVP.

12 HOU et al.: PROVIDING SCALABLE SUPPORT FOR MULTIPLE QOS GUARANTEES 2841 possble because the avalablty of QoS state nformaton of the entre path at the BB. For the rest of ths subsecton, we gve detals on admsson control for the GS, the PS, and the AS flows. For the ease of exposton, we employ statc lnk sharng polcy dscussed earler. That s, µ GS + µ PS + µ AS < 1, (20) where µ GS, µ PS, and µ AS are fxed maxmum allowed percentage share on the th lnk for the GS, the PS, and the AS, respectvely. Note that there s no fxed allocaton for the BE flows snce they are not subject to admsson control and can freely enter the network and share any remanng network bandwdth. Admsson Control for the GS Flows. Asfar asthe GS flowsare concerned, they are served by a network of CSVC schedulers snce each CSVC queue n our node archtecture (Fg. 9) sgven strct prorty (SP) over the other FIFO queue. Let j F GS denote that an GS flow j currently traverses the th node and C GS be the total bandwdth at th node allocated to the GS flow,.e., C GS = µ GS C. Then aslong as j F r j GS C GS, the th node can guarantee each GS flow j ts reserved bandwdth r j. We us e C,res GS to denote the resdual bandwdth at the th node for the GS flows,.e., C,res GS = CGS j F r j. We consder the two GS phases of the admsson control procedure. 1. Admsson Test. Let (σ ν,ρ ν,p ν,l ν,max ) be the traffc profle of a new GS flow ν, and D ν,req be tsend-toend delay requrement. Let h be the number of hops along P, the path for the new flow. From (17), n order to meet tsend-to-end delay requrement D ν,req, the reserved rate r ν for the new flow ν must satsfy the delay constrant d ν e2e D ν,req. That s, D ν,req d ν e2e = P ν r ν r ν σν L ν,max P ν ρ ν +(h +1) Lν,max r ν + D P tot f ρ ν r ν <P ν, (h +1) Lν,max r ν + D P tot f ρ ν P ν r ν. (21) Let r ν be the smallest r ν that satsfes (21) and recall that T ν =(σ ν L ν,max )/(P ν ρ ν ) (see (11)). Then we have r ν = T ν P ν +(h+1)l ν,max D ν,req D P tot +T ν f ρ ν r ν <Pν, (h+1)l ν,max D ν,req D P tot f ρ ν P ν r ν. (22) If the above soluton for r ν exst, then we have a feasble reserved rate requrement to satsfy the flow s end-toend delay requrement n (21). Otherwse, the flow s not admssble on ths path. Fgure 12 outlnesthe procedure to calculate the 0. Upon the arrval of a new flow ν on path P requestng 1. GS: 2. Flow ν s traffc profle: (σ ν,ρ ν,p ν,l ν,max ); 3. Delay requrement: D ν,req ; 4. Path P: h hops. 5. f Dtot P Dν,req 6. /* no feasble r ν exst */ 7. End. 8. else /* Dtot P <Dν,req, feasble r ν exsts */ 9. T ν = σν L ν,max P ν ρ ν ; 10. r ν = T ν P ν +(h+1)l ν,max D ν,req D tot P ; +T ν 11. f ρ ν r ν <P ν 12. return r ν ; 13. else /* r ν P ν */ 14. r ν = (h+1)lν,max D ν,req D tot P ; 15. return r ν ; 16. End. Fg. 12 GS. Computaton of the reservaton rate for a new flow for reservaton rate r ν for a new flow ν requestng GS. Accordng to Fg. 12, we frst check whether the path parameter Dtot P sgreater than or equal to the delay requrement D ν,req. If ths s true, then no feasble reservaton rate r ν exst. If Dtot P slessthan D ν,req, we frst try to use the upper formula n (22) to calculate r ν (lnes9 to 12 n Fg. 12). If such r ν can be found, we are done; otherwse, we wll use the lower formula n (22) (lne 14 n Fg. 12) to calculate r. ν Next, we examne f the path hassuffcent remanng bandwdth to accommodate thsnew rate request. That s, r ν must not exceed the mnmal resdual bandwdth CP,res GS along path P for the GS, where CP,res GS = mn P C,res GS smantaned (asa path QoS parameter assocated wth P) n the path QoS state MIB. If ths condton cannot be satsfed, the servce request of the new flow ν must be rejected. Otherwse, t s admssble, and r ν sthe mnmum feasble reserved rate for the new flow ν. Gven that the path QoS parameters Dtot P and Cres P assocated wth P are mantaned n the path QoS state MIB, the above admsson test can be done n O(1). 2. Bookkeepng. If the new GS flow ν sadmtted nto the network, several MIBs (e.g., the flow MIB, the path and lnk QoS state MIBs) must be updated. The flow d., traffc profle and servce profle of the new flow wll be nserted nto the flow MIB. The mnmal resdual bandwdth CP,res GS wll be subtracted by rν, the reserved rate for the new GS flow ν. Smlarly, for each lnk along P, tsresdual bandwdth C,res GS wll also be subtracted by r ν. Furthermore, for any path P that traverses S, P, tsmnmal resdual bandwdth C P,GS res may also be updated, dependng on whether the update of C,res GS changes CGS P,res. Provded that a powerful (and perhaps, parallel) database system s employed, these database update operatons can be performed n very short tme. Note that when an exstng flow departs the network, the relevant MIBs should also

13 2842 IEICE TRANS. COMMUN., VOL.E84 B, NO.10 OCTOBER 2001 be updated. Admsson Control for the PS Flows. The admsson control for the PS flows s smpler than that for the GS flowssnce a new PS flow ν can explctly submt a reserved rate requrement,.e., ts peak rate requrement P ν. Thus, n contrary to the case for a new GS flow, no calculaton of the reserved rate s needed for a new PS flow request. Smlar to admsson control for the GS flows, a path-orented approach can be appled to a new PS flow. 1. Admsson Test. Let peak rate P ν be the traffc profle of a new PS flow ν. To admt the new PS flow ν, P ν must not exceed the mnmal resdual bandwdth CP,res PS along path P for PS, where CP,res PS = mn P C,res PS s mantaned (as a path QoS parameter assocated wth P) n the path QoS state MIB. If ths condton cannot be satsfed, the servce request of the new PS flow ν must be rejected. Otherwse, t s admssble, and P ν wll be the reserved rate for the new flow ν. 2. Bookkeepng. If the new flow ν sadmtted nto the network, several MIBs (e.g., the flow MIB, the path and lnk QoS state MIBs) must be updated. Such operatons are very smlar to that for the GS flows. Admsson Control for the AS Flows. The admsson control for an AS flow s very smlar to that for an PS flow, except that the traffc profle and servce profle for a new AS flow ν s ts sustanable rate ρ ν. Due to paper length lmtaton, we omt to dscuss t further. 4. Dscussons In ths secton, we dscuss the performance and complexty of our archtecture. 4.1 Performance Under our archtecture, core routersof the network doman are releved from the QoS control functons: core routers do not mantan any QoS reservaton state, whether per-flow or aggregate. Therefore, our core network s core-stateless and fulflls our second desgn requrement scalablty. Snce our network archtecture buldsupon the VTRS, whch scapable of provdng an QoS abstracton of the network doman, we can use a centralzed BB to perform QoS control functonsby managng and updatng relevant databases. Therefore, the QoS control plane of the network doman s decoupled from tsdata plane. Furthermore, any new polcy or servce can be ntroduced solely by approprate software nstallaton/update n the BB, wthout ncurrng any hardware/software re-confguraton on the core routers. Therefore, our archtecture meetsour thrd desgn requrement decouplng QoS control from core routers for flexble resource allocaton and QoS provsonng. Now we show that our archtecture also meets our frst desgn requrement ntegrated transport of the GS, the PS, the AS, and the BE servces wth QoS guarantees for each servce. The followng corollary summarzesthe behavor of admtted GS and PS flowsunder our node archtecture. Corollary 1.1: An VTRS network doman where each core routersemployng our node archtecture (Fg. 9) provdesguaranteed rate and end-to-end delay to each admtted GS and PS flows, respectvely. In partcular, we have: (1) For an admtted GS flow j, the end-to-end delay of each packet sbounded by (17); and (2) For an admtted PS flow j, tspeak rate P j sguaranteed. Moreover, the end-to-end delay of each packet of the admtted PS flow j sbounded by (19). Proof: Asfar asan admtted GS or PS flow sconcerned, t s served by a network of CSVC schedulers. The lower prorty queue for the AS and the BE traffc doesnot nterfere (or hasno effect on) the lnk access for the CSVC scheduler. Gven the fact that we have consdered the error term of the CSVC scheduler, the corollary then followsfrom Theorem 1. Now we examne the performance for the AS and the BE flows. Note that at node, although the AS and the BE share the same FIFO queue, whch s served wth a strctly lower prorty than the CSVC queue, the rate allocated to the FIFO queue sat least (C C GS C PS ), whch s strctly greater than C AS, the allocated rate to the AS flows (see (20)). Now we show how the RIO mechansm can provde each AS flow j ts sustanable rate ρ j. Suppose we have suffcent buffer sze so that we can set the average buffer thresholds for the AS (.e., Mn In and Max In) reasonably large and average buffer thresholds for the BE (.e., Mn Out and Max Out) reasonably small (see Fg. 10). Then all of the BE packetswll be dropped f the average buffer occupancy (aggregated traffc of the AS and the BE) exceeds Max Out. Once the average buffer occupancy reachesor exceedsthspont, any BE traffc wll cease to enter the buffer (untl the average buffer occupancy decreases below Max Out) but the AS packetscan stll enter the buffer whle the buffer s beng draned by the FIFO scheduler, wth a rate of at least (C C GS C PS ), whch sgreater than C AS. Therefore, the sustanable rate for each AS can be guaranteed wthn a certan tme scale. 4.2 Complexty We frst dscuss mplementaton complexty on the data plane. A crucal queston on mplementng the VTRS/CSVC sthe overhead of packet state encodng. The packet state contans three parameters: (1) packet vrtual tme stamp ω j,k, (2) reserved rate r j, and (3) vrtual tme adjustment term δ j,k. There are several optons that we can use to encode the packet state: usng an IP opton, usng an MPLS label, usng the IP

14 HOU et al.: PROVIDING SCALABLE SUPPORT FOR MULTIPLE QOS GUARANTEES 2843 fragment offset feld. Issues on effcent packet state encodng wll be nvestgated n our future work. Another complexty assocated wth the VTRS/ CSVC s per-packet processng at core routers: (1) packet state lookup, (2) vrtual tme stamp update (for the next hop), and (3) nserton (.e., sortng) nto the CSVC queue. The last operaton,.e., packet nserton (.e., sortng) n the CSVC queue, s perhaps the most complex step n hardware mplementaton. We beleve that t s possble to reduce the complexty assocated wth sortng by usng a noton of slotted vrtual tme. Such smplfed mplementaton s smlar to the rotaton prorty queue ntroduced by Lebeherr and Wrege [17], whch sa trade-off between performance (delay bound) and mplementaton complexty (sortng). We now dscuss complexty assocated wth the BB on the control plane. There are several aspects regardng scalablty that need to be addressed: (1) lnk/path QoS control states updates after ether admttng a new GS/PS/AS flow or the departure of an exstng flow; (2) the sze of flow nformaton base used to mantan each flow straffc profle and servce profle; and (3) the potental bottleneck between the BB snput/output nterface and flow requests sent from all the edge routers. To address these ssues, a path-orented and quota-based approach (or PoQ for short) has been proposed [28]. It has been shown [28] that such PoQ approach can be used to desgn multple and herarchcal BB archtecture, under whch all the ssues rased above regardng scalablty and relablty of a centralzed BB can be satsfactorly resolved. Snce the focus of ths paper s on the performance of data plane for scalable and ntegrated support of the GS, the PS, the AS, and the BE traffc, we wll not elaborate further on scalable desgn of the BBs. But we stress that, due to the decouplng of data plane and control plane, whch s made possble by the VTRS, ssues related to the desgn of BBs can be addressed separately from, and wthout ncurrng addtonal complexty to, the data plane. 5. Smulaton Investgaton In ths secton, we conduct smulatons to nvestgate the performance of the ntegrated framework for supportng dverse servce guarantees. The focus of ths smulaton study s twofold: (1) the effectveness of the proposed node archtecture n satsfyng dfferent user requrements; and (2) the effcacy of the BB n controllng the network load for traffc requrng dfferent servce guarantees. 5.1 Smulaton Settngs The network topologes that we wll use for the smulatonsare depcted n Fgs. 13, 14, and 15, whch are referred to asthe peer-to-peer, parkng-lot, and chan network, respectvely. In these networks, a node la- Fg. 13 Fg. 14 Fg. 15 A peer-to-peer network. A parkng-lot network. A chan network. beled wth I denotesan ngressedge router, Cj a core router j, Ek an egress edge router k, and Gn an end host group n. Flowsgenerated from a source node group Gn (S) wll be destned to the destnaton node group Gn (D), traversng the shortest path from the source node to the destnaton node. Each ngress edge router (I) contanstwo functonalty blocks: an edge condtoner (see Sect ) and the node archtecture n Fg. 9. On the other hand, each core router (Cj) employsonly the node archtecture n Fg. 9. For all network confguratons, the capacty of the lnks between an end host and an edge (ether an ngress or egress) router s assumed to be nfnte and the propagaton delay sneglgble. In the peer-to-peer network, Such complextycomes from the fact that, n order to acheve scalablty, we are makng a trade-off between tme (.e., per-packet processng) and space (.e., per-flow reservaton state and per-flow schedulng state mantaned at a core router as n the case of WFQ schedulng). Under the peer-to-peer network, we onlyhave one group of flows (.e., n = 1) traversng G1(S) to G1(D), whereas under the parkng-lot or chan network, there are multple groups of flows traversng dfferent paths of the respectve network.

15 2844 IEICE TRANS. COMMUN., VOL.E84 B, NO.10 OCTOBER 2001 Table 1 Traffc patterns for a flow used n the smulaton study. Traffc Traffc Bucket depth (σ) Mean rate (ρ) Peak rate (P ) Mean on tme Mean off tme Packet sze Wndow sze name type n bt (b) (kb/s) (kb/s) n seconds n seconds n byte (B) (packets) exp1 UDP/EXP exp2 UDP/EXP exp3 UDP/EXP exp4 UDP/EXP ftp TCP/FTP the propagaton delay of the lnk between I1 and E1 s 10 ms. Whle n the parkng-lot and chan networks, all the lnksbetween routershave a propagaton delay equal to 5 ms. We wll specfy the capacty of the lnks between the routers when we present the smulaton results. We assume that the CSVC queue n the node (Fg. 9) hasnfnte buffer sze (.e., t wll never drop packets). On the other hand, the maxmum buffer sze of the low prorty RIO queue s set to 100 packets. For the RIO queue, the buffer confguraton for the n-profle AS traffc s(40, 70, 0.02),.e., the mnmum buffer threshold (Mn In) s40 packets, the maxmum buffer threshold (Max In) s70 packets, and the maxmum droppng probablty (M axp In) s2%. Meanwhle, the buffer confguraton for the out-of-profle traffc s(10, 30, 0.5),.e., the mnmum buffer threshold (Mn Out) s10 packets, the maxmum buffer threshold (Max Out) s30 packets, and the maxmum droppng probablty (M axp Out) s50%. The weght parameter (q weght) [9] used for estmatng the average queue lengthssset to The traffc patterns used for the flows n our smulatonsare lsted n Table 1. The feldsmarked as meansthat t snot applcable to the correspondng traffc type. Asshown n the table, the exponental on-off traffc types exp1 and exp2 are token bucket constraned, whle exp3 and exp4 are not. For the traffc type ftp, the TCP verson s Reno, and there snfnte traffc to be sent,.e., persstent source. As shown n Fg. 4, f a flow requests GS, then a dual-token bucket regulator s attached to the traffc source to ensure that the traffc conformsto the (σ, ρ, P, L max ) traffc envelop. There san nfnte buffer at the entrance of the dual-token bucket regulator to hold the outof-constrant traffc of the flow,.e., out-of-constrant packets from the source s shaped nstead of beng dropped. 5.2 Smulaton Results We present smulaton results as follows Data Plane In ths frst set of smulatons, we wll examne the effectveness of the proposed node archtecture. The performance metrcsthat are of nterest are the end-to-end packet delay, traffc throughput, and packet loss rate. In ths set of smulatons, the capacty of the lnks between edge (ngress or egress) routers and core routers, and the lnksbetween core routerss10 Mb/s n all three network confguratons. We employ statc lnk sharng polcy and set the targeted traffc load on a lnk for each servce type (ether GS, PS, or AS) to be 30%,.e., µ GS = µ PS = µ AS =0.3 for lnk. The smulated tme s 200 seconds, of whch the frst 100 seconds are the smulaton warm-up perod. To examne the key propertesof the proposed node archtecture, flowswth a traffc profle and servce requrement wll be generated randomly between the tme nterval [0, 1] seconds from a source node group Gn (S) to a destnaton group Gn (D), and once admtted by the BB, they wll never termnate,.e., they have nfnte holdng tme. Such long holdng tme of a flow wll provde usa steady perod where varouspacket level statstcs (e.g., per-packet delay, flow throughput, and packet loss rate) can be collected and analyzed. For the BE traffc, we only actvate four ftp flows randomly wthn the tme nterval [0, 0.5] seconds from the source node group G1(S) to the destnaton group G1(D); whle between other source and destnaton groups(n the case of parkng-lot and chan network confguratons), two BE ftp flows are randomly actvated wthn the tme nterval [0, 0.5] seconds. To demonstrate packet-level QoS performance for a flow wth a partcular traffc profle and servce requrement, we focuson the admtted flowstraversng the path G1(S) to G1(D), and purposely choose only 12 flowsamong all the admtted flowsalong thspath to present our smulaton results. These 12 flows are lsted n Table 2 and represent traffc profle and servce profle of all four types of servces of our nterest. In partcular, flows1 to 8 requre a partcular servce guarantee whle flows9 to 12 are BE ftp traffc, whch do not requre any servce guarantee. For a flow requestng GS, the BB wll use the algorthm n Fg. 12 to calculate the reservaton rate. As an example, n the parkng-lot network, flow 1 (see Table 2), whch traverses the path from G1(S) to G1(D) and hasa delay requrement of 120 ms, we obtan a reservaton rate of Kb/s wth Fg. 12. Fgure 16(a) shows the end-to-end packet delays of flows1 and 2, whch request GS n the peer-to-peer In the next subsecton, when we nvestgate the performance of the BB for controllng the network load for each type of servce, we wll use shorter flow holdng tme so as to generate dynamcs on the flow level.

16 HOU et al.: PROVIDING SCALABLE SUPPORT FOR MULTIPLE QOS GUARANTEES 2845 Table 2 Traffc profles and servce requrements of the 12 flows chosen on the path traversng G1(S) to G1(D) n the smulatons. Flow d Requred servce GS GS PS PS AS AS AS AS BE Traffc profle exp1 exp2 exp3 exp4 exp3 exp4 exp1 exp2 ftp For GS, end-to-end peer-to-peer delay bound parkng-lot requrement (ms) chan Fg. 16 End-to-end packet delays of the GS/PS flows on the path traversng G1(s) to G1(D) n the peer-to-peer network. Fg. 17 End-to-end packet delays of the GS/PS flows on the path traversng G1(s) to G1(D) n the parkng-lot network. network. Comparng the end-to-end delay requrement lsted n Table 2 (100 ms), we observe that the delay requrements of both flows are satsfed. Fgure 16(b) presents the end-to-end packet delays for the two PS flows 3 and 4 n the same smulaton. Recall that one of the objectvesof the PS sto acheve a relatvely small packet queueng delay throughout the network doman. From Fg. 16(b), we see that the traffc of the PS flows experences almost constant delay, wth almost no jtter! It snterestng to note that even though both GS and PS traffc share the same hgh prorty queue, the end-to-end packet delay jtter of GS traffc srelatvely hgher than that of the PS traffc. A close examnaton revealsthat the larger delay jtter of the GS traffc comes from the more conservatve traffc shapng at the network edge (see (10)). That s, the GS traffc s released nto the network accordng to the reserved rate of the flow, whle PS traffc sreleased accordng to ts peak rate. Therefore, a relatvely longer packet queue can accumulate at the edge condtoner for an GS flow. The same observatons on the end-to-end packet delayshold for the smulatonswth the parkng-lot and chan network confguratons, each of whch are presented n Fgs. 17 and 18, respectvely. So far we have confrmed that the end-to-end delay requrements for the GS and PS are satsfed. Next, we nvestgate the traffc throughput and packet loss rate of the 12 flows durng the same smulaton run under the three network confguratons. Table 3 presents the correspondng data wth the peer-to-peer network for the smulaton run (.e., 100 s 200 s). As expected, there s no packet lost n both GS and PS flows. Therefore, we omt to lst the packet loss rato (0) for the GS and the PS flows n the table. From the table, we see that our framework provdes the protecton for the GS, the PS, and the AS traffc from the burst of the BE ftp traffc, therefore, achevng some degree of rate guarantees. Note that flows 5 and

17 2846 IEICE TRANS. COMMUN., VOL.E84 B, NO.10 OCTOBER 2001 Fg. 18 End-to-end packet delays of the GS/PS flows on the path traversng G1(s) to G1(D) n the chan network. 6 are burst traffc, whch are more aggressve than the token bucket constraned flows 7 and 8. That s, n flows5 and 6, there are consderable packetsmarked as BE traffc at the network edge because of ther bursty behavor; whle n flows7 and 8, few packetsare marked asbe traffc. Therefore, flows5 and 6 have a hgher droppng rate than flows7 and 8. It sworth notng that all the dropped packetsn flows5 and 6 are outof-profle, therefore marked as BE packets. Thus, by properly confgurng the RIO, we can ndeed protect the n-profle AS packetsfrom both out-of-profle (although they are wthn the same AS flow, they are actually marked asbe traffc) and BE ftp traffc. In Table 3, flows 9 to 12 are BE flows. We see that these BE flows have a smlar packet droppng rate. Moreover, ther throughput are reasonably close to each other. Thus, we conclude that the resdual bandwdth of the lnk sdstrbuted among the BE ftp flowsn a relatvely far manner, whch sacheved by the random early droppng property of the RED buffer management mechansm. Agan, the same observatons on the traffc throughput and packet loss rate hold for the smulatons wth the parkng-lot and chan network confguratons, whch are lsted n Tables 4 and 5, respectvely. Table 3 Flow throughput (kb/s) and packet loss rates for the 12 flows chosen from the path traversng G1(S) to G1(D) n the peer-to-peer network. GS/PS AS BE Flow d. Throughput Flow d. Throughput Loss rate Flow d. Throughput Loss rate % % % % % % Table 4 Flow throughput (kb/s) and packet loss rates for the 12 flows chosen from the path traversng G1(S) to G1(D) n the parkng-lot network. GS/PS AS BE Flow d. Throughput Flow d. Throughput Loss rate Flow d. Throughput Loss rate % % % % % % Table 5 Flow throughput (kb/s) and packet loss rates for the 12 flows chosen from the path traversng G1(S) to G1(D) n the chan network. GS/PS AS BE Flow d. Throughput Flow d. Throughput Loss rate Flow d. Throughput Loss rate % % % % % %

18 HOU et al.: PROVIDING SCALABLE SUPPORT FOR MULTIPLE QOS GUARANTEES Control Plane In the second set of smulatons, we examne the capablty of the BB entty n controllng the network load for each type of servces. Because, among all the lnks n the network, only the bottleneck lnk wll play a role n flow admsson control, we wll only conduct smulatonswth the peer-to-peer network (Fg. 13) here. In the followng smulatons, the capacty of the lnk between the ngress router I1 and the egress router E1 s set to 100 Mb/s. The smulated tmes are 2000 seconds, of whch the frst 1500 seconds are the smulaton warm-up perod. Because the BE traffc does not requre any servce guarantee and doesnot contact the BB before sendng traffc, there sno need to consder the BE traffc n ths set of smulatons. Therefore, we wll focus on the three typesof traffc that doesrequre admsson control: the GS, the PS, and the AS. We set the GS flows be the exp2 type traffc, and all the PS and AS flowsbe the exp4 type traffc. Flowshave an exponentally dstrbuted nter-arrval tme wth a mean of seconds and an exponental holdng tme wth a mean of 120 seconds. Each new flow arrval wll request the GS, the PS, or the AS wth equal probablty. Agan, we set a target network load rato for a servce requrement type n the BB usng the statc lnk sharng polcy. Fgures19(a) and (b) present the network load for the followng two cases: (1) Case 1: µ GS =0.1, µ PS =0.2, and µ AS =0.3; and (2) Case 2: µ GS = µ PS = µ AS =0.3. As shown n the fgures, the BB entty can ndeed control the traffc load on the data plane so that each type of servce does not exceed tstarget rato. 6. Related Work A node archtecture based on strct prorty (SP) schedulng was proposed prevously [18] to provde ntegrated transport of the PS, the AS, and the BE servces. However, t hasbeen shown [2], [8] that, once over certan utlzaton, the concatenaton of such class-based strct prorty (SP) schedulers can cause delay jtter unbounded! Thsmeansthat such node archtecture [18] cannot always support the PS, whch should experence low delay jtter. On the other hand, the node archtecture proposed n ths paper, whch bulds upon the VTRS/CSVC, can provde hard delay jtter bound to the PS (see (19)). Furthermore, our node archtecture can also support the GS, whch s otherwse not possble under the FIFO CBQ and SP archtecture proposed n [18]. The dea of vrtual tme has been used extensvely n the desgn of packet schedulng algorthms such as VC [25] and WFQ [10], [19], [20]. The noton of vr- Fg. 19 Traffc load on the lnk between ngress and egress nodes for each type of servces n the peer-to-peer network. tual tme defned n these contexts s used to emulate an deal schedulng system and s defned local to each scheduler. Computaton of the vrtual tme functon requres per-flow nformaton (e.g., per-flow reservaton state and per-flow schedulng state) to be mantaned. In contrast, n ths paper the noton of vrtual tme emboded by packet vrtual tme stamps can be vewed as global to an entre doman. Itscomputaton scorestateless, relyng only on the packet state carred by packets. In [23], the frst core-stateless schedulng algorthm, core jtter vrtual clock (CJVC), wasdesgned and shown to provde the same end-to-end delay bound as a stateful VC-based reference network. In addton, an aggregate reservaton estmaton algorthm was developed for performng admsson control wthout perflow state. Our VTRS/CSVC was motvated and nspred by the results n [23]. However, the VTRS dffers from [23] n several aspects. The VTRS s defned to serve as a unfyng schedulng framework where dverse schedulng algorthms can be employed to support the GS [27]. The noton of packet vrtual tme stamps under the VTRS can lead to the desgn of new core-stateless schedulng algorthms, e.g., core-stateless vrtual clock (CSVC). Furthermore, the VTRS sdefned wth an am to decouple the data plane and the QoS control plane so that a flexble QoS provsonng and control archtecture can be developed at the net-

19 2848 IEICE TRANS. COMMUN., VOL.E84 B, NO.10 OCTOBER 2001 work edge, wthout affectng the core of the network. Thspaper sa sequel to our prevouswork on VTRS [27]. In partcular, thspaper showsthat, n addton to the GS, we can also support the PS, the AS, and the BE servces under the same node archtecture. Furthermore, we also present detaled desgn and operatonson the control plane (BB) and show how QoS control and provsonng can be performed solely at the BB wthout ncurrng any addtonal complexty on the core routers. The BB model was frst proposed n [18] for the PS and the AS usng the DffServ model. Under the BB archtecture, admsson control, resource provsonng, and other polcy decsons are performed by a centralzed BB n each network doman. However, many ssues regardng the desgn of the BB such as admsson control and QoS provsonng have not been addressed adequately. Furthermore, t s not clear, under the BB archtecture proposed n [18], what level of QoS guaranteescan be supported and whether or not core routersare stll requred to perform local admsson control and QoS state management. One the other hand, the BB archtecture proposed n ths paper have clearly addressed the above ssues. In partcular, we show that a BB archtecture, whch buldsupon VTRS/CSVC, s capable of decouplng QoS control functonsfrom core routers. That s, our BB performs admsson control and QoS state management solely at the BB, wthout ncurrng any addtonal complexty to the core routers on the data plane. Furthermore, our BB archtecture supports per-flow GS (n addton to the PS and the AS). 7. Concluson We presented archtecture and mechansms to support multple QoS under the DffServ paradgm. On the data plane, we presented a node archtecture based on the vrtual tme reference system (VTRS). The key buldng block of our node archtecture sthe corestateless vrtual clock (CSVC) schedulng algorthm, whch, n termsof provdng delay guarantee, hasthe same expressve power as a stateful weghted far queueng (WFQ) scheduler. Wth the CSVC scheduler as our buldng block, we desgned a node archtecture that s capable of supportng ntegrated transport of the GS, the PS, the AS, and the BE servce. On the control plane, we desgned a bandwdth broker archtecture to provde flexble resource allocaton and QoS provsonng. Smulaton results demonstrated that our archtecture and mechansms can ndeed provde scalable and flexble support for ntegrated traffc of the GS, the PS, the AS, and the BE servces. Acknowledgments The authorswsh to thank the anonymousrevewers for ther thorough revew and helpful commentsand suggestons, whch helped to mprove the presentaton of thspaper. References [1] P. Auka, M. Kodalam, P.V.N. Koppol, T.V. Lakshman, H. Sarn, and B. Suter, RATES: A server for MPLS traffc engneerng, IEEE Network Magazne, pp.34 41, March/Aprl [2] J.C.R. Bennett, K. Benson, A. Charny, W.F. Courtney, and J.Y. Le Boudec, Delay jtter bounds and packet scale rate guarantee for expedted forwardng, Proc. IEEE INFO- COM 2001, Anchorage, Alaska, Aprl [3] S. Blake, D. Black, M. Carlson, E. Daves, Z. Wang, and W. Wess, An archtecture for dfferentated servces, RFC 2475, Internet Engneerng Task Force, Dec [4] J. Boyle, R. Cohen, D. Durham, S. Herzog, R. Rajan, and A. Sastry, The COPS (common open polcy servce) protocol, RFC 2748, Internet Engneerng Task Force, Jan [5] R. Braden, D. Clark, and S. Shenker, Integrated servces n the Internet archtecture: An overvew, RFC 1633, Internet Engneerng Task Force, July [6] R. Braden, L. Zhang, S. Berson, S. Herzog, and S. Jamn, Resource ReSerVaton Protocol (RSVP) Verson 1functonal specfcaton, RFC 2205, Internet Engneerng Task Force, Sept [7] R. Callon, P. Doolan, N. Feldman, A. Fredette, G. Swallow, and A. Vswanathan, A framework for multprotocol label swtchng, Internet Draft, Internet Engneerng Task Force, Sept [8] A. Charny and J.Y. Le Boudec, Delay bounds n a network wth aggregate schedulng, Proc. Frst Internatonal Workshop of Qualty of future Internet Servces (QofIS 2000), Berln, Germany, Sept , [9] D.D. Clark and W. Fang, Explct allocaton of best-effort packet delvery servce, IEEE/ACM Trans. Networkng, vol.6, no.4, pp , Aug [10] A. Demers, S. Keshav, and S. Shenker, Analyss and smulatons of a far queueng algorthm, Proc. ACM SIG- COMM 89, pp.1 12, Austn, TX, [11] S. Floyd and V. Jacobson, On random early detecton gateways for congeston avodance, IEEE/ACM Trans. Networkng, vol.1, no.4, pp , Aug [12] S. Floyd and V. Jacobson, Lnk-sharng and resource management models for packet networks, IEEE/ACM Trans. Networkng, vol.3, no.4, pp , Aug [13] R. Guern and V. Pers, Qualty-of-servce n packet networks: basc mechansms and drectons, Computer Networks Journal (Elsever Scence), vol.31, no.3, pp , Feb [14] J. Henanen, F. Baker, W. Wess, and J. Wroclawsk, Assured forwardng PHB group, RFC 2597, Internet Engneerng Task Force, June [15] Y.T. Hou, D. Wu, B. L, T. Hamada, I. Ahmad, and H.J. Chao, A dfferentated servces archtecture for multmeda streamng n next generaton Internet, Computer Networks Journal (Elsever Scence), vol.32, no.2, pp , Feb [16] V. Jacobson, K. Nchols, and K. Podur, An expedted forwardng PHB, RFC 2598, Internet Engneerng Task Force, June [17] J. Lebeherr and D.E. Wrege, Prorty queue schedulers wth approxmate sortng n output buffered swtches, IEEE J. Sel. Areas Commun., vol.17, no.6, pp ,

20 HOU et al.: PROVIDING SCALABLE SUPPORT FOR MULTIPLE QOS GUARANTEES 2849 June [18] K. Nchols, V. Jacobson, and L. Zhang, A two-bt dfferentated servces archtecture for the Internet, RFC 2638, Internet Engneerng Task Force, July [19] A.K. Parekh and R.G. Gallager, A generalzed processor sharng approach to flow control n ntegrated servces networks: The sngle-node case, IEEE/ACM Trans. Networkng, vol.1, no.3, pp , June [20] A.K. Parekh and R.G. Gallager, A generalzed processor sharng approach to flow control n ntegrated servces networks: the multple node case, IEEE/ACM Trans. Networkng, vol.2, no.2, pp , Aprl [21] E.C. Rosen, A. Vswanathan, and R. Callon, Multprotocol label swtchng archtecture, RFC 3031, Internet Engneerng Task Force, Jan [22] S. Shenker, C. Partrdge, and R. Guern, Specfcaton of guaranteed qualty of servce, RFC 2212, Internet Engneerng Task Force, Sept [23] I. Stoca and H. Zhang, Provdng guaranteed servces wthout per flow management, Proc. ACM SIGCOMM 99, Cambrdge, MA, Sept [24] A. Terzs, L. Wang, J. Ogawa, and L. Zhang, A two-ter resource management model for the Internet, Proc. IEEE Global Internet 99, Ro de Janero, Brazl, Dec [25] L. Zhang, VrtualClock: A new traffc control algorthm for packet swtchng networks, ACM Trans. Computer Syst., vol.9, pp , May [26] L. Zhang, S. Deerng, D. Estrn, S. Shenker, and D. Zappala, RSVP: A new resource ReSerVaton protocol, IEEE Network Magazne, vol.7, no.5, pp.8 18, Sept [27] Z.-L. Zhang, Z. Duan, and Y.T. Hou, Vrtual tme reference system: A unfyng schedulng framework for scalable support of guaranteed servces, IEEE J. Sel. Areas Commun., vol.18, no.12, pp , Dec [28] Z.-L. Zhang, Z. Duan, and Y.T. Hou, On scalable desgn of bandwdth brokers, IEICE Trans. Commun., vol.e84-b, no.8, Aug Ywe Thomas Hou obtaned hs B.E. degree (Summa Cum Laude) from the Cty College of New York n June 1991, the M.S. degree from Columba Unversty n May 1993, and the Ph.D. degree from Polytechnc Unversty, Brooklyn, New York, n January 1998, all n Electrcal Engneerng. He was awarded a fve-year Natonal Scence Foundaton (NSF) Graduate Research Traneeshp for pursung Ph.D. degree n hgh speed networkng, and was recpent of the Alexander Hessel award for outstandng Ph.D. dssertaton ( academc year) from Polytechnc Unversty. Snce September 1997, Dr. Hou has been a Research Scentst at Fujtsu Laboratores of Amerca (FLA), Sunnyvale, Calforna. At FLA, he has been conductng and leadng research efforts n the areas of scalable archtecture, protocols, and mplementatons for dfferentated servces Internet; and qualty of servce (QoS) support for meda streamng over wred and wreless IP-based networks; and servce overlay archtecture over the Internet. Dr. Hou s a co-recpent of the 2001 IEEE Transactons on Crcuts and Systems for Vdeo Technology best paper award. He s a member of the IEEE, ACM, Sgma X and New York Academy of Scences. Zhenha Duan receved a B.S. degree from Shandong Unversty, Chna, and M.S. degree from Pekng Unversty, Chna, n 1994 and 1997, respectvely, both n Computer Scence. Currently he s pursung a Ph.D. degree n the Department of Computer Scence and Engneerng at the Unversty of Mnnesota. Hs research nterests are n the areas of computer and communcatons networks, currently n QoS provsonng over the Internet, traffc measurements and modelng. Mr. Duan s a student member of IEEE and ACM. Zh-L Zhang receved a B.S. degree n Computer Scence from Nanjng Unversty, Chna and hs M.S. and Ph.D. degrees n Computer Scence from Unversty of Massachusetts, Amherst, n 1992 and 1997 respectvely. In January 1997, he joned the Computer Scence faculty at the Unversty of Mnnesota, where he s currently an Assstant Professor. From 1987 to 1990, he conducted research n the Computer Scence Department at Århus Unversty, Denmark, under a fellowshp from the Chnese Natonal Commsson on Educaton. Dr. Zhang s current research nterests nclude hgh speed networks, multmeda and real-tme systems, and modelng and performance evaluaton of computer and communcaton system. He receved the Natonal Scence Foundaton CAREER Award n He s also a co-recpent of 1996 ACM SIGMETRICS Conference best paper award, and a member of IEEE, ACM and INFORMS. Takafum Chujo s Drector of Advanced Internet Research at Fujtsu Laboratores of Amerca, Sunnyvale, CA. Hs research nterests nclude IP/photonc networkng nfrastructure and next-generaton Internet archtecture. Mr. Chujo receved hs B.E. degree from Kanazawa Unversty (Japan) n 1976 and M.E. degree from Nagoya Unversty (Japan) n He served as a tutoral co-char for NOMS 96 and a TPC co-char for APNOMS 99. He s a member of IEEE.