Worst-Case Delay Bounds for Uniform Load-Balanced Switch Fabrics

Transcription

1 Wost-Case Delay Bounds fo Unifom Load-Balanced Switch Fabics Spyidon Antonakopoulos, Steven Fotune, Rae McLellan, Lisa Zhang Bell Laboatoies, 600 Mountain Ave, Muay Hill, NJ Abstact Numeous solutions have been poposed in the liteatue to eliminate eodeing in load-balanced switch fabics. A common appoach involves unifom fames, in which evey cell of a fame has the same destination. This can achieve 100% thoughput with elatively small aveage taffic delay; howeve, the wost-case delay may be unbounded. We show that with a slight speedup in the switch fabic we can guaantee satisfactoy wost-case delay bounds, without sacificing othe desiable popeties. Futhemoe, expeimental esults demonstate that ou scheme impoves wost-case delay in ealistic taffic scenaios, as compaed to pevious unifom-fame solutions. I. INTRODUCTION Load balancing in switch fabics has been studied extensively fo moe than a decade, as a building block to obtain high-capacity outes; see e.g. [1], [2], [3], [7], [8], [9], [12], among othes. The pimay motivation behind these eseach effots is the explosive gowth in Intenet taffic, vaiously estimated at 30-50% pe yea [4], [10]. As a esult of that tend, load-balanced switch fabics have become inceasingly impotant because they ae inheently paallel, enabling switch capacity to scale even without any incease in VLSI clock ate. Futhemoe, load-balanced switch fabics ae known to attain 100% thoughput fo a easonably geneal model of admissible taffic [1], [2]. A typical load-balanced switch fabic consists of thee stages of cell-handling elements. In this pape, we focus on fame-based fabics with a full-mesh inteconnect between successive stages, as illustated in Figue 1. Data packets, aleady split into fixed-size cells, aive at a fist-stage distibuto. Cells ae accumulated into a fame of cells that ae sent out in paallel to the middle stage (called the outing stage), one cell pe outing element. Each outing element similaly accumulates fames of cells to be sent out in paallel to the final collecto stage. Nevetheless, unlike a distibuto, each outing stage pefoms outing, so that each cell is sent to its intended collecto. Fom thee cells exit the switch fabic and ae then eassembled into packets. Fame-based fabics allow high-aggegate-capacity switches to be built fom elements that need not opeate much faste than the ate at which taffic can aive at a distibuto. Moeove, if thee ae m elements at each stage, each link between stages opeates at a faction 1/m of the ate into the distibuto. Ideally, the m 2 links between stages could be encapsulated in a single optical device, a polycyclic aay wave guide [9]. We emak that load-balanced switch fabics wee initially descibed with cossba-based inteconnects ealizing distibutos Fig. 1. mesh mesh outing elements Fame-based switch fabic collectos a fixed schedule of matchings [1], [2]; the full mesh descibed above eplaces the cossba, which would be challenging to implement at high ates. Cells fom a specific distibuto sent to a specific collecto may tavese diffeent outing elements. If these outing elements do not impose the same delay (e.g. because of diffeing queue occupancies), cells ae likely to aive at the collecto in a diffeent ode fom which they wee sent. Requiing that fames be unifom, i.e. all cells in each fame have the same destination, has been poposed as a way to pevent this celleodeing poblem [7], [8]. Indeed, unifom fames guaantee that, at any given time and fo any given collecto, all outing element queues to that collecto have identical occupancies and thus incu identical delays. In this way, cell eodeing absolutely cannot occu, since each fame aives at its destined collecto as a whole, exactly as it was sent fom the distibuto, and of couse no fame can ovetake an ealie fame sent to the same collecto. Seveal switch scheduling policies that ae based on the afoementioned appoach [7], [8] povably achieve 100% thoughput and ensue bounded aveage cell delay. Unfotunately, though, thee is no guaantee whatsoeve in tems of the wost-case delay, which can be abitaily lage. Poviding such a guaantee is the main goal of this pape. Moe specifically, we pesent the Bounded Latency Fames (BLF) scheduling policy, which pemits a fame to be sent fom a distibuto if: (a) thee ae enough cells with the same destination to fill a fame; o (b) some cell in the distibuto has been waiting fo moe than a specific peiod of time. We subsequently show that, fo any taffic aival patten, a unifom-fame-based switch fabic with this scheduling policy

2 2 delays a cell by at most a constant moe than would an ideal output-queued switch. To obtain the above esult, we equie that the switch fabic have a slight speedup, because sending fames befoe they ae completely filled wastes bandwidth. Additionally, we assume that intenal switch queues do not have maximum occupancy limits and may gow abitaily lage which is standad fo such theoetical compaisons. Theefoe, ou esult complements those of [7], [8]: a stonge assumption (namely, fabic speedup) yields a stonge guaantee (i.e., wost-case bounded delay athe than aveage-case bounded delay). The pape is oganized as follows. Section II descibes ou BLF scheduling policy and fomally states the pefomance guaantee that BLF povides. All technical poofs, howeve, ae elegated to Section III. In Section IV we evaluate the pefomance of BLF in ealistic settings via simulations. Last but not least, Section V pesents ou concluding emaks. A. Peliminaies II. BOUNDED LATENCY FRAMES As mentioned ealie, in a unifom load-balanced switch, evey fame sent by a distibuto is unifom, that is, evey cell in the fame has the same collecto as destination. Conside the schematic epesentation of Figue 1, and let m denote the numbe of elements on each of the thee stages (distibutos, outing elements, and collectos). Then, each fame consists of exactly m cells. At fixed intevals, each of which is called a fame time, evey distibuto sends a fame to the outing stage. Fo that pupose, evey distibuto must apply the scheduling policy independently to detemine the collecto fo which each fame is intended. If necessay, empty cells containing dummy data may be added to a fame to each the equisite numbe of m cells so that it can be sent. Similaly, at each fame time, evey outing element sends at most one cell to each collecto. Routing elements opeate deteministically, all using the same algoithm to fill queues. Fo example, if at a given fame time multiple distibutos send fames destined to the same collecto, then the ode in which the outing elements queues to that collecto ae filled with cells fom those fames is identical fo all elements (e.g. beak ties by the index of each fame s distibuto). Consequently, at any time, all the outing-stage queues associated with a specific collecto have the same length, and futhemoe all cells associated with a paticula distibuto fame ae at the same position in these queues. Thus, all cells in a fame aive at thei destination simultaneously. Note that an empty cell used to fill out a fame occupies a position within a outing element queue, and pesists until it eaches a collecto, whee it is dopped. As an exception to the above, a fame consisting only of empty cells 1 should be maked by the distibuto so that it can be discaded immediately afte aiving at the outing stage. 1 Sending such fames seems pointless, but in pactice it could be equied to maintain fabic synchonization. Finally, we assume that duing each fame time at most m = α cells may aive at a distibuto, and also at most cells may depat a collecto. The quantity α > 1 expesses the speedup of the switch fabic compaed to the capacity of the input and output links at the distibutos and collectos, espectively. B. Scheduling policy A vey obvious pefomance concen in unifom switch fabics is the delay incued while a distibuto waits fo enough cells with the same destination to ceate a fame. Filling up fames with empty cells enables them to be sent in a moe timely fashion, but doing this too often consumes a lot of bandwidth and leads to high queueing delays in the outing stage. Nevetheless, by applying the above technique judiciously and exploiting the speedup α, we demonstate a scheduling policy that bounds wost-case delay. As cells aive at a distibuto, they ae placed in the appopiate vitual output queue by destination, with the oldest cell at the head of the queue. At any given time, the cells in a queue ae patitioned in ode fom oldest to youngest into (potential) fames: zeo o moe full fames containing exactly m cells, and possibly a patial fame containing at least one but fewe than m cells. Now, define = αm α 1. (1) A patial fame is stale if the cuent fame time exceeds the aival time of the oldest cell in the fame by at least. In the Bounded Latency Fames (BLF) scheduling policy, each distibuto examines its vitual output queues in oundobin ode, choosing a queue to seve if it contains eithe a full fame o a stale fame. Thus, the fame at the head of the chosen queue is sent to the outing stage, padded with empty cells if it is stale. If no queue has a full o stale fame, a fame of all empty cells is sent instead. Lemma 1. Unde the BLF scheduling policy, no cell is delayed at a distibuto fo moe than + m fame times. We have aleady pointed out that sending stale fames squandes bandwidth. To establish that the speedup α suffices to make up fo that loss, we elate the delay expeienced by an abitay cell though the unifom switch fabic with that of an ideal output-queued switch. In the latte, at each fame time, any cells that aive at the distibutos ae instantaneously fowaded to the appopiate collecto, and then cells depat fom each collecto queue o the entie queue, if its occupancy is less than. An aival sequence {a t } (fo a specific collecto) indicates the total numbe a t of cells aiving ove all distibutos destined fo the collecto duing the t-th fame inteval. The intinsic backlog b t (fo a specific collecto) at time t fo an aival sequence {a t } is defined as ( ) f b t = max 0, max (a t i ). 0 f t i=0

3 3 It is easy to see that b t expesses the numbe of cells in the collecto queue of an ideal output-queued switch at time t, afte the cells have been emitted. This can also be veified by an inductive agument: it is obviously tue at time 0, and at time t the numbe of cells in the queue is max(0, b t 1 +a t ), which equals b t. Moeove, we define the intinsic delay δ t at time t as δ t = bt Clealy, δ t is the delay expeienced by the last cell to aive at the output-queued switch duing fame inteval t. Theoem 2. Suppose a unifom switch fabic with the BLF scheduling policy has unbounded intenal queues. Then, a cell aiving at a distibuto at time t will exit the collecto within δ t+ +m m fame intevals. In othe wods, fo any aival sequence, the delay seen by a cell tavesing the unifom switch fabic is only up to a constant (depending on m and α) geate than the delay expeienced by a cell aiving at almost the same time at an ideal output-queued switch. Remak. In contast to the above, suppose the switch has a congestion contol mechanism that pevents intenal queues fom gowing too lage by tempoaily focing upsteam elements to stop sending moe cells. In such a scenaio, the delay guaantees of Lemma 1 and Theoem 2 ae no longe valid even fo taffic to a collecto that is neve congested. This complication is not specific to the unifom load-balanced switch, and additional speedup is equied to addess it, even with coss-ba-based fabics [5]. C. Extensions Conside a situation in which the incoming taffic load to the switch is low, i.e. on aveage much fewe than cells aive at each distibuto pe fame time. Unde the BLF scheduling policy, these cells will expeience a delay of oughly fame times, in expectation. This leaves oom fo impovement, since the bandwidth of the switch is undeutilized. In paticula, we can educe aveage cell delay in the afoementioned setting by combining BLF with a technique inspied fom [7]. Namely, if a distibuto has no vitual output queue with a full o stale fame to seve, it again examines the queues in ound-obin ode, choosing a queue Q to seve if: (a) Q contains a patial fame; and (b) the outing-stage queues associated with the same collecto as Q cuently hold less than T fames. Hee, T is a paamete that we set as desied. Natually, if thee is still no suitable queue to seve, a fame of all empty cells is sent. As a esult, this modified BLF policy tends to decease the time that cells have to wait at a distibuto, povided that the outing-stage queues they will be sent to have sufficiently low occupancies. Moe impotantly, the wost-case delay guaantee of Theoem 2 is only minimally affected by that change. Theoem 3. Suppose a unifom switch fabic with the (modified) BLF scheduling policy has unbounded intenal queues.. Then, a cell aiving at a distibuto at time t will exit the collecto within δ t+ +m m + T fame intevals. III. PROOFS Poof of Lemma 1: We say that a fame in a distibuto queue is blocked if thee is a fame ahead of it in the queue, necessaily a full fame. Now, obseve that the lemma is staightfowad fo any fame that is not blocked, since at wost it will become stale in fame times, and will then be seved within anothe m fame times. So, let us assume that some fame is blocked at fame time t+1 (by the aival of the (m+1)-st cell to a queue), but thee wee no blocked fames at time t. We claim that at some fame time between t + 1 and t +, inclusive, thee will be no full fames, hence no blocked fames. Evey once-blocked fame will eithe have been seved o become head of its queue; in the latte case, the fame will be seved within m time units of becoming full o stale. To see the claim, fist note that thee must exist a full fame at evey fame time that thee is a blocked fame, which implies that some fame is seved then. In the fame times between t + 1 and t + δ, δ, thee ae clealy δ sevice oppotunities; how many fames will need to be seved? We ague at most m + δ/α. Indeed, thee ae at most m fames that ae patial o full at fame time t, one pe queue, but no blocked fames. Moeove, if a patial fame is seved because it became stale, no othe stale fame fom the same queue is seved duing that inteval. As a esult, any othe fame that is seved must have all its cells aive between fame times t and t + δ. Thee can be at most δ = δ m/α such cells, hence at most δ/α full fames. Now, since m + δ/α (α 1) /α + /α, thee exists some δ such that, by fame time t + δ, thee will have been as many sevice oppotunities as fames to seve, and all full fames will have been seved. Fo the following poofs, we only count queueing delays and ignoe any delays due to pocessing o tansfes between stages. Thus, a cell that aives at a distibuto at fame time t could be emitted fom a collecto also at time t if it wee not delayed because of queue occupancy. A collecto queue is backlogged at time t if it is not empty afte the depatue of the cells at time t. Suppose a cell is emitted fom a collecto queue at time t. The collecto backlog inteval of the cell is the inteval [s..t 1] such that the queue is backlogged at times s,..., t 1, but not backlogged at time s 1. Likewise, the outing-stage queues fo a specific collecto ae backlogged at time t if they contain anothe fame to be sent to the collecto (containing at least one nonempty cell). Futhemoe, the outing backlog inteval of a cell is defined similaly to the collecto backlog inteval. Poposition 4. Suppose the outing-stage queues fo a specific collecto ae backlogged fo an inteval I of g consecutive fame times, but not backlogged befoe I. Then, the fames sent

4 4 E D t C t B A distibuto outing stage collecto Fig. 2. Time lines (hoizontal) in ode fom top to bottom: aival time at distibuto; depatue fom distibuto/aival at outing stage; depatue fom outing stage/aival at collecto; depatue fom collecto. The diagonal dotted line epesents the tajectoy of cell c aiving at time t. to the collecto duing I contain at least (g m) nonempty cells. Poof: Since the outing stages wee not backlogged befoe I, any fames sent to the collecto must have aived duing I. Wite g = g 1 + g 2, whee g 1 = g/ and g 2 = g mod. At most g stale fames can aive at the outing stage duing the inteval fom a single distibuto. Howeve, thee can be at most min(g 2, m) such distibutos, since the aival times at the outing stage of the fist and last stale fames ae sepaated by at least g 1. Othe distibutos must contibute g 1 o fewe stale fames each. Hence, the total numbe of stale fames summed ove all distibutos is g 1 m + min(g 2, m). To finish the poof, the numbe η of nonempty cells is at least m times the numbe of full fames: η m (g ( g 1 m + min(g 2, m) )) ( mα α g 1 α 1 + g 2 g ) 1m(α 1) min(g 2, m) α 1 ( ) g1 mα = α 1 + g 2 + (α 1)g 2 α min(g 2, m) (g m), whee the inequality (α 1)g 2 α min(g 2, m) m is easily veified using a case split on g 2 m o g 2 > m. Poof of Theoem 2: Conside cell c aiving at a distibuto at time t, destined fo a specific collecto. Define intevals of fame times A, B, C, D, and E as follows. Let A = [s A..t A 1] be the collecto backlog inteval of c. B A is the inteval fom the stat of A, i.e. s A, to the aival time of c at the collecto. Obviously, at time s A one o moe cells must aive at this collecto fom the outing stage, othewise the collecto would not become backlogged. The outing backlog inteval of these cells is C, which may be empty. D B C is the inteval fom the stat of C (o of B, if C empty) to the aival of c at the outing stage. Finally, E is the inteval of length + m befoe D. Figue 2 povides a visualization of the above definitions. We wish to bound the numbe a of cells that ae emitted fom the collecto duing A. Any such cell must have aived at the collecto duing B, since A is the backlog inteval of c. Likewise, any cell that aives at the collecto duing B C must have aived at the outing stage duing D. By Poposition 4, the numbe of cells sent to the collecto duing C is at least ( C m), whee C is the length of C. By Lemma 1, any cell sent to the outing stage duing D must have aived at the distibuto duing D E. Consequently, a a s ( C m). We have A = a/ since A is the backlog inteval fo c. Let t be the endpoint of D. Using E = + m, 1 A + C D a s + m D 1 = (a s ) + + 2m The cell is emitted at time δ t + + 2m. t A = t + (t t) + A + C D t + (t t + δ t ) + + 2m t + ( + m + δ t+ +m ) + + 2m, whee the second inequality uses the obsevation that the quantity t t + δ t cannot decease as t vaies fom t to t + + m. Sketch of poof of Theoem 3: The poof is vey simila to that of Theoem 2, so hee we only highlight thei diffeences. To begin with, obseve that Lemma 1 holds fo the modified BLF policy as well. Moeove, let us amend two definitions fom Section III. In paticula, the outing-stage queues fo a specific collecto ae backlogged at time t if they contain moe than T fames, and the outing backlog inteval of a cell is analogously edefined. Unde the new definitions, Poposition 4 also caies ove, with a mino change in its statement: the fames sent to the collecto duing I + T (i.e. the inteval I shifted fowad by T fame times) contain at least (g m) nonempty cells. Lastly, fo the aguments in the poof of Theoem 2 to go though, we modify the intevals B and C: B now stats at time s A T instead of s A, and C is the backlog inteval of the cells, if any, that at time s A T occupy the T -th position in the outing-stage queues fo the collecto in question (which means that they will be sent to the collecto at time s A ). It is easy to deduce that the net esult of all those changes is simply an additional tem T in the delay bound. IV. SIMULATIONS To complement the theoetical analysis, we conducted pefomance simulations of the BLF scheduling policy, specifically its vesion descibed in Section II-C, unde ealistic taffic pattens. The benchmak used fo these simulations is the closely elated Padded Fames (PF) policy [7], which has

5 5 Max delay (fame times) % 20% 30% 40% 50% 60% 70% 80% 90% Taffic load PF BLF Fig. 3. Plot of maximum packet delay vesus taffic load fo the PF and BLF scheduling policies, on a switch. The hoizontal dotted line shows the value of, included fo efeence puposes. been expeimentally shown to exhibit lowe aveage delay than othe poposed unifom-fame policies. The load-balanced switch fabic on which PF and BLF wee applied had m = 32 elements pe stage, and a small speedup α = Thus, (1) yields = 147. Additionally, the paamete T was set to 4 fo both policies. Note that PF does not equie any speedup, unlike BLF; nevetheless, fo the pupose of a fai compaison we chose to keep the speedup in both cases. Futhemoe, the incoming taffic at distibuto i destined fo collecto j follows a Paeto distibution with aveage ate ρ ij. These ates ae andomly chosen fo each distibuto-collecto pai, subject to the condition ρ ij = ρ ij = l, i j fo all i and j. The paamete l above, which is independent of i and j, epesents the aveage taffic load on the switch, and in ou test instances it anges fom 10% to 90% of the capacity of each of the input and output links that cay taffic into the distibutos and out of the collectos, espectively. Figue 3 pesents the maximum obseved delay that was incued unde each scheduling policy, expessed in fame times. As expected, BLF ensues that all taffic taveses the switch in a timely manne. By contast, fo modeate to high loads, PF causes some packets to expeience vey lage delays. It is also instuctive to examine these esults in conjunction with the coesponding aveage delays, which ae displayed in Table I. Note that both policies attain almost identical aveage delays in each case, with BLF gaining a vey slight advantage as the load inceases. Consequently, it appeas that BLF s focus on bounding maximum delay (by expediently sending stale fames, which consume moe bandwidth) does not ham oveall pefomance in tems of aveage delay on the contay, it maginally impoves it. TABLE I AVERAGE PACKET DELAY FOR PF AND BLF Taffic load PF BLF 10% % % % % % % % % delay expeienced by taffic passing though the switch is at most a constant moe than the coesponding delay though an ideal output-queued switch; this distinguishes BLF fom peviously known unifom-fame policies. Moeove, in ode to be implemented, BLF equies only a small speedup in the switch fabic, e.g. in the ode of Last but not least, expeimental evidence indicates that in pactice the maximum delay bound that BLF achieves does not come at the expense of wose aveage delay, whence we conclude that BLF delives a well-ounded pefomance. REFERENCES [1] C. Chang, D. Lee, Y. Jou, Load-balanced Bikhoff-von Neumann switches: Pat I: One-stage Buffeing, Comput. Commun. 25:6, 2002, pp [2] C. Chang, D. Lee, Y. Jou, Load-balanced Bikhoff-von Neumann switches: Pat II: Multi-stage Buffeing, Comput. Commun. 25:6, 2002, pp [3] C. Chang, D. Lee, Y. Shih, C. Yu, Mailbox Switch: A Scalable two-stage Switch Achitectue fo Conflict Resolution of Odeed Packets, IEEE Tansactions on Communications, 56:1, Januay 2008, pp [4] CISCO Visual Netwoking Index: Foecase and Methodology, , available within [5] S.-T. Chuang, A. Goel, N. McKeown, B. Pabhaka, Matching Output Queueing with a Combined Input Output Queued Switch, Compute Systems Technical Repot CSL-TR Mach 1998, o Poceedings of INFOCOM 99, , IEEE, Apil 1999, o IEEE Jounal on Selected Aeas in Communications, 17:6, Decembe 1999, pp [6] W. Felle, An Intoduction to Pobability Theoy and its Applications, Volume I, Thid Edition, John Wiley and Sons, NY, [7] J. Jaamillo, F. Milan, R. Sikant, Padded Fames: a Novel Algoithm fo Stable Scheduling in Load-Balanced Switches, IEEE/ACM Tansactions on Netwoking, 16:5, Octobe 2008, pp [8] I. Keslassy, The Load-balanced Route, Ph.D. Dissetation, Stanfod Univ, Stanfod CA, [9] I. Keslassy, S.-T. Chuang, K. Yu, D. Mille, M. Hoowitz, O. Solgaad, N. McKeown, Scaling Intenet Routes Using Optics (Extended Vesion), Stanfod Technical Repot TR03-HPNG , [10] Minnesota Intenet Taffic Studies (MINTS), available at [11] M. Mitzenmache, E. Upfal, Pobability and Computing: Randomized algoithms and pobabilistic analysis, Cambidge Univesity Pess, Cambidge, [12] Y. Shen, S. Panwa, H. Chao, Design and Pefomance Analysis of a Pactical Load-Balanced Switch, IEEE Tansactions on Communications 57:8, August 2009, pp V. CONCLUSION In this pape, we intoduced a novel scheduling policy fo load-balanced switch fabics, called Bounded Latency Fames. We also poved that, unde the BLF policy, the wost-case