Exploiting Locality to Ameliorate Packet Queue Contention and Serialization

Transcription

1 Exploiting Lolity to Ameliorte Pket Queue Contention nd Seriliztion Silesh Kumr Wshington University Computer Siene nd Engineering St Louis, MO John Mshmeyer Wshington University Computer Siene nd Engineering St Louis, MO Ptrik Crowley Wshington University Computer Siene nd Engineering St Louis, MO ABSTRACT Pket proessing systems mintin high throughput despite reltively high memory ltenies y exploiting the orse-grined prllelism ville etween pkets In prtiulr, multiple proessors re used to overlp the proessing of multiple pkets Pket queuing the fundmentl mehnism enling pket sheduling, differentited servies, nd trffi isoltion requires red-modify-write opertion on linked list dt struture to enqueue nd dequeue pkets; this opertion represents potentil serilizing ottlenek If ll pkets witing servie re destined for different queues, these red-modify-write yles n proeed in prllel However, if ll or mny of the inoming pkets re destined for the sme queue, or for smll numer of queues, then system throughput will e serilized y these sequentil externl memory opertions For this reson, low lteny SRAMs re used to implement the queue dt strutures This redues the solute ost of seriliztion ut does not eliminte it; SRAM ltenies determine system throughput Permission to mke digitl or hrd opies of ll or prt of this work for personl or lssroom use is grnted without fee provided tht opies re not mde or distriuted for profit or ommeril dvntge nd tht opies er this notie nd the full ittion on the first pge To opy otherwise, or repulish, to post on servers or to redistriute to lists, requires prior speifi permission nd/or fee CF 6, My 3-5, 26, Ishi, Itly Copyright /6/5$5 In this pper we oserve tht the worst-se senrio for pket queuing oinides with the est-se senrio for hes: ie, when lolity exists nd the mjority of pkets re destined for smll numer of queues The min ontriution of this work is the queuing he, whih onsists of hrdwre he nd losely oupled queuing engine tht implements queue opertions The queuing he improves performne drmtilly y moving the ottlenek from externl memory onto the pket proessor, where lok rtes re higher nd ltenies re lower We ompre the queuing he to numer of lterntives, speifilly, SRAM ontrollers with: no queuing support, softwreontrolled he plus queuing engine (like tht used on Intel s IXP network proessor), nd hrdwre he Reltive to these models, we show tht queuing he improves worst-se throughput y ftors of 31, 15, nd 21 nd the throughput of rel-world trffi tres y ftors of 26, 13, nd 175, respetively We lso show tht the queuing he dereses externl memory ndwidth usge, on-hip ommunition, nd the numer of queuing instrutions exeuted under est-se, worst-se nd rel-world trffi worklods Bsed on our VHDL models, we onlude tht queuing he ould e implemented t low ost reltive to the resulting performne nd effiieny enefits Ctegories nd Sujet Desriptors: B3m [Memory Strutures]: Misellneous Generl Terms: Performne, Design Keywords: Pket queuing, uffering, he 1 INTRODUCTION Pket queues represent ritil seriliztion point in pket proessing systems An rriving pket is inserted into queue sed on the router s lssifition poliy, with the ssigned queue representing the type of servie the pket should reeive Similrly, when pket is sheduled for trnsmission, the sheduler removes it from its queue Queues re implemented s linked lists, nd these opertions require red-modify-write opertions on the queue desriptor tht keeps trk of the strt, end nd size of the queue When multiple queues re simultneously tive, the red-modify-write opertions n e rried out in prllel Modern pket proessing systems use orse-grined prllelism, in the form of multiple on-hip proessors nd memory ontrollers, to exploit this sitution However, the most hllenging requirement is to provide good performne for onseutive opertions on the sme queue due to the lk of opportunity for prllelism The worst-se, performne-limiting senrio in modern pket proessing systems rises when there is high ontention for smll numer of queues High-performne systems must support lrge numer of suh queues; therefore queue desriptors re kept in off-hip memory In high-speed networks, pket inter-rrivl times rivl memory ess times, so SRAMs re used for this purpose For exmple, QDR SRAM hs n effetive ess lteny of t lest 15 to 2 ns, when hip to hip interonnets re ounted for, nd the minimum pket rrivl time in n OC-768 link is 8 ns Therefore, when seriliztion ours in smll numer of queues, the lteny to perform red-modify-write opertions through off-hip memory will determine performne In this pper, we oserve tht ontention for smll numer of queues is form of lolity, nd is therefore idel for he We propose the queuing he, n on-hip, hrdwre-sed he for hip-multiproessors tht supports queue opertions diretly In

2 Physil Ports Link Interfe SRAM DRAM Ingress NP Egress NP SRAM DRAM Fri Interfe Swith Fri Figure 1: A Network Proessor-sed router line rd our evlution, we ompre queuing he performne to: 1) n unmodified system with no queuing or he support, 2) se system ugmented with hrdwre dt he, nd 3) system with softwre-mnged queuing he We show tht the queuing he provides superior throughput over wide rnge of syntheti nd rel-world worklods, while inresing effiieny y reduing on-hip ommunition, reduing memory ndwidth, nd reduing the numer of instrutions exeuted in softwre The rest of this pper is orgnized s follows Setion 2 provides kground on network proessor (NP)-sed pket proessing systems Setion 3 desries the queuing he s well s other trditionl memory system models for pket queuing Setion 4 presents our experimentl evlution of the queuing he nd the lternte models Setion 5 provides n nlysis of the results nd elortes queuing he implementtion detils Finlly, the pper onludes in Setion 6 2 PACKET PROCESSING SYSTEMS The orgniztion of n NP-sed router line rd is shown in Figure 1 There re vritions mong line rds, inluding those tht ugment NPs with queue mngement hipsets [17], ut the overll orgniztion nd use of SRAM nd DRAM for queuing, s desried elow, re ommon to ll vritions In oth the ingress nd egress diretions, n NP sits etween the swith fri nd the physil interfe The swith fri rries trffi etween line rds The physil interfe my onsist of single link, suh s 1 G Ethernet or SONET, or olletion of slower links, suh s 1/1 M Ethernet or DSL To inrese the numer of instrutions nd memory opertions tht n e pplied to eh pket while meeting trget line rte, NPs re typilly orgnized s highly-integrted hipmultiproessors For exmple, Intel s IXP28 [1] fetures 16 pipelined proessors, lled miro-engines (MEs), eh of whih support 8 thred ontexts nd zero yle ontext swithes in hrdwre The hip lso integrtes 4 QDR SRAM ontrollers nd 3 Rmus DRAM ontrollers, long with mny other hrdwre units unrelted to queuing In line rds like this, oth SRAM nd DRAM re used to implement pket queues Queues nd their desriptors re kept in SRAM, while the pkets re kept in DRAM The sheduling disipline is implemented in softwre on one or more proessors 21 Pket Queues Pket queues re used to provide pket sheduling, QoS, nd other types of differentited servies to pket ggregtes Mny routers use three-level queue hierrhy, where the first represents physil ports, the seond represents lsses of trffi nd the lst level onsists of virtul output queues Eh ingress NP mintins queue for eh output port to eliminte hed-of-line loking; eh of these output port queues hs numer of lss queues ssoited with it in order to enle servie differentition nd QoS; eh of these lss queues onsists of per-flow virtul output queues whih llow individul flows to e shped, eg, y throttling unresponsive flows using ongestion Eh inoming pket is enqueued into some virtul queue, nd the sttus for the orresponding lss nd physil queues re updted to reord the tivity A similr sequene ours when pket is dequeued from virtul queue y the sheduler Sheduling is typilly rried out from root to lef; ie, first, the port is seleted ording to the port seletion poliy, then lss from the seleted port is hosen whih is followed y virtul queue seletion It is importnt to note tht one enqueue nd one dequeue re expeted during eh pket rrivl/deprture period Sine oth opertions involve updtes to shred queues, seriliztion n our In order to provide good memory utiliztion, virtul queues re typilly kept in linked list dt struture [17] Port nd lss queues, however, re only kept in linked list dt struture when the seletion poliy for lss nd virtul queue is ring-sed Round roin or weighted defiit round roin [18] re exmples of ring sed seletion poliies, where next seletion is the next link in the ring of tive queues A queue s sttus needs to e updted for every inoming nd outgoing pket, so tht sheduling n e rried out effiiently For exmple, mny pket sheduling lgorithms use queue oupnies s inputs For this reson, some rhitetures pss every enqueue nd dequeue ommnd to the sheduler, whih mnges its own lol queue sttus dtse This keeps the sheduler from either using stle informtion or mking frequent queue desriptor red requests 22 A Pket Proessing Pipeline Pket proessing is typilly implemented s pipeline onsisting of multiple proessor stges Whenever stge mkes hevy use of memory (eg, queue opertions), multiple threds re used to mintin good throughput despite reltively high memory ltenies The proessing pipeline generlly onsists of the following tsks, eh typilly mpped to its own proessor(s) 1 Pket ssemly Severl interfes deliver pkets in multiplexed frmes or ells ross different physil ports 2 Pket lssifition Inoming pkets re mpped to queue 3 Admission ontrol Bsed on the QoS ttriute of the queues, suh s mximum size, pkets re either dmitted or dropped 4 Pket enqueue Upon dmission, the pket is uffered in the DRAM, nd the pket pointers re enqueued to the ssoited queues Most rhitetures uffer the pket in DRAM t the first stge nd then dellote the uffer lter if the pket is not dmitted 5 Sheduling nd dequeue The sheduler selets the queues sed on the QoS onfigurtion, nd then pket is dequeued nd trnsmitted 6 Dt Mnipultion, Sttistis A module my perform sttistis olletion nd dt mnipultion sed on the onfigu-

3 rtion Pket reordering, segmenttion nd ressemly my lso e performed 23 Queue Opertions nd Prllelism Both the queue desriptors, onsisting of hed nd til pointers nd the queue length, nd the linked lists (ie, the queues) re stored in SRAM SRAM nd DRAM uffers re lloted in pirs, so tht the ddress of the linked list node in SRAM impliitly indites the pket ddress in DRAM Thus, the linked lists in SRAM only ontin next-pointer ddresses With this struture, every enqueue nd dequeue opertion involves n externl memory red followed y write opertion Rell tht sine the ess time of externl memory requires mny proessor yles, multiple threds re used to hide the memory lteny A system n e lned in this wy y dding proessors nd threds, so long s eh thred esses different queue As soon s threds strt essing the sme queue, the prllelism is lost nd ll opertions re serilized, sine every queuing opertion involves red followed y write, nd the write k is lwys sed on the dt tht ws red In the worst se, ll threds ompete for the sme queue nd progress is serilized, regrdless of the numer of proessors or threds As we will show, using n on-hip he for queue desriptors n improve this worst-se performne 24 Relted Work The importne of pket queuing in routers nd swithes hs een motivted nd disussed in the literture [14],[15][16] The mjority of reserh on high-performne memory systems for networking hs foused on optimizing pket uffer memory ndwidth Reserhers from Stnford University hve proposed severl shemes to uffer pkets into DRAM t very high speeds [5],[6][7],[1] A group t ICS FORTH hs proposed severl ASIC sed rhitetures to effiiently perform the pket uffering nd mnge lrge numer of queues t very high speeds [13] Another rnh of relted work hs foused on mximizing the utiliztion of ville DRAM tehnologies MKee et l hve proposed numer of mehnisms to mximize DRAM ndwidth in streming omputtions [11] Rixner et l hve proposed relted performne enhnement vi effiient memory ess sheduling [8] Tehniques to effetively redue DRAM lteny hve lso een proposed [9] Hsn et l hve reently shown n pproh to effiiently utilize the ville DRAM ndwidth on network proessor [4] The hllenges of pket uffer design nd DRAM utiliztion re importnt, ut orthogonl issues In this work, we diretly improve the performne of pket queue dt strutures whose worst-se performne limits the performne of existing progrmmle pket proessing systems The softwre-ontrolled Q rry struture in the Intel IXP 2XXX fmily of NPs is similr to the softwre-mnged queuing he disussed in Setion 33 In the IXP, the softwre he is mnged nd kept oherent y using CAM in proessor (ie, miro-engine) The CAM keeps trk of the mpping etween queue desriptor ID nd he entry, s well s the lotion of the LRU entry for evition purposes As shown in this pper, using hrdwre-mnged queuing he in the memory ontroller gretly inreses performne while lso improving the overll effiieny DRAM Controllers (Pket Dt) On-Chip SRAM SRAM Controllers Queue support P P P P P P P P P P P P P P P P Coproessors (eg,hsh, rypto) I/O Interfe (eg, SPI) Control Proessor Figure 2: Struture of the NP rhiteture with shred interonnet for memory nd other resoures Eh proessor (P) is ssumed to e multithreded The queuing susystem involves proessors, DRAM, nd SRAM HW-sed queuing support would e integrted t the SRAM ontrollers 3 MEMORY SYSTEM MODELS In this setion, we will desrie the struture of the queuing he nd its lterntives se system with no he or queuing support, dt he, nd softwre-ontrolled queuing he s well s how queue opertions re rried out on eh orgniztion We onlude this presenttion of memory models with rief summry of the lterntives 31 Arhiteture nd Memory Orgniztion Figure 2 shows the struture of our si system It is highly integrted system, feturing multiple proessors, eh multithreded, nd memory hnnels ll shring us-sed interonnet This generl orgniztion is ommon mong hip multiproessors nd network proessors [2] Sine we re modeling shred memory in multiproessor system, memory onsisteny is ritil issue Speifilly, our requirement is tht requests for prtiulr queue our in the orret order without orrupting the queue desriptor dt struture The queuing he synhronizes ess to given queue utomtilly For the other memory orgniztions, we model zeroost synhroniztion mehnism tht ssures ordered, oherent ess to ll queues A rel system would require some sort of softwre synhroniztion to mintin onsisteny, ut these osts re not ounted for in the non-queuing he models Adding relisti synhroniztion mehnism would gretly inrese the omplexity of the lternte models As we will see, the queuing he provides superior performne despite this disdvntge An enqueue opertion in the se model involves three memory referenes, ) initilly the queue desriptor is red, ) then the rriving pket is linked to the queue s til, nd ) finlly the updted queue desriptor (til) is written k The lst two referenes n e rried out onurrently, s they re independent writes A dequeue opertion lso involves three memory referenes, ) initilly the queue desriptor is red, ) then the next node of the hed is red (whih will eome the new hed), nd ) finlly the updted queue desriptor (hed) is written k All three referenes, in this instne, must e rried out sequentilly The time needed to omplete given memory referene depends on two ftors: 1) the externl memory lteny nd 2) the lteny

4 System type System prmeter Vlue Bse CMP prmeters Interonnet lok frequeny Memory ess time (round trip) Interonnet us w idth Proessor lok frequeny Interonnet dely (round trip) Synhroniztion server dely 4 MHz 4 ns 8 w ords 1 GHz 4 ns ns Memory lok frequeny 2 MHz Totl instrution yle time 1 ns Che ess time 25 ns CAM ess lteny 5ns Softw re he Totl instrution yle time 35 ns Queuing he Totl instrution yle time 5ns Tle 1: Prmeters hosen for the se multithreded CMP queuing system model of the shred interonnet etween the proessor nd the memory interfe This seond ftor is onsiderle nd n rivl or exeed externl memory lteny in highly-integrted CMPs, even in unloded systems (ie, those without interonnet ontention) 32 Generl-Purpose Dt Che A dt he n e pled etween the queue desriptor memory hierrhy nd the proessors It provides no diret support for queuing opertions, ut n shorten the lod/store lteny for opertions pplied to hed queue desriptors Sine there is little re-use in the linked list memory, the he only needs to hold the reently essed queue desriptors 1 In ddition, for good performne it must e le to servie multiple requests onurrently, nd hene the he must e lok-up free [3] The dt he improves queuing performne s long s there is some lolity in the queue desriptor esses Sine hit does not require n externl memory ess, the ost of seriliztion is redued However, the interonnet lteny persists 33 Softwre-ontrolled Queuing Che The ost of seriliztion n lso e redued with the ddition of smll linked list proessor (lso referred to s queuing engine) nd softwre-ontrolled he etween the queue desriptor memory hierrhy nd the proessors This is the pproh used in Intel s NPs The linked list reds nd writes will e hndled internlly y the he, resulting in interonnet lteny svings Eh thred sends n enqueue or dequeue ommnd to the he nd the he performs the link list red or write followed y the queue desriptor updte However, sine the he is softwre ontrolled, efore issuing n enqueue or dequeue ommnd, threds must ensure tht the queue desriptor is hed If it is not hed, then threds need to issue ommnds in order to evit n entry nd ring in the pproprite queue desriptor to the he Thus, threds issue two ommnds on miss nd one ommnd on hit, in order to servie n enqueue or dequeue request In ddition, the thred synhroniztion must e rried out y the threds 1 In situtions where the link list nd queue desriptors re stored in single physil memory, either ) ISA n hve speifi lod nd store instrutions whih use the he, or ) the he n e onfigured to hold the dt from n ddress rnge 34 Queuing Che A queuing he is smll fully ssoitive he with tightly oupled queuing engines It redues the interonnet lteny y enling the threds to issue only one ommnd for n enqueue or dequeue opertion The Queuing he internlly mnges the queue desriptor he nd performs the pproprite queuing opertions on hit or miss Furthermore, queuing he opertes in prllel nd epts ommnds from multiple threds (ie, it is lso lok-up free) When multiple requests re destined to single queue, it ensures tht ll requests re servied without ny ontention nd in the orret order Thus, individul threds need not mintin synhroniztion or pket ordering We onsider prtiulr queuing he design nd its implementtion osts in Setion 5 35 Summrizing the Queuing Su-systems The four lterntives re est differentited y onsidering how they 1) redue the numer of off-hip memory esses, nd 2) redue the numer of opertions sent ross the shred on-hip interonnet The three he-sed shemes ll redue externl memory esses, nd therey redue lteny, on hit this is the trditionl enefit of hing The two shemes tht employ queuing engines, the queuing he nd softwre mnged queuing he, redue the numer of memory ommnds sent y sending queue opertions rther thn the memory referenes tht implement them The softwre mnged he, however, must send he mngement opertions on misses, so the enefit is redued These differenes, long with some others, re quntified nlytilly in Setion Error! Referene soure not found 4 EXPERIMENTAL EVALUATION In this setion, we evlute the performne of eh queuing susystem under vriety of syntheti nd rel-world worklods Sine our gol is to mesure queuing performne, queuing susystem remins the ottlenek Our im is to show whih queuing orgniztion provides the est performne nd effiieny 41 Simultion Methodology We use mix of ehviorl nd RTL-style VHDL to simulte our system nd memory models RTL ws used to model the ontrol nd logi detils of the queuing he nd other units, nd ehviorl VHDL ws used to model proessor tivity, interonnet pths nd ritrtion, nd memory strutures We hve uilt se CMP model with hrdwre support for multithreding in ehviorl VHDL nd developed three vrints of queuing susystems round it 411 Bse Multithreded CMP model The prmeters hosen for our se CMP system is shown in Tle 1 These prmeters losely pproximte those found in Intel s IXP NPs In the dt he model, we dd he with support for the following prmeters: ssoitivity, line size, pity, numer of miss sttus holding registers (MSHRs), optionl write-llotion, write-through or write-k, nd LRU or rndom replement [2] In the softwre ontrolled he model we hve tthed the he rry nd queuing engine t the memory ontroller In the queuing he model, we hve dded the RTL design of the queuing he t the memory ontroller

5 Worklod type Nottion Distriution of λ ross queues Desription Logrithmi Lk Exponentil The weights of eh queue were exponentilly distriuted Thus, λ i = k* λ i-1 Uniformly rndom R Uniform Weights of ll queues were sme, λ Stritly uniform U n The rrivl proess, insted of eing Poisson, is round roin one, in whih queues send pkets in round roin order It results in no temporl lolity Set dominting SDn_k Two sets of uniform Single dominting SD1_k Two sets of uniform A set of n equl priority queues hd ptured k% of the ndwidth nd remining (N-n) queues took the remining ndwidth Thus, n* λ 1 = k/(1-k)* (N-n)* λ 2, where λ 1 nd λ 2 re the men inter-rrivl time of first nd seond set of queues respetively This is the limiting se of the previous worklod, wherein the vlue of n is one Tle 2: Desription of the syntheti worklods, their nottion nd the distriution of prmeters In most experiments, the sizes of the queuing he nd the softwre-ontrolled he were kept 32 entries, fully ssoitive, with 32 queuing engines However, the dt he ws onfigured to e 2-wy set ssoitive with 128 words This is euse this dt he onfigurtion hd ) the est performne to omplexity rtio, nd ) n re footprint equivlent to the queuing he nd softwre ontrolled he [12] 412 Synhroniztion As desried in Setion 3, synhroniztion etween queues is neessry when performing prllel opertions in order to prevent pket reordering nd to keep queue dt strutures oherent Of the strutures modeled in this pper, only the queuing he ounts for the overhed osts of synhroniztion, sine the ssoited logi ws implemented inside it The other models would require some kind of softwre-sed synhroniztion involving loks Our other models do not implement these softwre mehnisms, ut if they did the net effet would e greter opertion lteny nd ommunition ndwidth By not inluding the osts of synhroniztion in the non-queuing he solutions, we overestimte the performne nd effiieny of these models As we will see, the queuing he gives strong reltive performne despite this disdvntge 413 Worklods When performed olletively y multiple threds, queuing throughput n e highly sensitive to the worklod As we hve noted, system throughput is determined y the frtion of urrent opertions tht n proeed in prllel To evlute rnge of possiilities, we hve gthered olletion of syntheti nd relworld pket tres Eh tre hs different onentrtion of pkets to queues For rel-world worklods, we use two tres: network-edge tre from NLANR [22] tht onsists of trffi reorded on n OC-3 link etween University nd the Internet, nd ore Internet tre from CAIDA [21] tht onsists of trffi tken from n OC-48 link onneting two kone routers Trffi is mpped to the queues y pplying hsh funtion on the pket heder Our syntheti tres were designed to pture wide rnge of temporl lolity in queue referenes In order to generte vrious syntheti tres, we modeled pket rrivl nd deprture server At the rrivl server, pkets rrived for eh queue ording to Poisson proess [19] In order to model the vrying rtes of eh queue, we ssign eh queue different Poisson proess prmeter 2, λ The deprture server modeled n pproximte defiit round roin seletion 3 [18], wherein queue with higher rte nd klog sends not only more pkets ut is lso likely to send them in lrger ursts Our sets of syntheti worklods re desried in Tle Desription of the nottion in grphs nd tles We use the following nottion for eh experimentl urve: Worklod_typeSystem_typeChe_sizeChe_type The nottion for Worklod_type is desried in Tle 2 System_type is, ) sq for softwre-ontrolled he, ) n for se system, ) d for dt he, nd d) q for queuing he Thus, SD1_7sq32f indites tht ) the trffi type is single dominting queue tking 7% of the link pity, ) system is softwre-ontrolled he, ) the he rry size is 32, nd d) it is fully ssoitive 42 Results with Uniformly Rndom Trffi Figure 3 shows the queuing throughput, in millions of pkets per seond, of eh queuing su-system orgniztion on uniformly rndom worklod This sitution orresponds to the est-se senrio in multithreded system without hes, in whih ll opertions n operte in prllel In this worklod, there re lrge numer of tive queues, nd pkets re spred eqully Thus, there is no queue lolity or ontention, s refleted y the low, ner-zero hit rtes All models sturte t slightly over 8M opertions per seond The queuing he hs greter slope, however, nd hieves pek performne with only 12 threds s ompred to the 18-2 threds required y the others This result shows tht the queuing he is more effiient, ie needs fewer onurrent opertions, in hieving pek performne We lso note tht this performne level is pek performne hievle in system without he 2 1/ λ is the men of the exponentil rndom vrile of the Poisson proess 3 Defiit round roin (DRR) is lss of fir queuing lgorithm nd is widely used ommeril routers

6 Throughput, million pkets / se Rn Rsq32f Rq32f Rd1282wy Hit (dt $) No of threds Figure 3: Plots throughput versus numer of threds under uniformly rndom worklod The hit rte for he-sed systems is lso shown 43 Results with Weighted Rndom Trffi 431 A Single Dominting Queue Figure 4 reports throughput for tre onsisting of trffi dominted y single queue tht rries 3% of ll pkets, with the remining trffi spred uniformly over lrge numer of queues This senrio presents moderte mount of nrrow lolity, sine 3% of ll pkets trget single queue Therefore we see higher hit rtes, round 3%, nd more ontention Clerly, this lolity is dominted y the osts of seriliztion, sine performne for ll models is worse thn tht of the uniformly rndom se Here, the queuing he hieves superior performne due to its shorter seriliztion pth Hit rte (%) Figure 4 reports the throughput with 32 threds s the weight of the single dominting queue vries from 5% to 1% of the totl trffi As n e seen, the he-sed models see initil enefit s the weight inreses, rise to pek, nd then eventully desend to the worst-se performne levels The initil enefits re due to the inresed hit rtes while the eventul deline is due to seriliztion of trffi t single queue The queuing he hs the highest pek, nd mintins it over the widest rnge of weights But lolity eses to e enefit for the queuing he eyond weight of 2%, wheres the other he-sed models lose the enefit of lolity t round 1% As expeted, hit rtes inrese with the queue weight 432 A Set of Dominting Queues We now onsider worklods tht extend the lolity from one queue to set of queues Figure 5 shows throughput when set of 16 dominting queues ounts for 9% of ll trffi As expeted, hit rtes re high, etween 75% nd 9%, nd the hesed models hieve performne gretly in exess of the memory ndwidth limit This senrio represents roder form of lolity, where 16 different opertions re onurrently tive on verge The queuing he hieves greter performne with fewer threds due to its greter effiieny nd lower opertion lteny It hieves throughput of 15 M opertions per seond, representing improvement ftors of 115 nd 13 over the softwre queuing he nd dt he, respetively Figure 5 reports results for 32 threds s the numer of queues in the dominting set (9% trffi) rnges from 1 to 256 The grph shows tht ll he-sed models inrese with dominting set size until the he pity is met: 32 for the queuing he nd softwre queuing he, wheres the lrger dt he egins its deline t 64 By exmining the hit rtes, it n e seen tht throughput trks hit rte preisely The lrger dt he mintins higher hit rte, nd thus mintins higher throughput longer All models eventully onverge on their uniformly rndom performne levels 433 Exponentilly Distriuted Queue Weights Figure 6 reports throughput when queues hve exponentilly distriuted weights We use exponentilly distriuted weights euse they provide mens of weighting queues more relistilly thn previous tres As n e seen from the performne plots, the nrrower exponentil distriution shown in Figure 6 results in higher hit rtes nd throughput thn does the wider distriution Throughput, million pkets / se S1_3n S1_3sq32f S1_3q32f S1_3d1282wy Hit (dt $) Perent (%) Throughput, million pkets / se S1_xn S1_xsq32f S1_xq32f S1_xd1282wy Hit (dt $) No of threds = Hit rte (%) No of threds Weight of single dominting queue Figure 4: ) Plots throughput versus numer of threds under trffi with single dominting queue utilizing 3% of the link pity ) Plots throughput versus weight of single dominting queue for systems with 32 threds This simultes grdully inresing lolity We note tht smll weights improve the performne of he-sed systems nd s lolity inreses, seriliztion degrdes the performne

7 Throughput, million pkets / se S16_9n S16_9sq32f S16_9q32f S16_9d1282wy Hit (dt $) Hit rte (%) Throughput, million pkets / se No of threds = 32 Sx_9n Sx_9sq32f Sx_9q32f Sx_9d1282wy Hit (dt $) Hit rte (%) No of threds No of dominting queues Figure 5: ) Plots throughput versus numer of threds under trffi with set of 16 dominting queues, whih tke 9% of the link pity ) Plots throughput versus numer of dominting queues for systems with 32 threds nd the dominnt queues tke 9% of the link pity This simultes grdully deresing lolity We see tht s the numer of dominnt queues pprohes the he size, performne gets mximized nd fterwrds, s the hit rtes fll, the performne pprohes to the memory ndwidth Throughput, million pkets / se L11n L11sq32f L11q32f L11d1282wy Hit (dt $) Hit rte (%) Throughput, million pkets / se L11n L11sq32f L11q32f L11d1282wy Hit (dt $) Hit rte (%) No of threds No of threds Figure 6: ) Plots throughput versus numer of threds under trffi with exponentilly weighted queues nd exponent of 11 ) Plots throughput versus numer of threds under trffi with exponentilly weighted queues nd exponent of 11 It is evident tht trffi with n exponent of 11 results in higher hit rtes nd hene higher performne, euse the weight is distriuted suh tht there is roder lolity shown in Figure 6 The first distriution hs moderte mount of rod lolity; the seond spreds the weight over more queues, therefore the lrger dt he sees etter performne y pturing slightly lrger set of tive queues 44 Rel-World Worklod Results Figure 7 shows throughput for the edge nd ore tres respetively As expeted, the edge tre shows moderte lolity nd hit rtes, nd hene high-performne in he-sed systems In ft, the throughput results re similr to those seen in the nrrow L11 exponentil syntheti tre On edge trffi with 32 threds, the queuing he hieves throughput of 13 M opertions per seond, representing improvement ftors of 13 nd 175 reltive to the softwre queuing he nd dt he, respetively The ore tre hs little lolity, low hit rtes nd, hene, low throughput As expeted, this tre fetures mny tive queues Therefore, performne is similr to, ut slightly etter thn, the uniformly rndom se Under this worklod, the he-sed models ll onverge etween 9-95M opertions per seond One gin, the queuing he is more effiient in hieving pek performne It sturtes t 17 threds, ompred to 25 for the other he-sed models 45 Tuning Che Prmeters We hve performed experiments on severl onfigurtions of the dt he We hve hnged ) the evition poliy from LRU to rndom, ) the llotion poliy from write llote to nollote, ) the write poliy from write k to write through, nd

8 Throughput, million pkets / se Edgen Edgesq32f Edgeq32f Edged1282wy Hit (dt $) Hit rte (%) Throughput, million pkets / se Coren Coresq32f Coreq32f Cored1282wy Hit (dt $) Hit rte (%) No of threds No of threds Figure 7: ) Plots throughput versus numer of threds under rel-world trffi olleted t n edge router ) Plots throughput versus numer of threds under rel-world trffi olleted t the ore router It is evident tht the edge trffi hs higher degree of temporl lolity nd hene higher hit rtes nd etter performne d) line size of the he We didn t notie more thn single perent point hnge in the overll system performne with the first two hnges The third hnge, nmely the write poliy tully deteriorted the performne in the setup where the memory hd single shred red nd write us This is euse write through poliy performs redundnt writes, whih if not performed, ould hve een used for reds We hve lso oserved tht overll performne deteriorted s we inresed the he line size This is due to the lk of sptil lolity etween queues Thus, n inresed line size results in extrneous memory esses tht feth entries tht re not likely to e used Therefore, we onlude tht the est he onfigurtion for the queuing opertions under network worklods is single word per line nd write k poliy Allotion nd evition poliy don t hve signifint impt on performne 46 Summry In the preeding setions, the enefit of hing queue desriptor dt hs een shown To summrize these hing enefits, we mke the following points When ontention for queue is high, keeping the queue desriptor in n on-hip he shortens the seriliztion pth Therefore, worst-se performne is improved Of the models we onsidered, the queuing he hd the shortest seriliztion nd therefore the est worst-se performne It improves throughput y ftors of 31, 15, nd 21 over the system with no he, with softwre-ontrolled he nd with dt he, respetively When the numer of tive queues is muh greter thn the numer of queue entries, then hing provides little enefit However, the queuing he improves effiieny sine it hieves pek performne with the fewest numer of threds, rehing pek performne with only 12 threds, s opposed to the 18-2 required y the other orgniztions When the numer of tive queues is greter thn one ut not muh greter thn the he pity, then this rod lolity will gretly inrese throughput In effet, eh he entry n support onurrent opertion; this provides opertion throughput ove tht whih is hievle sed on externl memory ndwidth As the numer of tive queues exeeds the he pity, the request sequene ppers more rndom nd misses egin to dominte When the numer of tive queues is lose to the he size, the performne of the queuing he peks t pproximtely 15M pkets/seond As the numer of tive queues inreses, the throughput pprohes the 9 M pkets/seond vlue seen in uniformly rndom trffi 5 ANALYSIS AND DISCUSSION In this setion, we disuss dditionl quntittive metris tht n e helpful in evluting pket queuing memory system, inluding: worklod hrteriztion, opertion lteny, externl memory ndwidth, pket queuing instrution ount, nd on-hip ommunition ndwidth Tle 3 summrizes mny of these metris 51 Input Chrteriztion The input tres n e hrterized y the inter-rrivl times, where we refer to the inter-rrivl time s the time elpsed to servie ll other requests etween two requests for the sme queue Seriliztion ours only when the inter-rrivl time is less thn the enqueue or dequeue servie time of single thred, whih we will ll the loss threshold In fully ssoitive he, hits re gurnteed to our when the numer of inter-rriving pkets is less thn the he size, whih we will ll the gin threshold Aove this threshold, misses n our resulting in no time gin Furthermore, misses re more likely to our s the inter-rrivl time inreses By plotting the distriution of inter-rrivl times for given input tre, the performne of queuing system n e estimted For exmple in Figure 8, for the single weighted queue tre, ninety perent of the inter-rrivl times re less thn the servie time of single pket, inditing tht seriliztion is ourring On the other hnd, the mjority of inter-rrivl times for the rndom nd ore tres re ove the gin threshold, nd therefore he gives little enefit A lrge perentge of the inter-rrivl times for the tre in whih 8 queues re hevily weighted lie etween

9 Model Enqueue Lteny Dequeue Lteny Off-hip Bndwidth On-hip Bndwidth Instrutions Queuing Che Softwre Queuing Che Dt Che T e(hit) = 8 ns T e(miss) = 12 ns T e(hit) = 12 ns T e(miss) = 2 ns T e(hit) = 13 ns T e(miss) = 17 ns T d(hit) = 8 ns T d(miss) = 12 ns T d(hit) = 12 ns T d(miss) = 2 ns T d(hit) = 17 ns T d(miss) = 21 ns (1-h)*2T+T T 3-4 (1-h)*2T+T 3T 6 (1-h)*2T+T 3T 3 No Che T e = 17 ns T d =25 ns 3T 3T 3 Tle 3: Tle quntittively summrizing the hrteristis of eh of the four queuing su-systems Cumultive proility Interrrivl lteny (in no of pkets) S1_9 S8_9 L11 Edge Core R Figure 8: Plots the proility distriution of the verge interrrivl time of ll pkets within eh queue It is ler tht s we move from single dominnt queue to set of dominnt queues to exponentilly distriuted weighted queues nd relworld tres, the inter-rrivl time distriution eomes wider In the limit, rndom distriution hve the widest rnge of interrrivl times the loss nd gin thresholds, resulting in lrge time gins nd smll losses The exponentil nd edge tres hve out 5 perent of inter-rrivl times elow the gin threshold (onsidering he size of 32, nd therefore gin threshold of 32 pkets servied) nd smll frtion elow the loss threshold These trends re refleted in the throughput plots in the experiments 52 Off-Chip Bndwidth With no he, every queue opertion results in three off-hip referenes Eh enqueue opertion requires one red nd two writes to memory A dequeue requires two reds nd one write Thus on verge, if the system throughput is T, the required offhip ndwidth will e 15*T for oth reds nd writes For he sed systems with hit rte of h, the effetive ndwidth will get redued y h, nd will e (15-h)*T for oth reds nd writes 53 On-Chip Bndwidth A he-less system without ny hrdwre support for queuing uses the sme mount of ndwidth on-hip s off With the ddition of dt he, while the off-hip ndwidth is redued, the on-hip ndwidth stys the sme The softwre queuing he dds smll mount of on-hip ndwidth due to he mngement A queuing he signifintly redues the on-hip ndwidth euse n enqueue or dequeue requires only single instrution 54 Effiieny On the Intel IXP, the queue mnger softwre tht ontrols the Q rry struture in the SRAM ontroller (whih is similr to our softwre queuing he model) requires pproximtely 6 instrutions The dt he nd he-less systems eh use smller numer of instrutions sine only the instrutions to red nd write the queue desriptors, updte the queue desriptor, nd modify the links re required This requires pproximtely 3 instrutions A queuing he requires only 3-4 instrutions for either n enqueue or dequeue Thus, queuing he simplifies the progrmming of the pket proessor In ft, it frees up resoures whih re otherwise used for queue mngement 55 Slility nd Flexiility One of the primry dvntges of the queuing he model is its slility As hs een shown, system with queuing he is often le to hieve higher throughput with out hlf to twothirds s mny threds In ddition, queuing he elimintes the synhroniztion overhed whih is needed in the other models The softwre queuing he model requires dditionl onhip ndwidth for synhroniztion For exmple, the Intel IXP Q rry is kept oherent with CAM t single miro-engine; this restrits queue opertions nd mngement to single proessor Consequently, there is limit to the numer of threds tht n e used nd, hene, the mximum throughput of the system In queuing he, there is no suh limittion on either the numer of threds used for queuing or the lotion of these threds Thus, throughput n e improved y inresing the numer of threds nd inresing the size of the queuing he without lrge inrese in the on-hip ndwidth utiliztion Moreover, in the dt he nd he-less models, synhroniztion mehnism is needed to resolve ontention As we hve noted, we hve not ounted for the synhroniztion overheds of these systems, nd therefore performne loss is likely in rel implementtion 56 Implementtion Complexity A queuing he implementtion onsists of: fully ssoitive he to hold the reently essed queue desriptors, logi mintining the LRU entry for evition purposes, nd n rry of queuing engines In order to ensure orret servie order within eh queue, eh rriving request is pled into irulr uffer A queuing engine is llowed to hoose request from the uffer

10 Request register rry (Buffers requests in irulr order) Thred ID, Enqueue it, Vlid it, Pket pointer, Queue ID Queuing engine rry (Servies requester) Stores the requester ID urrently servied Requester in servie it Che rry Fully ssoitive Queuing engine Queuing engine thred_id, queue_id, enqueue, dequeue, pkt_ptr Cross onnet X 1 Queuing engine Queuing Engine Interfe (RD/WR) Cross onnet X wr_ptr_r Vlid its Queuing engine Round roin logi Finds first empty slot in the register rry from lst wr_ptr 'Z' wired-and logi 'Z' Queue ID from request register rry nd requester in servie its re fed into this network Contention Network (NxN mtrix struture) Selets from requester rry, requests whose queue IDs re not in servie 'Z' Rotting token is used to initite n engine Set the in servie it Pik request Evit LRU Red QD in he This stte mhine performs pproprite memory opertions to servie the enqueue nd dequeue requests token = '1' idle If not present (reset in use it) Servie request Pik next Pik next requester with sme queue ID Current queue ID eing servied Queue ID from requester rry = = = Current requester Round roin logi Next requester = NOT ((queue_id XOR queue_id) OR in_use OR in_use) Figure 9: High level orgniztion nd iruit digrm of the queuing he we implemented for the experiments Blok nme Flip Flop Count Comintoril Logi Gte ount Request register rry 176 2K Contention Network - 18K Queuing engine rry 512 5K Che rry 1K 45K Cross onnets - 75K, 1K Tle 4: Gte ount estimte of the exmple queuing he implementtion every lok yle This is hieved y using rotting token method In order to resolve ontention involving multiple requests for the sme queue, n NxN ontention network, where N is the mximum numer of requesting threds, selets the queues whih re not urrently in servie One of them is hosen y the queuing engine urrently holding the token Susequently, ll requests with the sme queue ID re utomtilly msked y the ontention network Thus, every engine servies unique queue sequentilly An evition nd n llotion re performed on miss Sine queuing engines re initited sequentilly, only one evition nd llotion in the he n our during given lok One request is servied, the engine heks the irulr uffer for jos pending for the queue it hs just served A liner serh is performed eginning from the position from where the lst request ws piked This ensures the orret servie order within eh queue The high level orgniztion nd iruit digrm of the queuing he is shown in Figure 9 Tle 4 summrizes the omintoril gte ount nd flip-flop ount estimtes of the queuing he, for onfigurtion in whih there re 32 prllel queuing engines nd fully ssoitive 32 word he rry The queue ID is ssumed to e 16-its wide, whih in turn implies mximum of 64K queues, nd pket pointers re 32-its

11 As we hve determined from our (non-optimized) smple implementtion, the omplexity of queuing he with 32 entries is omprle to the omplexity of 128-entry, 2-wy set ssoitive, lok-up free he tht n support 32 onurrent requests This indites tht queuing he is ost-effetive mens of improving queuing performne in roust wy 6 CONCLUSION This pper desries nd evlutes the queuing he, hrdwre he with tightly-oupled queuing engine We hve shown tht queuing he n improve pket queuing performne over wide vriety of syntheti nd rel-world worklods reltive to numer of lterntive memory models The queuing he improves performne y reduing the numer of on- nd off-hip requests generted, nd therey reduing opertion lteny We ompre the queuing he to: he-less system with no support for queuing, system with softwre-ontrolled queuing he (similr to the mehnism used in the IXP 2XXX network proessors), nd system with dt he Under worst-se onditions, when ll pkets elong to single queue, the queuing he provides throughput improvement ftors of 31, 15, nd 21 over these models, respetively Under uniformly rndom trffi, ll orgniztions sturte externl memory ndwidth nd hieve the sme performne, ut the queuing he shows greter effiieny y requiring 33%-4% fewer threds to hieve mximum performne As expeted, when lolity exists in worklod, ll he-sed models see improved performne In rel-world tre of edge Internet trffi, the queuing he improved throughput y ftors of 26, 13 nd 175, respetively The queuing he lso results in gins in on-hip ndwidth, offhip memory ndwidth, nd progrmmer ese-of-use Finlly, our RTL-style VHDL implementtion demonstrtes tht the footprint of queuing he is not too lrge, mking it highly osteffetive mens of improving queuing performne 7 ACKNOWLEDGMENTS This work ws supported in prt y NSF grnts CCF-4312 nd CNS nd y gift from Intel Corp REFERENCES [1] M Adilett, et l The Next Genertion of Intel IXP Network Proessors, Intel Tehnology Journl, vol 6, no 3, pp 6-18, Aug 22 [2] Ching M, Sohi G, Evluting Design Choies for Shred Bus Multiproessors in Throughput-Oriented Environment, IEEE Trnstions on Computers 41(3), pp , 1992 [3] Christoph Sheurih, Mihel Duois, Lokup-free Ches in High-Performne Multiproessors, J Prllel Distri Comput 11(1), pp 25-36, 1991 [4] Jhngir Hsn, Stish Chndr, T N Vijykumr, Effiient Use of Memory Bndwidth to Improve Network Proessor Throughput, The 3th Annul Interntionl Symposium on Computer Arhiteture, June 9-11, 23 [5] S Iyer, R R Compell, nd N MKeown, Designing Buffers for Router Line Crds, Stnford University Tehnil Report - TR2-HPNG-311, Stnford, CA, 22 [6] S Iyer, R R Kompell, nd N MKeown, Anlysis of Memory Arhiteture for Fst Pket Buffers, IEEE HPSR 2, Dlls, My 21 [7] G Shrimli nd N MKeown, Sttistil Gurntees for Pket Buffers: The Monolithi DRAM Cse, Stnford University HPNG Tehnil Report - TR4-HPNG-263, Stnford, CA, 24 [8] S Rixner, W J Dlly, U J Kpsi, P Mttson, nd J D Owens, Memory ess sheduling, In Proeedings of the 27th Annul Interntionl Symposium on Computer Arhiteture, pp , June 2 [9] W Lin, S Reinhrdt, D Burger, Reduing DRAM Ltenies with n Integrted Memory Hierrhy Design, In Pro 7th Int symposium on High-Performne Computer Arhiteture, Jnury 21 [1] Y Joo nd N MKeown, Douling Memory Bndwidth for Network Buffers, Pro IEEE Infoom 1998, vol 2, pp , Sn Frniso [11] SA MKee, W Wulf, JH Aylor, RH Klenke, MH Slins, SI Hong, DAB Weikle, Dynmi Aess Ordering for Stremed Computtions, IEEE Trns on Computers, vol 49, no 11, pp , Novemer 2 [12] R L Lee, P C Yew, D H Lwrie, Multiproessor he design onsidertions, Proeedings of the 14th nnul interntionl symposium on Computer rhiteture, p , June 2-5, 1987 [13] A Nikologinnis, M Ktevenis, Effiient Per-Flow Queuing in DRAM t OC-192 Line Rte using Out-of-Order Exeution Tehniques, Pro IEEE Interntionl Conferene on Communitions (ICC'21), Helsinki, pp , June 21 [14] S-T Chung, A Goel, N MKeown, B Prhkr, Mthing output queuing with omined input nd output queued swith, in Pro IEEE INFOCOM, 1999 [15] MG Hluhyj, MJ Krol, Queuing in high-performne pket swithing, IEEE J Sel Ares Commun 6 (9), , 1988 [16] Sleem N Bhtti nd Jon Crowroft QoS-Sensitive Flows: Issues in IP Pket Hndling, IEEE Internet Computing, vol 4, no 4, pp 48-57, July 2 [17] J-G Chen, et l Chpter 14 Implementing High- Performne, High-Vlue Trffi Mngement Using Agere Network Proessor Solutions, in Network Proessor Design, volume 2, Morgn Kufmnn, 24 [18] M Shreedhr nd George Vrghese, Effiient fir queuing Using defiit round-roin, IEEE Trnstions on Networking, Vol 4, No 3, pp , June 1996 [19] V Frost nd B Melmed, Trffi Modeling for Teleommunitions Networks, IEEE Communitions Mgzine, 32(3), pp 7-8, Mrh, 1994 [2] Aln Jy Smith, Che Memories, ACM Computing Surveys (CSUR), v14 n3, p473-53, Sept 1982 [21] Bkone pket heder tres t OC192 nd OC48, olleted t Coopertive Assoition for Internet Dt Anlysis (CAIDA), [22] Bkone nd edge pket heder tres olleted from the Internet Mesurement, Internet Anlysis, Ntionl Lortory for Applied Network Reserh (NLANR),