Less Pessimistic Worst-Case Delay Analysis for Packet-Switched Networks

Transcription

1 Less Pessimisic Wors-Case Delay Analysis for Packe-Swiched Neworks Maias Wecksén Cenre for Research on Embedded Sysems P O Box 823 SE Halmsad maias.wecksen@hh.se Magnus Jonsson Cenre for Research on Embedded Sysems P O Box 823 SE Halmsad magnus.jonsson@hh.se Absrac The rapid growh of disribued real-ime sysems creaes a need for cheap and available nework soluions while sill fulfilling he real-ime requiremens. In his paper we propose a mehod for less pessimisic delay analysis for packe swiched firs come firs serve nework, when knowing he inervals of possible message generaion. Experimens show ha he proposed mehod generaes he expeced resuls according o heoreical limiaions of he experimen cases. The experimens also show ha he proposed mehod could be pracically used for non-rivial sysems. Suggesions are given for fuure work on how o relax raffic requiremens and how o cope wih circular dependencies. 1. Inroducion As he size of real-ime sysems grows, from single board compuers o blade sysems and even comprising whole local area neworks, he requiremen for cheap and available real-ime communicaion has increased rapidly. As commercially off he shelf (COTS) neworking equipmen usually have no or limied suppor for hard real-ime guaranees here is a need for compeen analysis ools. In his paper we presen a mehod for calculaion of he packe delay in muli-hop swiched neworks when he packe offse is aken ino accoun. The resul is a igher predicion of he delay compared o general mehods used for packe delay analysis. Requiring a bounded packe offse migh seem like a very sric requiremen bu when i comes o real-ime sysems deerminism is ofen preferred over performance and flexibiliy. On he oher hand, no inelligence is required in he nework, jus simple COTS, firs come firs serve (FCFS), sore and forward, packe swiches. The only raffic shaping ha akes place is performed in he end nodes wihou any aleraions of he nework proocols. Wors-case iming analysis of packe swiched mulihop neworks have in many cases relied on raffic regulaion, deadline scheduling or similar in inermediae nodes in some form [1][2][3][4], or have focused on special cases of he raffic disribuion [5]. However, in [6] he auhor presens a generic mehod for calculaions of he delays in a muli-hop swiched nework where raffic shaping, in he form of earlies deadline firs scheduling, is only allowed in he source nodes. The main drawback wih his soluion is ha i is sill somewha pessimisic, aking ino accoun siuaions ha will no occur during runime. Our suggesed soluion is o consider he packe offses which limi he exisence of packes o small ime windows, which reduce he pessimism because of he increased knowledge of where and when a packe will exis. The argumen for his soluion is ha he designer of a real-ime sysem migh prefer sabiliy and predicabiliy over flexibiliy and performance. The res of he paper is organized as follows: In Secion 2 he assumpions and sysem definiion for he curren work are oulined and described. In Secion 3 he algorihm for calculaion of he packe delay is presened. In Secion 4, he parameers for he experimens are defined, while he experimens and he resuls are presened in Secion 5. In Secion 6 he whole paper and is resuls are concluded. Finally in Secion 7 a number of differen suggesions for fuure work are presened. 2. Assumpions and sysem definiion The focus of his work is o perform packe delay calculus for real-ime compuer sysems wih muliple nodes conneced by a packe swiched nework wih predicable rouing, for he case where he inerval of possible packe generaion is known for each packe. The nework consiss of simple sore and forward, FCFS, oupu queued swiches ha allow predicable rouing, e.g. source rouing or saic rouing ables. I is also assumed ha all he resource scheduling is performed off-line and ha he only ype of real-ime conrol is of he release ime for he packes from he nodes. The analysis assumes a single op prioriy raffic class, bu his could easily be generalized o allow prioriized raffic and raffic classes of lower prioriy. Despie his /8/$ IEEE 1213

2 limiaion of conrol he goal of his work is o derive he wors-case end-o-end delay for any packe enering he nework (see Figure 1). Packe-swiched nework Figure 1. A packe eners he nework from a source node a a known ime. How large will he wors-case end-o-end delay be? The sysem consiss of N processor nodes, no necessarily of he same sor, conneced by a swiched nework. The nework is collision-free in he sense ha only poin-o-poin links are used, and all bidirecional links are full-duplex links. The swiches are oupu buffered, suppor sore and forward and handle packes in FCFS order. The buffers in he swiches are assumed o be large enough o avoid any buffer overflow for he raffic paern applied. When passing a swich, s, here is an non-deerminisic bu bounded swich-dependen delay, X s. The cabling is considered o consis of shor cables of similar lengh wih a propagaion delay of p i for link i. Since he swiches are assumed o handle packes by sore and forward, a ransmission delay, r i,j, is inroduced for each hop, equaling he ime i akes for a packe of size j bis o leave he sending queue ono link i, where r i,min is he delay for a packe of minimal size. Any node in he sysem can be scheduled o communicae wih any oher node in he sysem o which here is a pah. The communicaion beween wo nodes consiss of packes wih real-ime consrains, wih a leas he earlies allowed sar ime (EST) for sending a packe o he desinaion specified. To be able o handle packes wih real-ime consrains i is required ha he nodes in he sysem are clock synchronized down o a deviaion for any clock o less han from he global ime T. This means ha a packe sen from a lae node o an early node will seem o arrive 2 laer han if all nodes were perfecly synchronized. Because of node specific behavior, such as operaing sysem ask inerference or cache or pipeline acceleraion indeerminism, a packe scheduled for ransmission a local ime will sar is ransmission in he frame o +, where is he node-specific wors-case delay for such delays. I is assumed ha all packes leaving a node have been off-line scheduled, which resuls in ha he packe order from each node is known and preserved. In oher words, if a packe is ready o be sen ou from he node earlier han scheduled, ransmission will sill be delayed o he release ime. Alhough he end nodes are synchronized wihin a known facor,, and all raffic has been off line scheduled, he packes are sen wihou any form of synchronizaion or raffic shaping when hey once have enered he nework. This leads o a queuing delay, c i, as a packe is passing a swich oupu queue ou ono link i. However, because of he resricion on how packes are allowed o be released from a node (no ahead of ime and in non overlapping order) he queuing delay for packes enering he ougoing link of a node will be. An overview of he delay componens for a packe w of size w bis, going from node N1 o node N2 via he swich S1, is seen in Figure 2. Figure 2. Example of delay componens affecing a packe. 3. Algorihm To be able o calculae a bounded wors-case delay for a packe being sen over a nework i is essenial ha all inerference leads o a bounded delay. This is he case in he fully packe-swiched neworks assumed. The reason for his is ha all communicaions are buffered in he sense ha no wo nodes are compeing for he same link resource a any ime, bu raher queued in order o access he link sequenially. The queuing delay will be possible o calculae and will depend on inerference from packes from oher links sharing he same oupu buffer. However, he delay ha could be inroduced when passing a swich sill depends on possible inerference from packes on oher links and he chosen packe queuing order. As saed earlier, in his paper he queuing order is assumed o be FCFS. To be able o calculae he queuing delay for a cerain packe w in he buffer B i, conneced o swich s via he link i, we have o know he earlies saring ime (EST) and laes finishing ime (LFT) for all possibly inerfering packes as his will affec he oal delay. To calculae he EST for nex hop, EST, for packe w as i eners he link i, we jus add he minimal possible delay for he hop o he EST. The minimal per hop delay for a packe is he case where no packe queuing delay occurs and hus equals he sum of he swich dependen delay, X s, he propagaion delay, p i, and he minimal ransmission delay, r i,min (see Equaion 1). EST ' EST + X s + pi + ri, MIN = (1) To calculae he LFT for he nex hop, LFT, he worscase delay has o be calculaed. Besides he swich dependen delay, X s and he propagaion delay, p i, he wors-case ransmission delay, r i, w, where w is he 1214

3 packe size in bis, has o be aken ino accoun along wih he queuing delay, c i, caused by inerference wih oher packes compeing for he same buffer (see Equaion 2). LFT ' + c (2) = LFT + X s + pi + ri, w To be able o compee for he buffer he compeing packes mus be able o ener he queue a he LFT for packe w a laes. This leads o he fac ha he LFT for a packe w in buffer B i depends on all packes desined for he buffer B i wih an EST ha is less or equal o he LFT of w. To calculae he LFT for packe w, LFT w, we need o know he wors-case buffer load a he LFT of w, LFT w, since his is he laes ime a which packe w will be queued and hus no longer affeced by inerfering packes afer ha. The queuing delay c i will equal he ransmission delay for all daa in he buffer B i a LFT w. To calculae he buffer sae here is acually no need o calculae he conribuion of each and every packe individually, he flows from each inpu could be generalized o wors-case daa flows called work loads. To generae a workload for a packe a from a link i o a buffer B j, we simply assume ha he packe will be sen as igh as possible owards he LFT w. A workload for a single packe a, from he link i, wih he EST equal o EST a, a ransmission delay of r i, a and a LFT equal o LFT a have hree possible oucomes: If LFT a hen he workload will range from LFT a -r i, a o LFT a, see Figure 3. Figure 3. LFT a If EST a + r i, a LFT w LFT a hen he workload will range from LFT w -r i, a o LFT w, see Figure 4. i In he case where a packe workload has no overlap he workload is simply he sum of he packe workloads. If wo or more packe workloads are overlapping he resul will be a single larger workload. For example, consider he wo packes a and b ha have overlapping packe workloads; he resuling workload will range from MAX(LFT a, LFT b ) r i, a r i, b o MAX(LFT a, LFT b ), see Figure 6. Figure 5. EST a < EST a +r i, a Figure 6. Overlapping packe workloads. Wih all link workloads calculaed i is now possible o calculae he buffer sae. This is done by adding all workloads a incoming links and operaing a smoohing operaor over he resuling funcion o simulae he buffer behavior, ha is; consume one uni daa for each ime uni if daa is available. When LFT w is reached he buffer could be empy which means ha he queuing delay was zero or he buffer could hold a number of daa unis ha mus be consumed from he buffer before he packe w will be sen. The ransmission delay for he amoun of daa lef in he buffer will equal he queuing delay, see Figure 7. Link Link Buffer Figure 4. EST a + r i, a LFT w LFT a And if EST a + r i,min < EST a +r i, a hen he workload will range from EST a o LFT w, see Figure 5. To generae he workload for a whole incoming link we have o consider wo cases, eiher he packe workloads are overlapping or hey are no overlapping. Buffer Figure 7. The workload funcions are added and smoohed ou. 1215

4 On a side noe; a possible pracical problem when designing he algorihm is ha an evaluaion of he delays for each possible poin of ime beween and LFT w will lead o an algorihm complexiy dependen on he iming granulariy. However, here is no need o evaluae he delay funcions in all possible poins of ime; i is sufficien o check he poins where any change of he underlying parameers are made. In pracice his means ha he number of evaluaion poins for each packe w is approximaely he number of packes wih an EST less or equal o LFT w. buffers depend on boh packes from upward buffers and downward buffers in he ree (see Figure 1). Our mehod o guaranee ha here are no unsolved dependencies a any ime is o sar he calculaions in he lowes layer and calculae delays for all upward buffers. When no more upward buffers are available he downward buffers are reaed, saring a he roo (op) node and working downwards hrough he layers. 4. Experimen se-up To be able o use he algorihm described in he previous secion o calculae individual packe delays, one of he requiremens is ha he delays for all packes he calculaion depends on mus be previously calculaed. The feasibiliy of ha ask is srongly dependen on he raffic paern, opology and he rouing algorihm. In his paper we will make assumpions abou he opology and rouing algorihm ha simplifies he calculaions, while a furher discussion on he general case will be found in he fuure work secion. Figure 9. Packes in he upward buffers only depend on packes from he lower levels. Figure 8. The used inernal connecions of a swich in a shores pah roued binary ree. The opology chosen for he experimens is a binary ree, wih he compuaion nodes in he leaves and swich nodes in he res of he ree. This is done for clariy reasons; any ree opology could be used wihou any modificaion of he algorihm. Shores pah rouing is used. This combinaion leads o a number of ineresing properies. The shores pah in a binary ree (or any nonfa ree acually) resuls in a unique pah for any source, desinaion pair. I is also possible o generae a relaively simple calculaion order for he delay analysis of he packes ha guaranees ha all dependencies (LFTs and ESTs for earlier seps) are solved. When analyzing how he shores pah algorihm roue packes over he opology i is eviden ha he buffers in he swiches of he opology will be of one of wo ypes, eiher an upward buffer or a downward buffer (see Figure 8). The delay analysis of packes in he upward buffers will only depend on packes from lower layers in he ree (see Figure 9) while packes in he downward Figure 1. Wih all upward raffic solved, he downward raffic will only have unsolved dependencies in higher levels. Besides opology and rouing algorihm he operaing sysem laencies, clock drif, cable delay, link daa rae and inernal swich delay have o be deermined. Typical real-ime operaing sysems have laencies in he range of microseconds o ens of microseconds. For his experimen i is assumed ha he wors-case operaing sysem delay is 1 microseconds, which is realisic [7]. The nodes are assumed o be clock synchronized bu in pracice here will be a clock drif in beween synchronizaions. However, ypical clock drif in a clock synchronized wide area real-ime sysem range from hundred nanoseconds up o microseconds [8], so in his experimen i is assumed ha he clock drif is negligible. I is assumed ha all cables in he sysem are shor, less han a meer, and of approximaely same lengh which resuls in a cable delay in he nanosecond range, which is negligible. The inernal swich delay is he wors-case delay for a packe passing hrough a swich excluding he queuing delay. This facor is in he micro second range and assumed o be idenical for all swiches and is se o 1 microsecond. Since i is assumed ha sore and 1216

5 forward swiches are used here are no resricions on allowing links of differen daa rae (i is suppored by he algorihm as well) bu o keep hings simple in he experimen se-up all links in he sysem will have a daa rae of 1 Mbi/s. For comparison, he duraion of a packe wih he seleced daa rae of 1 Mbi/s range from approximaely en microseconds up o approximaely 1 microseconds. In our experimens we simulae balanced binary ree opologies wih n=2 N leaf nodes and hus 2 N+1-1 nodes. This way he number of leaf nodes uniquely describes he opology. Traffic for he experimen should be randomly generaed, meeing he requiremens for he packes; fixed ransmission order and ha packes from he same source does no overlap. The parameers of a packe are source node, desinaion node, packe size, EST and LFT. To generae hese parameers for a cerain raffic inensiy, TI, while guaraneeing he requiremens, he following algorihm is used: 1. Randomly generae a source node, s, in he range [, n-1] 2. Randomly generae a desinaion node, d s a. Randomly generae a desinaion offse, o d, in he range [1, n-1] b. Calculae he desinaion d as he wrap around sum of he source and he offse, ((s + o d ) % n ) 3. Randomly generae a packe size, p, in he range [512, 12] bis 4. Randomly generae an EST a. Ge a ime offse, o, by rerieving he LFT for he las packe generaed a node s, or if his is he firs packe b. Randomly generae a packe o packe ime inerspace, g, in ms wih he expeced value ((1/TI)-1) c. Calculae he EST as g + o d. Calculae he LFT as he sum of EST, he operaing sysem delay, φ, and he ransmission delay for a packe of size p. This is repeaed unil here are packes wih a LFT larger han he experimen lengh for all source nodes. The randomizaion disribuions could of course be of any ype, bu in our case we decided on recangular disribuions. 5. Experimens Given he experimen seup oulined in previous secion, a number of experimens were performed for opologies wih 4, 8, 16, 32 and 64 leaf nodes. Random packes were generaed a a given raffic inensiy and ransmied over he nework. Above some level of raffic inensiy he buffers in he ho spos will receive more daa on average han hey are able o ransmi, hus he buffers will saurae. This will show very clearly when analyzing he wors-case packe delay, since he buffers are sauraed, he delay will increase rapidly. To calculae his poin we formulae he funcion for he average raffic inensiy TOP hrough he op node of a balanced binary ree opology. According o previous secion he desinaion for any packe is evenly disribued which means ha wih n end-nodes in oal here will be n/2 possible desinaions where he packe will pass hrough he op node. The probabiliy ha a packe will pass hrough he op node hen equals o he number of posiive oucomes (n/2) divided by he number of possible oucomes ha is n-1. The raffic inensiy a each source node is TI and sauraion occurs when TOP is equal o 1 (see Equaions 6 and 7). n 2 n TOP = TI = 1 (6) n 1 2 n 1 TI = (7) 2 n 4 Evaluaed for he opologies included in he experimens i gives he following resul: Leaf nodes Sa. level 4 75% 8 44% 16 23% 32 12% 64 6% Table 1. Sauraion levels for differen number of nodes in he opology. If an experimen is repeaed wih he same parameers excep for he lengh of he simulaion he resuling wors-case delays will be close o he same in average for raffic inensiies lower han he sauraion level for he opology. However, for raffic inensiies greaer han he sauraion level he longer experimen will show a greaer increase in averaged wors-case delay, since here are sauraed buffers. We decided o double he experimen lengh for each parameer se and plo he wo curves side by side for comparison, resuling in experimen lenghs of 1 ms and 2 ms respecively. For raffic inensiies below he sauraion level he boh curves should keep close ogeher and above he sauraion level hey should be spread apar. The experimens were repeaed 1 imes for each parameer seing and he average of he wors-case delay was ploed agains he raffic inensiy. In Figures i is seen ha he delays sar increasing before he calculaed sauraion poins, which was expeced behavior. For he experimen in Figure 11 he delay seems o increase linearly unil approximaely a raffic inensiy of 6-7%, afer which he delay urns ino an exponenial growh. In Figure 12 he change seems o come a a raffic inensiy of 35%, in Figure

6 Wors case end-o-end delay nodes, 1 averaged runs Figure 11. The experimens for opologies wih 4 nodes show a clear increase in delay a he sauraion level (75%). Wors case end-o-end delay nodes, 1 averaged runs Figure 13. The experimens for opologies wih 16 nodes show a clear increase in delay a he sauraion level (23%). Wors case end-o-end delay nodes, 1 averaged runs Figure 12. The experimens for opologies wih 8 nodes show a clear increase in delay a he sauraion level (44%). a a raffic inensiy of 2%, in Figure 14 a a raffic inensiy of 1-11%, and in Figure 15 a a raffic inensiy of 5-6%. 6. Conclusions The aim of his work has been o ake advanage of he knowledge abou packe offses in he end nodes o generae igher end-o-end delay esimaes for packe swiched raffic in COTS swiched neworks. When experimenally evaluaed, our algorihm shows resuls ha behave as expeced from he heoreical limiaions for he ype of neworks ha were used. I has also been shown ha he algorihm is applicable for nonrivial cases consising of up o 64 end-nodes wih inensive raffic paerns. Alhough he focus for his paper was o analyze he delay for single packes i is also possible o analyze he wors-case buffer uilizaion or for how long he sysem mus remain silen o recover afer a burs. 7. Fuure Work The basis of his work was o formulae an algorihm ha made i possible o calculae igh or non pessimisic boundaries on packe delays in swiched neworks. As Wors case end-o-end delay 32 nodes, 1 averaged runs Figure 14. The experimens for opologies wih 32 nodes show a clear increase in delay a he sauraion level (12%). Wors case end-o-end delay 64 nodes, 1 averaged runs Figure 15. The experimens for opologies wih 64 nodes show a clear increase in delay a he sauraion level (6%). shown in earlier secions his is possible in a deerminisic number of seps. However, some of he requiremens assumed in his paper migh seem sric. In his secion we will explain how some of hese requiremens could be relaxed. One of he requiremens on he end nodes was ha all raffic was off-line scheduled and ha packes are of a known order and non-overlapping. This requiremen was added o ge a saring poin in he calculaions since each packe mus have an EST and a LFT. However, as long as he packes on he firs link have heir EST and a 1218

7 LFT calculaed neiher order nor overlap avoidance is needed. To sill be able o calculae hese parameers we have o model he scheduling algorihm used in he end nodes. If i is assumed ha he scheduler is a simple FCFS scheduler, i is he case ha his could be modeled using one of he swich nodes, see Figure 16. The parameers needed for he scheduler would be he release ime (i.e. EST) and he deadline for sending (i.e. LST). P P P P P P S P P P P P P Figure 16. By inroducing a virual swich, he exac packe order does no have o be known. The resricion in using sore and forward swiches could easily be relaxed since his is only affecing he iming variables ha are added in each hop beween he swiches. So, wih minor changes he algorihm will suppor cu hrough rouing as well. Similarly for he choice of FCFS queuing, as long as inerference is limied o he live span of he packe, any reasonable sraegy should be possible o adap o. Anoher resricion inroduced in his paper is he opology and rouing resricion. This was inroduced o make i very simple and sraigh forward o find he calculaion order and avoiding circular dependencies. Insead of creaing a siuaion where a calculaion order is given i is possible o creae a funcion ha checks he number of unsolved dependencies a cerain packe in a cerain buffer has for he momen. This makes i possible o sor ou hose packes ha do no have any unsolved dependencies and calculae heir LFT. However, if he opology or rouing requiremens are relaxed his could resul in circular dependency chains which preven he hierarchical solving. The suggesed soluion for his is o break deeced circular dependencies by inroducing a pessimisic esimae for one of he packes in he dependency chain. The choice of he esimae could simply be he case where all unknown packes wih dependencies are assumed o generae wors possible blocking. This is possible since, alhough he exac live S S span migh be unknown, he wors-case workload of he packe is known, which makes i possible o calculae he LFT ieraively. When he maximum change in LFT of he packes in he chain is below a cerain facor he ieraions are sopped. Given ha he iniial esimae is ruly pessimisic he resuls down he dependency chain will never be opimisic since a pessimisic inpu always generaes a pessimisic oupu. I migh be noed ha if he difference in LFT for any packe reaches his means ha a sable soluion has been reached. As an alernaive here are oher mildly resricive soluions for avoidance of dependency chains, for example x-y rouing or any oher deadlock free proocol. References [1] Ferrari, D.; Verma, D.C., "A scheme for real-ime channel esablishmen in wide-area neworks," Seleced Areas in Communicaions, IEEE Journal on, vol.8, no.3, pp , Apr 199 [2] Palencia, J.C., Harbour, M.G., "Response ime analysis of EDF disribued real-ime sysems," Journal of Embedded Compuing, vol.1, no.2, pp , 25 [3] Qin Zheng; Shin, K.G., "On he abiliy of esablishing real-ime channels in poin-o-poin packe-swiched neworks," Communicaions, IEEE Transacions on, vol.42, no.234, pp , Feb/Mar/Apr 1994 [4] Kandlur, D.D.; Shin, K.G.; Ferrari, D., "Real-ime communicaion in muli-hop neworks," Disribued Compuing Sysems, 1991., 11h Inernaional Conference on, pp.3-37, 2-24 May 1991 [5] Lee, K.C.; Lee, S.; Lee, M.H., "Wors Case Communicaion Delay of Real-Time Indusrial Swiched Eherne Wih Muliple Levels," Indusrial Elecronics, IEEE Transacions on, vol.53, no.5, pp , Oc. 26 [6] Fan, X., Real-Time Services in Packe-Swiched Neworks for Embedded Applicaions. Göeborg: Chalmers Universiy of Technology. Docoral hesis, 27 [7] Vicor Yodaiken e al., "RTLinux/RTCore dual kernel real-ime operaing sysem", FSMLabs, Whie Paper, Mar 23 [8] Schossmaier, K.; Loy, D., "An ASIC supporing exernal clock synchronizaion for disribued realime sysems," Real-Time Sysems, 1996., Proceedings of he Eighh Euromicro Workshop on, pp , June