A Simple Approximation to Minimum-Delay Routing

Transcription

1 A Smple Approxmaton to Mnmum-Delay Routng Srnvas Vutukury Computer Scence Department J.J. Garca-Luna-Aceves Computer Engneerng Department Baskn School of Engneerng Unversty of Calforna, Santa Cruz, CA Abstract The conventonal approach to routng n computer networks conssts of usng a heurstc to compute a sngle shortest path from a source to a destnaton. Sngle-path routng s very responsve to topologcal and lnk-cost changes; however, except under lght traffc loads, the delays obtaned wth ths type of routng are far from optmal. Furthermore, f lnk costs are assocated wth delays, sngle-path routng exhbts oscllatory behavor and becomes unstable as traffc loads ncrease. On the other hand, mnmumdelay routng approaches can mnmze delays only when traffc s statonary or very slowly changng. We present a near-optmal routng framework that offers delays comparable to those of optmal routng and that s as flexble and responsve as sngle-path routng protocols proposed to date. Frst, an approxmaton to the Gallager s mnmum-delay routng problem s derved, and then algorthms that mplement the approxmaton scheme are presented and verfed. We ntroduce the frst routng algorthm based on lnk-state nformaton that provdes multple paths of unequal cost to each destnaton that are loop-free at every nstant. We show through smulatons that the delays obtaned n our framework are comparable to those obtaned usng the Gallager s mnmum-delay routng. Also, we show that our framework renders far smaller delays and makes better use of resources than tradtonal sngle-path routng. 1 Introducton The standard approach to routng n computer networks today conssts of computng a sngle shortest path from a source to each destnaton usng some heurstc lnk-cost metrc, whch s typcally not drectly assocated wth the transmsson and queueng delays over lnks and paths. A less common approach to routng s that of defnng the routng problem as an optmzaton problem (e.g., multcommodty problem [5]) wth a specfc objectve functon, such as mnmzng delays or maxmzng throughput, and solvng the problem usng any of several known optmzaton technques. These two tradtonal approaches to routng have nherent strengths and drawbacks. In order to provde mnmum delays, all optmal routng algorthms requre the nput traffc and the network topology to be statonary or very slowly changng (quas-statc), and requre a pror knowledge of global constants that guarantee convergence of the routng algorthm. Ths makes optmal routng algorthms m- Ths work was supported n part by the Defense Advanced Research Projects Agency (DARPA) under grants F and F C practcal for real networks, because n real networks traffc s very bursty at any tme scale and the network topology frequently experence changes. Moreover, defnng global constants that work for all nput traffc patterns are mpossble to determne. On the other hand, routng algorthms based on sngle shortestpath heurstcs adapt very quckly to changng network condtons, makng them far more preferable than optmal routng for mplementaton n real networks. The man shortcomng of sngle shortestpath routng s that the delays achevable wth such heurstcs are far longer than those achevable usng optmal routng algorthms. In addton, sngle-shortest-path routng becomes unstable under heavy loads or very bursty traffc when the lnk cost metrc used n the routng algorthm s related to the delays or congeston experenced over the lnks [3]. The fact that shortest-path routng over sngle paths s far less effcent than optmal dynamc routng and the oscllatory behavor of shortest-path routng when lnk costs are ted to lnk delays has been known for many years. However, mplementng optmal dynamc routng n a computer network has smply been nfeasble to date. The key contrbutons of ths paper consst of: (a) ntroducng a new framework for near-optmum delay routng; (b) verfyng, for the frst tme, a set of nvarants that permt routng-algorthm desgners to approxmate Gallager s necessary and suffcent condtons for mnmum-delay routng wth loop-free routng condtons that can be acheved usng dstrbuted routng algorthms that do not requre any global varables or global synchronzaton; and (c) showng an example that provdes end-to-end delays that are comparable to the optmal, whle beng as fast as today s shortest-path routng schemes. Secton presents the mnmum-delay routng problem (MDRP) as descrbed by Gallager, and Gallager s mnmum-delay routng algorthm [8]. Gallager s algorthm s unsutable for practcal networks and nternetworks, because ts speed of convergence to the optmal routes depends on a global constant, and because t requres that the nput traffc and network topology be statonary or quasstatonary. Several algorthms have been proposed to date that mprove over Gallager s mnmum-delay routng algorthm [, 6, 3, ]. Segall and Sd [3, ] extended Gallager s mnmum-delay routng algorthm to handle topologcal changes usng technques developed by Merln and Segall [19]. Cassandras et al. [6] present a better technque for measurng margnal delays. Bertsekas and Gallager [] used second dervatves to speed up convergence of Gallager s algorthm. However, all these algorthms are stll dependent on global constants and the requrement that network traffc be statc or quas-statc. Because of ts oscllatory behavor when lnk costs are related to delays, attempts to mprovng shortest-path routng have been restrcted manly to usng better lnk cost metrcs (e.g., [18, 13]) or usng multple-paths. To avod undetected loops, OSPF permts multple paths to a destnaton only when they have the same length [0]. More recently, Zaumen and Garca-Luna-Aceves [7] proposed an algorthm based on dstance vectors that supports multple paths of unequal costs to each destnaton; however, lnk costs are not ted to delays. Wang and Crowcroft [6] addressed the drawbacks of the shortest-path frst (SPF) algorthm by usng alternate paths to detour traffc around ponts of congeston or network falures. However, the alternate paths n SPF-EE (for emergency

2 exts) are computed on a reactve bass,.e., once congeston occurs, whch s less effectve n dealng wth short bursts of traffc. Can et al. [] descrbe a routng algorthm for mnmzng delays. However, ths algorthm requres that the routng-table updates at all the routers be synchronzed, otherwse loopng occurs, whch ncreases end-to-end delays. Because the synchronzaton ntervals requred by ths algorthm must be known by all routers, ths s akn to usng a global constant as n Gallager s algorthm. Ths approach s not scalable to very large networks, because the tme needed for routng-table update synchronzaton becomes large, and ths n turn lmts ts responsveness to short-term traffc fluctuatons. What s serously lackng n ths algorthm s a technque for asynchronous computaton of multple paths wth nstantaneous loop-freedom. Secton 3 presents a new framework for approxmate solutons to MDRP. The novelty of ths framework stems from parttonng the computaton of mnmum-delay paths n two parts. Frst, multple loop-free paths of unequal cost to a destnaton are frst establshed usng long-term lnk-cost nformaton. Ths s followed by the allocaton of flows to destnatons along the multple loopfree paths avalable at each router; such an allocaton s based on heurstcs that attempt to mnmze delays usng short-term lnkcost nformaton. It s ths parttonng of MDRP that permts us to mplement routng algorthms that provde routers wth nearoptmum delays whle keepng the routng algorthm as responsve to traffc or topology changes as the best of today s shortest-path routng algorthms. A set of nvarants s also presented that permts Gallager s necessary and suffcent condtons for mnmumdelay routng to be approxmated wth loop-free routng condtons achevable wth smple dstrbuted routng algorthms that do not requre any global varables or global synchronzaton. Secton descrbes a specfc routng algorthm based on our new routng framework. Ths algorthm conssts of two key components: (a) the frst lnk-state routng algorthm that provdes multple loop-free paths of arbtrary postve cost at every nstant, and (b) flow allocaton heurstcs that approxmate mnmum delays along the predefned multple loop-free paths avalable for each destnaton Ṡecton 5 presents results of smulaton experments desgned to llustrate the effectveness of our soluton n statc and dynamc networks. We compare our approach aganst the optmal routng approach and shortest-path routng based on Djkstra s shortestpath frst (SPF) algorthm, because t s used wdely n the Internet today. The smulaton results llustrate that the routng delays obtaned wth our new algorthm are comparable to the optmal delays. Furthermore, the complexty of mplementng our routng framework s smlar to the complexty of routng protocols that provde sngle-path routng n the Internet today. Mnmum Delay Routng.1 Problem formulaton The mnmum-delay routng problem (MDRP) was frst formulated by Gallager [8], and we provde the same descrpton n ths secton. A computer network G = (N;L) s made up of N routers and L lnks between them. Each lnk s bdrectonal wth possbly dfferent costs n each drecton. Let r j 0 be the expected nput traffc, measured n bts per second, enterng the network at router and destned for router j. Let t j be the sum of r j and the traffc arrvng from the neghbors of for destnaton j. And let routng parameter jk be the fracton of traffc t j that leaves router over lnk (; k). Assumng that the network does not lose any packets, from conservaton of traffc we have t j = r j + X where N s the set of neghbors of router. kn t k j k j (1) Let f k be the expected traffc, measured n bts per second, on lnk (; k). Because t j jk s the traffc destned for router j on lnk (; k) we have the followng equaton to fnd f k. f k = X jn t j jk () Note that 0 f k C k,wherec k s the capacty of lnk (; k) n bts per second. Property 1 For each router and destnaton j, the routng parameters jk must satsfy the followng condtons: 1. jk =0f (; k) = L or = j. Clearly, f the lnk does not exst, there can be no traffc on t.. jk 0. Ths s true, because there can be no negatve amount of traffc allocated on a lnk. 3. P kn jk =1. Ths s a consequence of the fact that all ncomng traffc must be allocated to outgong lnks. Let D k be defned as the expected number of messages or packets per second transmtted on lnk (; k) tmes the expected delay per message or packet, ncludng the queueng delays at the lnk. We assume that messages are delayed only by the lnks of the network and D k depends only on flow f k through lnk (; k) and lnk characterstcs such as propagaton delay and lnk capacty. D k (f k ) s a contnuous and convex functon that tends to nfnty as f k approaches C k. The total expected delay per message tmes the total expected number of message arrvals per second s gven by D T = X (;k)l D k (f k ) (3) Note that the router traffc-flow set t = ft jg and lnk-flow set f = ff k g can be obtaned from r = frjg and = f jkg. Therefore, D T can be expressed as a functon of r and usng Eqs. (1) and (). The mnmum-delay routng problem can now be stated as follows: MDRP: For a gven fxed topology and nput traffc flow set r = frjg, and delay functon D k (f k ) for each lnk (; k), the mnmzaton problem conssts of computng the routng parameter set = f jkg such that the total expected delay D T s mnmzed.. A Mnmum Delay Routng Algorthm Gallager [8] derved the necessary and suffcent condtons that must be satsfed to solve MDRP. These condtons are summarzed n Gallager s Theorem stated below. The partal dervatves of the total delay, D T, of Eq.(3) wth respect to r and play a key role n the formulaton and soluton of the problem; these dervatves jk = X jk [D0 k (f k) k ] () kn j = t j [D0 k (f k) k j ] (5) where D 0 k (f k) k (f k )=@f k. and s called the margnal delay or ncremental delay.

3 T =@rj s called the margnal dstance from router to j. Gallager s Theorem [8]: The necessary condton for a mnmum of D T wth respect to for all 6= j and (; k) L = jk = j jk > 0 j jk =0 (6) where j s some postve number, and the suffcent condton to mnmze D T wth respect to s for all 6= j and (; k) L s D 0 k(f j (7) Eq. () shows the relaton between a router s margnal dstance to a partcular destnaton and the margnal dstances of ts neghbors to the same destnaton. Eqs. (5)-(7) ndcate the condtons for perfect load balancng,.e., when the routng parameter set gves the mnmum delay. The set of neghbors through whch router forwards traffc towards j s denoted by Sj and s called the successor set. 1 Under perfect load balancng wth respect to a partcular destnaton, the margnal dstances through neghbors n the successor set are equal to the margnal dstance of the router, and the margnal dstances through neghbors not n the successor set are hgher than the margnal dstance of the router. Let Dj denote the margnal dstance from to T =@rj. Let the margnal delay Dk 0 (f k) from to k be denoted by lk whch s also called the cost of the lnk from to k. Accordng to Gallager s Theorem, the mnmum delay routng problem now becomes one of determnng, at each router for each destnaton j: the routng parameters f jkg, Sj and Dj, such that the followng fve equatons are satsfed: D j = X kn jk (Dk j + l k ) (8) S j = fkj jk > 0 ^ k N g (9) D j D k j + l k k N (10) (D p j + l p ) = (Dq j + l q ) p; q S j (11) (D p j + l p ) < (Dq j + l q ) p S j q = S j (1) Ths reformulaton of MDRP s crtcal, because t s the frst step n allowng us to approach the problem by lookng at the nexthops and dstances obtaned at each router for each destnaton. Gallager [8] descrbed a dstrbuted routng algorthm for solvng the above fve equatons. When the algorthm converges, the aggregate of the successor sets for a gven destnaton j (Sj for every ) defne a drected acyclc graph. In fact, n any mplementaton, Sj must be loop-free at every nstant, because even temporary loops cause traffc to recrculate at some nodes and results n ncorrect margnal delay computatons, whch n turn can prevent the algorthm from convergng or obtanng mnmum delays. Gallager s dstrbuted algorthm uses an nterestng blockng technque to provde loop-freedom at every nstant [8, 3, ]. We refer to ths algorthm as OPT n the rest of the paper. Unfortunately, OPT cannot be used n real networks for several reasons. A major drawback of OPT s that a global step sze needs to be chosen and every router must use t to ensure convergence. Because depends on the nput traffc pattern, t s mpossble to determne one n practce that works for all nput traffc patterns and for all possble topology modfcatons. The routng parameters are drectly computed by OPT and the multple loop-free paths are 1 The term successor set was frst ntroduced n [7]. smply mpled by the routng parameters n Eq. (9). The computaton of routng parameters s, for all practcal purposes, a very slow process as t s a destnaton-controlled process. The destnaton ntates every teraton that adjusts the routng parameters at every router; furthermore, each teraton takes a tme proportonal to the dameter of the network and number of messages proportonal to number of lnks. Ths renders the algorthm slow convergng and useful only when traffc and topology are statonary for tmes long enough for all routers to adjust ther routng parameters between changes. Also, dependng on the global constant, the destnaton must ntate several teratons for the parameters to converge to ther fnal values. The number of such teratons needed for convergence tends to be large for a small, and small for a large value of. Unfortunately, cannot be made arbtrarly large to reduce the number of teratons and to speed up convergence, because the algorthm may not converge at all for large values of. Hence, Gallager s algorthm can be vewed only as a method for obtanng lower bounds under statonary traffc, rather than as an algorthm to be used n practce. The next secton shows how the theory ntroduced n the Gallager s method can be adapted to practcal networks. 3 A New Framework for Mnmum-Delay Routng We noted that n Gallager s algorthm the computaton of the routng parameter set s slow convergng and works only n the case of statonary or quas-statonary traffc. In the Internet, traffc s hardly statonary and perfect load balancng s nether possble nor necessary. Intutvely, an approxmate load balancng scheme based on some heurstc whch can quckly adapt to dynamc traffc should be suffcent to mnmze delays substantally. The key dea n our approach s, n a sense, to reverse the way n whch Gallager s algorthm solves MDRP. The ntuton behnd our approach s that establshng paths from sources to destnatons takes a much longer tme than shftng loads from one set of neghbors to another, smply because of the propagaton and processng delays ncurred along the paths. Accordngly, t makes sense to frst establsh multple loop-free paths usng long-term (end-to-end) delay nformaton, and then adjust routng parameters along the predefned multple paths usng short-term (local) delay nformaton. Ths new approach allows us to attempt to use dstrbuted algorthms to compute multple loop-free paths from source to destnaton that, hopefully, are as fast as today s sngle-path routng algorthms, and local heurstcs that can respond quckly to temporary traffc bursts usng local short-term metrcs alone. Therefore, we map Eqs. (8)-(1) derved n Gallager s method nto the followng three equatons: D j = mnfd k j + l kjk N g (13) S j = fkjd k j <D j ^ k N g (1) jk = (k; A j;b j ) k N (15) where A j = fd p j + l pjp N g and Bj = f jpjp N g. These equatons smply state that, for an algorthm to approxmate mnmum-delay routng, t must establsh loop-free paths and use a functon to allocate flows over those paths. We observe that Eq. (13) s the well-known Bellman-Ford (BF) equaton for computng the shortest paths, and Eq. (1) s the successor set consstng of the neghbors that are closer to the destnaton than the router tself. Note that the paths mpled by the neghbors n the successor set of a router need not be of the same length. The functon n Eq. (15) s a heurstc functon that determnes the routng parameters. Because changng the routng parameters effects the margnal delay of the lnks (hence lnk-costs), we use regular updates of the lnk costs. The man problem wth attemptng to solve MDRP usng Eqs. (13) to (15) drectly s that these equatons assume that routng nformaton s consstent throughout the network. In practce, a node (router) must choose ts dstance and successor set usng routng nformaton obtaned through ts neghbors, and ths nformaton may

4 be outdated. At any tme t, for a partcular destnaton j, the successor sets of all nodes defne a routng graph SG j (t) =f(m; n)jn Sj m (t); m Ng. In sngle-path routng, Sj (t) has at most one neghbor: the neghbor that s on the shortest path to destnaton j. Accordngly, SG j(t) for sngle-path routng s a snk-tree rooted at j f loops are never created. The routng graph SG j(t) n our case should be a drected acyclc graph n order for mnmum delays to be approached. The blockng technque used n Gallager s algorthm ensures nstantaneous loop-freedom. Lkewse, to provde loop-free paths even when the network s n transent state wthn the context of our framework, addtonal constrants must be mposed on the choce of successors at each router, whch essentally must preclude the use of neghbors that may lead to loopng. Several algorthms have been proposed n the past to provde loop-free paths at every nstant for the case of sngle-path routng (e.g., the Jaffe-Moss algorthm [15], DUAL [9], LPA [11], and the Merln-Segall algorthm [19]) and one algorthm, DASM, has been proposed for the case of multple paths per destnaton [7]. All these algorthms are based on the exchange of vectors of dstances, together wth some form of coordnaton among routers spannng one or multple hops. The coordnaton among routers determnes when the routers can update ther routng tables. Ths coordnaton s n turn guded by local condtons that depend on values of reported dstances to destnatons and that are suffcent to prevent loops from occurrng. We generalze the work to date on loop-free routng over sngle paths or multple paths by means of the followng loop-free nvarant (LFI) condtons, whch are applcable to any type of routng algorthm. We adopt the same termnology and nomenclature frst ntroduced for DUAL [9] to descrbe the LFI condtons. Loop-free Invarant (LFI) condtons: Any routng algorthm desgned such that the followng two equatons are always satsfed, automatcally provdes loop-free paths at every nstant, regardless of the type of routng algorthm beng used: FD j D k j k N (16) S j = f k j D jk <FD j ^ k N g (17) where Djk s the value of Dj k reported to by ts neghbor k; and FDj s called the feasble dstance of router for destnaton j and s an estmate of Dj, n the sense that FDj equals Dj n steady state but s allowed to dffer from t temporarly durng perods of network transtons. In lnk-state algorthms, the values of Djk are determned locally from the lnk-state nformaton suppled by the router s neghbors; n contrast, n dstance-vector algorthms, the dstances are drectly communcated among neghbors. The followng theorem verfes ths key result of our framework. Theorem 1 If the LFI condtons are satsfed at any tme t, the routng graph SG j (t) mpled by the successor sets S j (t) s loopfree. Proof: Let k Sj (t) then from Eq. (17) we have D jk(t) < FD j (t) (18) At router k, because router s a neghbor, from Eq. (16) we have FDj k (t) Djk (t). Combnng ths result wth Eq. (18) we obtan FD k j (t) < FD j (t) (19) Eq. (19) states that, f k s a successor of router n a path to destnaton j, thenk s feasble dstance to j s strctly less than the feasble dstance of router toj. Now, f the successor sets defne a loop at tme t wth respect to j, then for some router p on the loop, we arrve at FD p j (t) <FDp j (t), an absurd relaton. Therefore, the LFI condtons are suffcent for loop-freedom. Wth the result of Theorem 1, Eq. (1) can be approxmated wth the LFI condtons to render a routng approach that does not requre routng nformaton to be globally consstent, at the expense of renderng delays that may be longer than optmal. Accordngly, our framework for near-optmum-delay routng les n fndng the soluton to the followng equatons usng a dstrbuted algorthm: D j = mnfd k j + l k jk N g (0) FD j D k j k N (1) S j = f k j D jk <FD j ^ k N g () jk = (k; fd p j + l pjp N g; f jp jp N g) k N (3) Implementng Near-Optmum-Delay Routng We present an approach based on lnk-state nformaton, rather than dstance nformaton, because extendng our results to mnmumdelay routng wth addtonal constrants can be done more effcently by workng wth lnk parameters than path parameters, whch are the combnaton of lnk parameters. Our approach conssts of three components: computng multple loop-free paths, dstrbutng traffc over such paths, and computng lnk costs..1 Computng Multple Loop-free Paths We descrbe the computaton of multple loop-free paths n two parts: computng Dj usng a shortest-path algorthm based on lnkstate nformaton, and computng Sj by extendng that algorthm to support multple successors along loop-free paths to each destnaton..1.1 Computng D j There are many dstrbuted algorthms for computng shortest paths, and any of them can be extended to provde multple paths of equal and unequal costs as long as the extenson obeys the LFI condtons ntroduced n the prevous secton. The partal-topology dssemnaton algorthm (PDA) propagates enough lnk-state nformaton n the network, so that each router has suffcent lnk-state nformaton to compute shortest paths to all destnatons. In ths respect, t s smlar to other lnk-state algorthms (e.g., OSPF [0], SPTA [5], LVA [10], ALP [1]). PDA combnes the best features of LVA, ALP and SPTA. As n LVA and ALP, a router communcates to ts neghbors nformaton regardng only those lnks that are part of ts mnmum-cost routng tree, and lke SPTA, a router valdates lnk nformaton based on dstances to heads of lnks and not on sequence numbers. PDA assumes that a router detects the falure, recovery and lnk-cost change of an adjacent lnk wthn a fnte amount of tme. An underlyng protocol ensures that messages transmtted over an operatonal lnk are receved correctly and n the proper sequence wthn a fnte tme and are processed by the router one at a tme n the order receved. These are the same assumptons made for smlar routng algorthms and can be easly satsfed n practce. Each router runnng PDA mantans the followng nformaton: 1. The man topology table, T, stores the characterstcs of each lnk known to router. Each entry n T s a trplet [h; t; d] where h s the head, t s the tal and d s the cost of the lnk h! t.. The neghbor topology table, Tk, s assocated wth each neghbor k. The table stores the lnk-state nformaton communcated by the neghbor k. Thats,Tk s a tme-delayed verson of T k.

5 procedure INIT-PDA finvoked when the router comes up.g begn Intalze all tables; call PDA; end INIT-PDA procedure PDA fexecuted at each router. Invoked when an event occursg begn (1) call NTU; () call MTU; /* Updates T */ (3) f (there are changes to T ) then Compose an LSU message consstng of topology dfferences usng add, delete and change lnk entres; () Wthn a fnte amount tme, send the LSU message to all neghbors; end PDA Fgure 1: The Partal-topology Dssemnaton Algorthm 3. The dstance table stores the dstances from router to each destnaton based on the topology n T and the dstances from each neghbor k to each destnaton based on the topologes n T k for each k. The dstance of router to node j n T s denoted by D j; the dstance from k to j n T k s denoted by D jk.. The routng table stores, for each destnaton j, the successor set Sj and the feasble dstance FDj, whch s used by MPDA to enforce LFI condtons. 5. The lnk table stores, for each neghbor k, thecostlk of the adjacent lnk to the neghbor. The unt of nformaton exchanged between routers s a lnkstate update (LSU) message. A router sends an LSU message contanng one or more entres, wth each entry specfyng addton, deleton or change n cost of a lnk n the router s man topology table T. Each entry of an LSU conssts of lnk nformaton n the form of a trplet [h; t; d] where h s the head, t s the tal, and d s the cost of the lnkh! t. An LSU message contans an acknowledgment (ACK) flag for acknowledgng the recept of an LSU message from a neghbor (used only by MPDA). The INIT-PDA procedure n Fg. 1 ntalzes the tables of a router at startup tme; all varables of type dstance are ntalzed to nfnty and those of type node are ntalzed to null. All successor sets are ntalzed to the empty set. PDA s executed each tme an event occurs; an event can be ether a recept of an LSU message from a neghbor or the detecton of an adjacent lnk-cost change. Procedure NTU (Neghbor Topology Table Update) shown n Fg. s used to process the receved message and update the necessary tables. Procedure MTU n Fg. 3 constructs the router s own shortest path tree from the topologes reported by ts neghbors. The new shortest-path tree obtaned s compared wth the prevous verson to determne the dfferences; only the dfferences are then reported to the neghbors. The router then wats for the next event and, when t occurs, the whole process s repeated. The algorthm MTU at router merges the topologes T k and the adjacent lnks l k to obtan T. The merge process s straghtforward f all neghbor topologes contan dsjont sets of lnks, but when two or more neghbors report conflctng nformaton regardng a partcular lnk, the conflct has to be resolved. Sequence numbers may be used to dstngush between old and new lnk nformaton as n OSPF, but PDA resolves the conflct as follows. If two or more neghbors report nformaton of lnk (m; n) then the router should update topology table T wth lnk nformaton reported by procedure NTU begn (1) f (LSU message s receved from a neghbor k) then (1a) Update neghbor table Tk. That s, add lnks, delete lnks or change lnks accordng to the specfcaton of each entry n the LSU; (1b) Run Djkstra s shortest path algorthm on the resultng topology Tk ; /*Ths results n fndng mnmum dstances from k to all other nodes n Tk.NoteT k s a tree*/ (1c) Update Djk wth new dstances n T k ; () f (adjacent lnk (; k) s up) then Update lk and send an LSU message to the neghbor k wth lnk nformaton of all lnks n ts man topology table T ; (3) f (cost of an adjacent lnk (; k) changed)then Update lk ; () f (adjacent lnk (; k) faled)then Update lk and clear the table T k ; end NTU Fgure : Neghbor Topology Table Update algorthm the neghbor that offers the shortest dstance from the router to the head node m of the lnk. Tes are broken n favor of neghbor wth the lowest address. For adjacent lnks, router tself s the head of the lnk and thus has the shortest dstance. Therefore, any nformaton about an adjacent lnk suppled by neghbors wll be overrdden by the most current nformaton about the lnk avalable to router. Djkstra s shortest path algorthm s run on T and only the lnks that consttute the shortest-path tree are retaned. Note that, because there are potentally many shortest-path trees, tes should be broken consstently durng the run of Djkstra s algorthm. In what follows, we show that PDA works correctly by showng that the topology tables at all nodes converge to the shortest paths wthn a fnte tme after the last lnk cost change n the network. After convergence, because there are no more changes to the topology tables, no more LSU messages are generated. Defntons: The n-hop mnmum dstance of router to node j n a network s the mnmum dstance possble usng a path of n lnks or less. A path that offers the n-hop mnmum dstance s called n-hop mnmum path. If there s no path wth n hops or less from router to j then the n-hop mnmum dstance from to j s undefned. An n-hop mnmum tree of a node s a tree n whch router s the root and all paths of n hops or less from the root to any other node s an n-hop mnmum path. Note that there could be more than one n-hop mnmum tree. Let G denote the fnal topology of the network after all lnk changes occurred as seen by an omnscent observer; we use bold font to refer to all quanttes n G. LetH n denote an n-hop mnmum tree rooted at router n G and let M n be the set of nodes that are wthn n hops from n H n.letd ;j n denote the dstance of to j n H n.letd j be the cost of the lnk! j. The notaton ; j ndcates a path from to j of zero or more lnks. Property From the prncple of optmalty (a sub-path of a shortest path between two nodes s also a shortest path between the end nodes of the sub-path), f H and H 0 are two n-hop mnmum trees rooted at router and M and M 0 are sets of nodes that are wthn n hops from n H and H 0 respectvely, then M = M 0 = M n. Also, for each j M n the length of path ; j n both H and H 0 s equal to Dn ;j. Also, D ;j h D;j n f h n.

6 procedure MTU at router begn (1) oldt T ;/* Save copy */ () f (node j occurs n at least one of Tk ) then add j to the man topology table T ; (3) foreach node j n T do MIN mnfdjk + l k jk N g; let p be such that MIN =(Djp + l p ); /* Neghbor p s the preferred neghbor for destnaton j. Tes are broken n favor of lower address neghbor */ done () foreach j n T and ts preferred neghbor p do Copy all lnks (j; n) from Tp to T ; /*.e., copy all lnks n Tp for whch j s the head node */ done (5) Update T wth nformaton of each lk ; (6) Run Djkstra s shortest path algorthm on T and remove those lnks n T that are not part of the shortest path tree; (7) Update Dj wth new dstances n T ; (8) Compare oldt wth T and note all dfferences; end MTU Fgure 3: Man Topology Table Update Algorthm We say a router knows at least the n-hop mnmum tree, f the tree represented by ts man topology table T s at least an n-hop mnmum tree rooted at n G and there are at least n nodes n T that are reachable from the root. Note that the lnks n T that are more than n hops may have costs that do not agree wth the lnk costs n G. Lemma 1 If a router has the fnal correct costs of the adjacent lnks and for each neghbor k the topology T k s an n-hop mnmum tree, then the topology T s (n +1)-hop mnmum tree after the executon of MTU. Proof: The proof s presented n the Appendx. Theorem At each router, the man topology T gves the correct shortest paths to all known destnatons a fnte tme after the last change n the network. Proof: The proof s by nducton on t n, the global tme when for each router, T s at least n-hop mnmum tree. Because the longest loop-free path n the network has at most N,1 lnks where N s number of nodes n the network, t N,1 s the tme when every router has the shortest path to every other node. We need to show that t N,1 s fnte. The base case s t 1, the tme when every node has 1-hop mnmum dstance and because the adjacent lnk changes are notfed wthn fnte tme, t 1 < 1. Let t n < 1 for some n < N. Gven that the propagaton delays are fnte each router wll have each of ts neghbors n-hop mnmum tree n fnte tme after t n. From Theorem 1 we can see that the router wll have at least the (n +1)-hop mnmum tree wthn a fnte tme after t n. Therefore, t n+1 < 1. From nducton, we can conclude that t N,1 < Computng S j The LFI condtons ntroduced n Secton 3 suggest a technque for computng S j such that the mpled routng graph SG j s loopfree at every nstant. To determne FD j n Eq.(16), router needs to know D k j, the dstance from to node j n the topology table procedure MPDA at router fnvoked when an event occursg begn (1) call NTU; () f (node s n PASSIVE state) then (a) call MTU; /* update T and Dj */ (b) FDj mnffd j ;D j g; (3) f (node s n ACTIVE state and the last ACK s receved) then (3a) temp j D j ; Set node to PASSIVE state; (3b) call MTU to update T ; (3c) FDj mnftemp j ;D j g () Sj fkjd jk <FD j g; (5) f (changes occur n T )then Set node to ACTIVE state; f (no changes occur n T and the event s the last ACK) then Set node to PASSIVE state; (6) f (there are changes to T ) then Compose a new LSU wth the topology changes expressed as add lnk, delete lnk and change lnk; (7) f (nput event receved s an LSU message)then Add the ACK entry to newly composed LSU; (8) Send the new LSU message. end MPDA Fgure : Multple-path Partal-topology Dssemnaton Algorthm (MPDA) T k. Because of propagaton delays, there may be dscrepances between the man topology table T at router and ts copy T k at the neghbor k. However, at tme t, the topology table T k s a copy of the man topology table T at some earler tme t 0 < t. Logcally, f a copy of Dj s saved each tme an LSU s sent, a feasble dstance FDj that satsfes the LFI condtons can be found n the hstory of values of Dj that have been saved! The multple-path partal-topology dssemnaton algorthm, or MPDA, shown n Fg. s a modfcaton of PDA that enforces the LFI condtons by synchronzng the exchange of LSUs between neghbors. In MPDA, each LSU message sent by a router s acknowledged by all ts neghbors before the router sends the next LSU. The nter-neghbor synchronzaton used n MPDA spans only a sngle hop, unlke the synchronzaton n dffusng computatons [7] whch potentally spans the whole network. A router s sad to be n ACTIVE state when t s watng for ts neghbors to acknowledge the LSU message t sent; otherwse, t s n PASSIVE state. Assume that, ntally, all routers are n PASSIVE state wth all routers havng the correct dstances to all destnatons. Then a seres of lnk cost changes occurs n the network resultng n some or all routers to go through a sequence of PASSIVE-to-ACTIVE and ACTIVE-to-PASSIVE state transtons, untl all routers become PASSIVE wth correct dstances to destnatons. If a router n a PASSIVE state receves an event that does not change ts topology T, then the router has nothng to report and remans n PASSIVE state. However, f a router n PASSIVE state receves an event that affects a change n ts topology, the router sends those changes to ts neghbors, goes nto ACTIVE state and wats for ACKs. Events that occur durng the ACTIVE perod are processed to update Tk and lk but not T ; the updatng

7 Passve-to-actve transtons Implct transton Tme X Topology changng Events Topology changng Events Actve-to-passve transtons Fgure 5: Actve-passve phase transtons n MPDA. procedure IH begn (1) 8k = Sj ; jk 0; () f (jsj j =1) then 8k Sj ; jk 1; (3) f (jsj j > 1) then D 1, P jk +l k ms (D jm +l m ) j jk (js j ; 8k S j,1) j ; end IH of T by MTU s deferred untl the end of the ACTIVE phase. At the end of the ACTIVE phase, when ACKs from all neghbors are receved, router updates T wth changes that may have occurred n T k due to events receved durng the ACTIVE phase. If no changes occurred n T that need reportng, then the router becomes PASSIVE; otherwse, as shown n Fg. 5, there are changes n T that may have resulted due to events and the neghbors need to be notfed. Ths results n a new LSU, and the router mmedately becomng ACTIVE agan. In ths case, there s an mplct PASSIVE perod, of zero length of tme, between two back-toback ACTIVE perods, as llustrated n Fg. 5. A router recevng an LSU message from k must send back an LSU wth the ACK bt set after updatng T k. If the router does not have any updates to send, ether because t s n ACTIVE state or because t does not have any changes to report, t sends back an empty LSU wth just the ACK flag set. When a router detects that an adjacent lnk faled, any pendng ACKs from the neghbor at the other end of the lnk are treated as receved. Because all LSUs are acknowledged wthn a fnte tme, no deadlocks can occur. The followng theorem proves that MPDA provdes loop-free multpaths at every nstant. Theorem 3 (Safety property) At any tme t, the drected graph SG j (t) mpled by the successor sets S j (t) computed by MPDA at each router s loop-free. Proof: The proof s presented n the Appendx, and s based on showng that FDj and Sj, as computed by MPDA, satsfy the LFI condtons. Theorem (Lveness property) A fnte tme after the last change n the network, Dj gves the correct shortest dstance and S j = fkjd k j <D j;k N g at each router Proof: The convergence of MPDA follows drectly from the convergence of PDA, because the update messages n MPDA are only delayed a fnte tme as allowed n lne n algorthm PDA. Therefore, the dstances D j n MPDA also converge to shortest dstances. Because changes to T are always reported to the neghbors and are ncorporated by the neghbors n ther tables n fnte tme, D jk = Dk j for k N after convergence. From lne 3c n MPDA, we observe that when router becomes PASSIVE, and FD j = D j holds true. Because all routers are PASSIVE at convergence tme t follows that the set fkjd jk <FD j;k N g s the same as the set fkjd k j <D j;k N g.. Dstrbutng Trac over Multple Paths In general, the functon can be any functon that satsfes Property 1, but our objectve s to obtan a functon that performs load balancng that s as close as possble to perfect load balancng (Eqs.(10)-(1)). Fgure 6: Heurstc for ntal load assgnment. procedure AH begn (1) D j mn mnfd jk + l k jk S j g; () let D j mn =(D jk + l 0 k ); 0 // That s, k 0 be the neghbor that offers ths mnmum) (3) foreach k Sj do a jk D jk + l k, Dj mn ; done 1 () mnf jk jk S j ^ a jk 6=0g; a jk () foreach k 6= k 0 ^ k S j do jk jk, a jk ; done (5) for k = k 0 do jk jk +P qs j a jq ; done end AH Fgure 7: Heurstc for ncremental load adjustment. The functon should also be sutable for use n dynamc networks, where the flows over lnks are contnuously changng, causng contnuous lnk-cost changes. To respond to these changes, queueng delays at the lnks must be measured perodcally and routng paths must be recomputed. However, re-computng paths frequently consumes excessve bandwdth and may also cause oscllatons. Therefore, routng-path changes should only be done at suffcently long ntervals. Unfortunately, a network cannot be responsve to short-term traffc bursts f only long-term updates are performed. For ths reason, we use lnk costs measured over two dfferent ntervals; lnk costs measured over short ntervals of length T s are used for routng-parameter computaton and lnk costs measured over longer ntervals of length T l are used for routngpath computaton [17]. In general, T l must be several tmes longer than T s. Long-term updates are desgned to handle long-term traffc changes and are used by the routng protocol to update the successor sets at each router, so that the new routng paths are the shortest paths under the new traffc condtons. The short-term updates made every T s seconds are desgned to handle short-term traffc fluctuatons that occur between long-term routng path updates and are used to compute the routng parameters jk n Eq. (15) locally at each router. Accordngly, our traffc dstrbuton heurstcs assume a constant successor set and successor graph. When Sj s computed for the frst tme or recomputed agan due to long-term route changes, traffc should be freshly dstrbuted. In ths case, the allocaton heurstc functon s a functon of only the margnal dstances through the successor set. That s, Eq. (15)

8 reduces to the form f jkg = (k; fd p j + l pjp N g). Whena new successor set Sj s computed, algorthm IH n Fg. 6 s frst used to dstrbute traffc over the successor set [17]. Note that f jkg, computed n IH, satsfy Property 1. Furthermore, when more than one successor s present, f Djp + lp >Djq + lq for successors p and q,then jp < jq. The heurstc makes sense because the greater the margnal delay through a partcular neghbor becomes, the smaller the fracton of traffc that s forwarded to that neghbor. After the frst flow assgnment s made over a newly computed successor set usng algorthm IH, a dfferent flow allocaton heurstc algorthm AH shown n Fg. 7 s used to adjust the routng parameters every T s seconds untl the successor set changes agan. The heurstc functon computed n AH s ncremental and, unlke IH, s a functon of current flow allocaton on the successor sets and the margnal dstances through the successors. AH also preserves Property 1 at every nstant. In AH traffc s ncrementally moved from the lnks wth large margnal delays to lnks wth the least margnal delay. The amount of traffc moved away from a lnk s proportonal to how large the margnal delay of the lnk s compared to the best successor lnk. The heurstc tends to dstrbute traffc n such a way that Eqs. (10)-(1) hold true. Ths s mportant, because the ntal dstrbuton obtaned by IH s far from beng balanced. The computaton complexty of the heurstc allocaton algorthms s O(N ). Because the heurstcs are run for each actve destnaton, the whole load-balancng actvty s O(N ). Unlke n Gallager s algorthm, T l and T s are local constants that are set ndependently at each router. Convergence of our algorthm does not crtcally depend on these constants lke optmal routng does on. Also, T l and T s need not be statc constants and can be made to vary accordng to congeston at the router. The value of T l, however, should be such that t s suffcently longer than the tme t takes for computng the shortest paths. The longterm update perods should be phased randomly at each router, because of the problems that would result due to synchronzaton of updates [3]..3 Computng Lnk Costs As mentoned earler, the cost of a lnk s the margnal delay over the lnk D 0 (f k ). If the lnks are assumed to behave lke M/M/1 queues, then the margnal delay D 0 (f k ) can be obtaned n a closed form expresson by dfferentatng the followng equaton [16]. D k (f k )= f k (C k, f k ) + kf k () where f k s the flow through the lnk (; k), andc k and k are the capacty and propagaton delay of the lnk. Because the M/M/1 assumpton does not hold n practce n the presence of very bursty traffc, and because Eq. () becomes unstable when f k approaches C k, an on-lne estmaton of the margnal delays s desrable. There are several technques for computng margnal delays that are currently avalable (e.g., [3,, 6]). For the purposes of smulatons, we borrow a technque ntroduced by Cassandras, Abd and Towsley [6] for on-lne estmaton of the margnal delay D 0 (f k ). The technque uses perturbaton analyss (PA) for the on-lne estmaton and s shown to perform better than the M/M/1 estmaton. In addton, the PA estmaton does not requre a pror knowledge of the lnk capactes. Ths s very sgnfcant, because the capacty avalable to best-effort traffc n real networks vares accordng to the capacty allocated to other types of traffc, such as real-tme traffc. We must emphasze that our approach does not depend on whch specfc technque s used for margnal-delay estmaton, although some methods may be better than others. The convergence or stablty of our routng algorthm does not depend on the specfc technque used for margnal-delay estmaton. ucsc pslon csco-w parc ucb sr lbl anl netstar toc 5 Smulatons s CAIRN Topology ucla csco-e mt sdsc bbn s-e bell mc-r ts nasa nrl-v6 sac udel darpa cmu ucl NET1 Topology 3 Fgure 8: Topologes used n smulatons The smulatons dscussed n ths secton llustrate the effectveness of our near-optmal framework, and demonstrate the sgnfcant mprovements acheved by our approach over sngle-path routng n statc and dynamc envronments. The delays obtaned by optmal routng, sngle-path routng and our approxmaton scheme are compared under dentcal topologcal and traffc envronments. The results show that the average delays acheved va our approxmaton scheme are comparable (wthn a small percentage dfference rather than several tmes dfference) to the optmal routng under quas-statc envronment and the same are sgnfcantly better than sngle-path routng n a dynamc envronment. For optmal routng, we mplemented the algorthm descrbed by Gallager [8], and label t wth OPT. The plots of our approxmaton scheme are labeled wth MP. To obtan representatve delays for sngle-path routng algorthms, we opted to restrct our multpath routng algorthm to use only the best successor for packet forwardng, nstead of smulatng any specfc shortest-path algorthm. Because of the nstantaneous loop-freedom property that MPDA exhbts, the shortest-path delays obtaned ths way are better than or smlar to the delays obtaned wth ether EIGRP [1], whch s based on DUAL and requres much more nternodal synchronzaton than our scheme, renderng longer delays, and RIP [1] or OSPF [0], whch do not prevent temporary loops. We use the label SP for sngle-path routng n the graphs. We performed smulatons on the topologes shown n Fg. 8. CAIRN ( s a real network and NET1 s a contrved network. We are only nterested n the connectvty of CAIRN, and ts topology as used dffers from the real network n the capactes and propagaton delays assumed n the smulaton experments. We restrcted the lnk capactes to a maxmum of 10Mbs, so that t becomes easy to suffcently load the networks. NET1 has a connectvty that s hgh enough to ensure the exstence of multple paths, and small enough to prevent a large number of onehop paths. The dameter of NET1 s four and the nodes have degrees between 3 and 5. In each network we setup flows between several source-destnaton pars and measure the average delays of each flow. The flows n CAIRN are setup between these sourcedestnaton pars: (lbl, mc-r),(netstar, se), (s, darpa), (parc, sdsc), (sr, mt),(toc, sdsc),(mt, sr),(se, netstar), (sdsc, parc),(mc-r, toc),(darpa, s). For NET1, the source-destnaton pars are: (9,), (8,3), (7,0), (6,1), (5,8), (,1), (3,8), (,9), (1,6), (0,7). The flows have bandwdths n the range Mbs. For smplcty, we used a stable topology (lnks or nodes do not fal) n all the smulatons. In the presence of lnk falures, MP can only perform better than SP, because of avalablty of alternate paths. Furthermore, OPT s not fast enough to respond to drastc topology changes. Because MP s parameterzed by the T l and T s update ntervals, ts delay plots are represented by MP-TL-xx-TS-yy, where xx s the T l update nterval and yy s the T s update nterval measured n seconds. Smlarly, the delays of shortest-path routng are represented by SP-TL-xx,wherexx s the T l update perod

9 3.5 Comparson of MP and OPT delays OPT MP-TL-10-TS- OPT Comparson of MP and SP delays OPT MP-TL-10-TS-10 MP-TL-10-TS- SP-TL-10 Avg. Delays n mllseconds 3.5 Avg. Delays n mllseconds Fgure 9: Delays of OPT and MP n CAIRN. Fgure 11: Delays of MP and SP n CAIRN Comparson of MP and OPT delays OPT MP-TL-10-TS- OPT Comparson of MP and SP delays OPT MP-TL-10-TS-10 MP-TL-10-TS- SP-TL Avg. Delays n mllseconds Avg. Delays n mllseconds Fgure 10: Delays of OPT and MP n NET1. Fgure 1: Delays of MP and SP n NET Performance under Statonary Trac Fg. 9 shows the average delays of flows n CAIRN for OPT and MP routng. The flow IDs are plotted on the x-axs and average delays of the flows are plotted on the y-axs. Plot OPT-5 represents the 5% envelope, that s, the delays of OPT are ncreased by 5% to obtan the OPT-5 plot. As can be seen, the average delays of flows under MP routng are wthn the OPT-5 envelope. Smlarly, n Fg. 10, the delays obtaned usng MP routng for NET1 are wthn 8% envelopes of delays obtaned usng OPT routng. We say delays of MP are comparable to OPT f the delays of MP are wthn a small percent of those of OPT. Fg. 11 compares the average delays of MP and SP for CAIRN. We observe that the delays of SP for some flows are two to four tmes those of MP. In Fg. 1, for NET1, MP routng performs even better; average delays of SP are as much as fve to sx tmes those of MP routng whch s due to hgher connectvty avalable n NET1. Also observe that, because of load-balancng used n MP, the plots of MP are less jagged than those of SP. MP routng performs much better than SP under hgh-connectvty and hgh-load envronments. When connectvty s low or network load s lght, MP routng cannot offer any advantage over SP. 5. Eect of Tunng Parameters T l and T s The performance of MP depends on the update ntervals T l and T s. The settng of T l and T s, however, s smple. They are local and can be set ndependently at each node wthout affectng convergence, unlke the global constant whch s crtcal for convergence of OPT. For CAIRN, Fg. 13 show the effect of ncreasng T l when T s and the nput traffc s fxed. Observe that when T l s ncreased from 10 to 0 seconds, the delays n SP have more than doubled, whle the delays of MP reman relatvely unchanged. Ths effect ndcates that T l can be made longer n MP wthout sgnfcantly effectng performance. Ths s sgnfcant, because sendng frequent update messages consume bandwdth and can also cause oscllatons under hgh loads. Smlarly, for NET1, delays for SP ncreased sgnfcantly whle there s neglgble change n delays of MP as can be observed n Fg. 1, respectvely. Our new routng framework provdes the means for a trade-off between update messages and local load-balancng. At T s ntervals, the load-balancng heurstcs are executed, whch are strctly local computatons and requre no communcaton. Therefore, T s can be set accordng to the processng power avalable at the router. T l can be made from a few tmes to orders of magntude greater than T s. In the smplest case, T s can be set to the same value of T l and stll gan sgnfcant performance as shown n Fgs. 11 and 1. In the fgures, we observe that MP-TL-10-TS- 10 s much closer to OPT than SP-TL-10. Just the long-term routes wth load-balancng, wthout short-term routng parameter updates,