Performance Evaluation of Dynamic Reconfiguration in High-Speed Local Area Networks

Transcription

1 Performnce Evlution of Dynmic Reconfigurtion in High-Speed Locl Are Networks Rfel Csdo, Aurelio Bermúdez, Frncisco J. Quiles, JoséL.Sánchez Depto. de Informátic Universidd de Cstill-L Mnch 271- Albcete, Spin frcsdo, bermu, pco, JoséDuto Depto. Informátic de Sistems y Computdores Universidd Politécnic de Vlenci Vlenci, Spin jduto@gp.upv.es Abstrct 1. Introduction High-speed locl re networks (LANs) consist of set of switches connected by point-to-point links, nd hosts linked to switches through network interfce crd. Highspeed LANs my chnge their topology due to switches nd hosts being turned on/off, link rempping, nd component filures. In these cses, distributed reconfigurtion lgorithm nlyzes the topology, computes the new routing tbles, nd downlods them to the corresponding switches. Unfortuntely, in most cses, user trffic is stopped during the reconfigurtion process to void dedlock. Although network reconfigurtions re not frequent, sttic reconfigurtion such s this my tke hundreds of milliseconds to execute, thus degrding system vilbility significntly. Severl distributed rel-time pplictions hve strict communiction requirements [9, 11]. Distributed multimedi pplictions hve similr, lthough less strict, qulity of service (QoS) requirements. Both stopping pcket trnsmission nd discrding pckets due to the reconfigurtion process prevent the system from stisfying the bove requirements. Therefore, in order to support hrd rel-time nd distributed multimedi pplictions over high-speed LAN, we need to void stopping user trffic nd discrding pckets when the topology chnges. In this pper, we propose new dedlock-free distributed reconfigurtion lgorithm tht is ble to synchronously updte routing tbles without stopping user trffic. This lgorithm is vlid for ny topology, including regulr s well s irregulr topologies. Simultion results show tht the behvior of our lgorithm is significntly better thn for other lgorithms bsed on spnning-tree formtion. This work ws prtly supported by the Spnish CICYT under Grnt TIC C4, nd Cj Cstill-L Mnch. High-speed locl re networks (LANs) consist of set of switches connected by point-to-point links, nd hosts linked to switches through network interfce crd (NIC). Current high-speed LANs (Autonet [8], Myrinet [2], nd ServerNet [1]) use techniques tht hve been successfully pplied in interconnection networks for prllel computers, such s point-to-point links between switches nd pipelined switching techniques. These networks hve lso inherited some chrcteristics from conventionl LANs, such s wiring flexibility nd topology vribility. The unique properties of high-speed LANs give rise to some problems relted to topology configurtion nd messge routing. In prticulr, high-speed LANs my chnge their topology due to switches nd hosts being turned on/off, link rempping, nd component filures. In these cses, in order to provide high system vilbility, reconfigurtion lgorithm must updte the routing tbles so tht communiction is possible between different components, s long s the network remins connected. Reconfigurtion mechnisms in current high-speed LANs re bsed on sttic reconfigurtion techniques. Autonet [8] is, perhps, the most representtive exmple. In this technique, distributed reconfigurtion lgorithm is triggered when significnt chnge in the topology occurs, spreding it to the whole network, nd updting the routing tbles in hosts nd switches. This lgorithm voids dedlocks during the reconfigurtion process by stopping ppliction trffic before strting the reconfigurtion process. When the reconfigurtion finishes, pcket trnsmission is llowed to resume. As consequence, performnce degrdtion of the interconnection network is produced. In [16, 13] the reconfigurtion effect on verge pcket ltency is nlyzed. Figure 1 shows this effect.

2 Instntneous ltency (cycles) Simultion time (cycles x 1) Figure 1. Consequences of sttic reconfigurtion in Autonet. Under norml conditions, the network ltency vries from 4 to 1 cycles. However, ech reconfigurtion needs from 2 to 25 cycles. These vlues depend on prmeters such s topology or lod. Severl distributed rel-time pplictions hve strict communictions requirements [9, 11], with rigorous limittions on CDV (cell dely vrition), CTD (mximum cell trnsfer dely) nd CLR (cell loss rtio) prmeters. Distributed multimedi pplictions hve similr, lthough less strict, qulity of service (QoS) requirements. Nowdys, mny distributed multimedi pplictions such s rel-time video compression nd decompression, video-ondemnd servers, distributed dtbses, etc., require computing power beyond tht vilble in current uniprocessors. These pplictions require very high network bndwidth, which cn be provided by mens of high-speed LAN. When multimedi pplictions re executed on locl re switch-bsed network, topology chnges (for exmple, due to components filures) my ffect their behvior. If sttic reconfigurtion is used, user trffic is stopped nd the verge pcket ltency increses drmticlly during the reconfigurtion. Thus, it will not be possible to gurntee the required QoS [12]. Stopping user trffic hs even worse effects on distributed rel-time pplictions. In our pproch, we tckle reconfigurtion of the interconnection network from dynmic point of view by performing network reconfigurtion without stopping the trnsmission of user pckets. The ppliction of dynmic reconfigurtion technique will reduce the negtive effects of the reconfigurtion process, eliminting the spikes observed in Figure 1. Dynmic reconfigurtion provides higher system vilbility [14] nd is especilly suitble to support distributed multimedi pplictions, which require gurnteed QoS. It should be noted tht dynmic reconfigurtion by itself does not provide QoS gurntees. However, the converse is true: if dynmic reconfigurtion is not implemented, it will be impossible to gurntee QoS during reconfigurtion becuse messge trffic will be stopped for tens or hundreds of milliseconds. Dedlocks re the most importnt problem tht rises when using dynmic reconfigurtion. As switches operte synchronously [2, 1], it is not possible to updte the routing tbles of severl switches t once. For this reson, if user trffic is not stopped during reconfigurtion process, certin switches will route messges ccording to the old routing tble while other switches will lredy be using the new one. Gurnteeing dedlock freedom when fcing this sitution is complicted [3]. In this pper, we propose new distributed dedlockfree dynmic reconfigurtion lgorithm suitble for generic topologies, including irregulr ones. This lgorithm is ble to synchronously updte the routing tbles without stopping user trffic. It hs been developed for virtul cutthrough (VCT) switching becuse it is esier to void dedlocks in VCT networks thn in wormhole (WH) networks. This is not serious constrint becuse VCT my replce WH in the ner future to trnsmit messges in networks of worksttions (NOW) [7]. We lso evlute the performnce of the proposed reconfigurtion lgorithm, compring it with conventionl techniques bsed on previous spnning-tree formtion. In Section 2, we present n informl description of the lgorithm. Section 3 introduces different spects relted to the up*/down* routing lgorithm. Section 4 presents our dynmic reconfigurtion technique, clled Prtil Progressive Reconfigurtion. Finlly, the performnce of the dynmic reconfigurtion technique is evluted in Section 5, nd the lst section presents our conclusions. 2. Informl Description This section describes the protocol for dynmic reconfigurtion in n informl wy. As indicted in the introduction, the key contribution of this protocol is its bility to updte routing tbles synchronously without stopping messge trffic while gurnteeing the bsence of dedlock. We ssume distributed routing. The extension of this protocol to source routing is the subject of ongoing reserch. The first step in network reconfigurtion is detecting the ddition nd removl (or filure) of network components (links, switches, nd/or hosts). This issue hs been ddressed elsewhere [8, 16, 13] nd will not be nlyzed here. Once chnge in the topology hs been detected, it is necessry to updte the routing tbles t one or more switches nd/or hosts. Our primry focus is on chieving distributed updte of the routing tbles without stopping messge trffic nd without introducing dedlocks. As most commercil switches do not provide ny support to synchronize tht opertion, routing tbles must be updted

3 synchronously. It is importnt to note tht chnges in the host routing tbles cnnot led to dedlocks in the network, provided tht the routing lgorithm implemented by the switches is dedlock-free. The reson is tht hosts only specify the destintion switch but not the pth to rech it. However, switch routing tble updtes my led to dedlock. Therefore, in this pper we will only focus on updting switch routing tbles without introducing dedlock. Severl reserchers proposed distributed dedlock-free routing lgorithms for irregulr interconnection networks [8, 15, 19, 18, 1], s well s generl methodologies for the design of routing lgorithms [19, 18]. A strightforwrd wy to void dedlock consists of removing cyclic dependencies between network resources (i.e., links nd buffers) [5]. It my seem tht when the topology chnges, we only need to define dedlock-free routing lgorithm for the new topology nd updte the routing tbles. This would be true if ll the routing tbles could be updted synchronously nd the switches purge pending messge trffic. As this is not possible, previously proposed solutions either my led to dedlock or require stopping messge trffic until ll the routing tbles hve been updted. The reson is tht the new routing lgorithm my introduce some resource dependencies tht did not exist in the old one. Of course, it will hve to remove other dependencies to void cyclic dependencies nd dedlock. The problem rises when routing tbles re updted synchronously becuse the new dditionl dependencies my rise before the old ones re removed, possibly leding to dedlock. This problem cnnot be solved by estblishing n pproprite ordering to updte the routing tbles. Even for some very smll networks, we found tht there is no sequence of switch routing tble updtes tht could gurntee dedlock freedom t ll times. It is importnt to note tht every routing tble updte t given switch must led to connected routing lgorithm, i.e., the routing lgorithm must be ble to route messges destined for ny host t ny switch 1. Otherwise, some messges could not be routed nd would hve to be discrded or would remin in the network. As mentioned bove, in generl there is no sequence of routing tble updtes tht, fter ech tble updte, leds to connected nd dedlock-free routing lgorithm. The solution proposed in this pper consists of performing sequences of prtil routing tble updtes. When the routing tble is updted t switch, the proposed protocol does not replce the old routing tble with the one corresponding to the new routing lgorithm. Insted, entries in ech tble re progressively removed nd dded systemticlly until the routing tble corresponding to the new routing lgorithm is reched. After ech prtil updte, ech switch must syn- 1 Sometimes, this my be impossible to chieve just fter some tble updtes. In those cses, the next few updtes must re-connect the routing lgorithm gin. chronize with some of its neighbors. The protocol proposed in this pper gurntees tht the globl routing lgorithm remins dedlock-free t ll times. The proposed protocol is very efficient. It interleves short control messges between user messges. However, we need some intelligence in the switches to decode control messges nd perform the required ctions. A single control messge serves two purposes: crrying informtion bout the required updte in the routing tble t the receiving switch, nd gurnteeing tht ll the messges tht hve to be routed with the old routing tble hve lredy been forwrded from the sender switch. Therefore, prticulr switch processes messges with the current routing tble until it receives control messge requesting prtil tble updte. When this messge is received, the routing tble is immeditely updted, nd successive messges rriving through the sme link re routed ccording to the updted routing tble. The efficiency of the proposed protocol comes from the fct tht tble updte dely is severl orders of mgnitude shorter thn the time required to stop trffic in ll the network nd downlod the new routing tbles. Moreover, dynmic reconfigurtion only ffects (usully smll) region of the network. Trffic in the regions of the network not requiring routing tble updtes is not ffected t ll by the reconfigurtion process. However, with sttic reconfigurtion, trffic is stopped in the entire network until ll the routing tbles hve been updted. In this pper we hve focused on the up*/down* routing lgorithm [8]. This routing lgorithm does not provide good performnce in mny cses. However, it is esy to extend our reconfigurtion technique to more efficient dptive routing lgorithms tht use up*/down* routing s escpe chnnels to void dedlock [19, 18]. The up*/down* routing lgorithm defines logicl tree in the network. Messges re first routed towrd the root of the tree until they find common ncestor. Then messges re routed down the tree until they rech the destintion switch. For this lgorithm, we hve found tht it is esy to dd switches without introducing cyclic dependencies between resources. Simply, we dd them s leves of the tree. Unfortuntely, this my be inefficient in some cses becuse messges cnnot be routed through lef switches. Therefore, we my need to reconfigure the tree to mke it more efficient. Moreover, if the new switch connects two subnetworks tht were initilly isolted 2, it will become lef belonging to two trees. This sitution is depicted in Figure 3. The lef switch is node q. It should be noted tht both subnetworks remin disconnected fter dding the new switch becuse the up*/down* routing lgorithm does not llow 2 Although we ssume tht the network ws initilly connected, this cse corresponds to the sitution when cluster is expnded by dding nother (recently instlled) cluster becuse the new cluster is usully fully tested before interconnecting it with the old one.

4 trffic through lef switches. Additionlly, now the network hs more thn single root node. Similrly, when the root switch of tree is removed, two or more root nodes my pper. Figure 4 shows n exmple. Agin, the up*/down* routing lgorithm does not llow trffic between root nodes, which become logiclly isolted. In order to llow trffic between ll the switches, the tree hs to be reconfigured. This cn be done by chnging the direction of the links in the tree. As n exmple, Figure 9 shows the network presented in Figure 3 fter chnging the direction of some links. As cn be seen, there is single root node fter reconfiguring the tree. These link direction chnges must be performed systemticlly, possibly updting ech routing tble prtilly, nd synchronizing with neighbor switches. In order to simplify link direction chnges, we nlyze the network in hierrchicl wy. Severl switches cn be grouped together, forming region. Figure 5 shows how the switches in Figure 3 cn be grouped into severl regions. Note tht the use of regions considerbly simplifies the network grph (see Figure 7) while retining the importnt properties, i.e., the existence of more thn single root node. Finlly, in order to estblish the order nd the consequences of link direction chnges, nd gurntee the correctness of the proposed protocol, we propose some definitions (root node, brek node, etc.). These definitions will considerbly simplify link direction chnges through nontrivil results. For exmple, moving the position of the root switch in the network does not require ny chnge in the routing tbles s long s it does not cross ny brek node. The following sections propose some definitions, nd formlly present the dynmic reconfigurtion protocol. 3. The Up*/Down* Routing Algorithm Up*/Down* routing is prtilly dptive dedlock-free routing lgorithm suitble for regulr nd irregulr topologies. This lgorithm is bsed on cycle-free ssignment of direction to the opertionl links in the network. This ssignment is lwys possible, regrdless of network topology [8]. Therefore, the network is configured s n cyclic directed grph. For ech link, direction is nmed up nd the opposite one is nmed down. To void dedlocks, legl routes never use link in the up direction fter hving used one in the down direction. Messges cn cross zero or more links in the up direction, followed by zero or more links in the down direction. In this wy, cycles in the chnnel dependency grph [5] re voided, thus preventing dedlock. Figure 2 shows n exmple where ech link hs been ssigned direction. Arrows indicte the up direction. In this pper, the grphs will only include switches Properties of Correct nd Incorrect Grphs for Up*/Down* Routing A root node is node in directed grph tht is not the source of ny rc. The up*/down* routing lgorithm requires the existence of single root node in the grph. The reson is tht there re no legl routes between two root nodes becuse ech possible route would require down to up trnsitions. This restriction is required for network connectivity. In Figure 2, the root node is node d. A brek node is node tht is the source of two or more rcs. In the up*/down* routing lgorithm, these nodes prevent certin connections (input port - output port) from being used by the messges crossing them. These restrictions re necessry for dedlock freedom. There must exist one brek node in every cycle, but its position is unrestricted. We cn see two brek nodes lbeled s c nd f in Figure 2. d Figure 2. Exmple of link ssignment in n up*/down* routing lgorithm. In order to void dedlocks, certin routes such s (! c! d) nd(c! f! e) re not llowed. In up*/down* routing, the ssocited directed grph will contin one nd only one root node. Additionlly, tht grph will be cyclic. A directed grph tht is cyclic nd contins single root node is clled correct grph. A correct grph my include severl brek nodes within its topology, s mny s necessry to brek ll the cycles. Figure 2 shows correct grph. Obviously, n incorrect grph is one tht does not meet the restrictions imposed in the previous definition. This implies the bsence of root node, the existence of more thn one root node, or the existence of cycles. If there is no root node, then the grph will contin one or more cycles nd up*/down* routing cnnot gurntee dedlock freedom. If there re severl root nodes, then up*/down* routing cnnot gurntee network connectivity. There is lwys t lest one flse brek node between two root nodes. A flse brek node is brek node in which two links with the down end connected to it do not belong to the sme cycle in the undirected grph of the network. A flse brek node splits the network into two unrechble regions. Obviously, correct grph contins no flse brek nodes. Figure 3 shows n exmple of incorrect grph with severl root nodes. f c b e

5 b e f g c h q n d l p j o Figure 3. Incorrect grph. Root nodes re lbeled s, i, ndr, respectively. Brek nodes re lbeled s c, g, h, m, p, u, q, nds, respectively. The lst two re flse brek nodes. i m k s r u t Switch Dectivtion When switch detects tht one of its neighbors hs been dectivted, it strts reconfigurtion process similr to the previous one. Switch dectivtion cnnot leve the directed grph without root node, but it my produce the ppernce of severl new ones. The dectivtion of severl switches (including the root node) produces t lest one new root node. Note tht the directed grph is cyclic before switch dectivtion. Figure 4 shows n exmple of correct grph tht evolves into n incorrect grph fter the dectivtion of two switches, which produces severl root nodes. Switch dectivtions imply tht messges routed to removed components must be discrded. Also, messges requesting removed components must be discrded if they cnnot use nother route. In this cse, shorter reconfigurtion time implies less discrded messges Hndling Chnges in Topology For given network topology, up*/down* routing is bsed on link direction ssignment tht cn be represented s correct grph. When some switches re dded to or removed from the network, its topology chnges. Then, correct grph my evolve into n incorrect grph. Next, we detil these situtions Switch Activtion During the switch ctivtion process we will void incorrect grph situtions produced by either the bsence of root node or the presence of cycles in the grph. However, n incorrect grph my temporrily rise due to the ppernce of severl root nodes (it will be solved lter). When new switch is dded, direction must be ssigned to the links connecting to it. This ssignment should not produce cycles in the directed grph. A simple pproch consists of ssigning direction to those links in such wy tht the down direction goes towrd the new switch. By doing so, messges will be ble to use the new links to route to/from the new switch, but not to cross it. If the new switch is connected to the network through two or more links, it will become brek node or flse brek node. Moreover, it is possible tht two recently ctivted switches negotite the direction of the link connecting them. It is obvious tht the proposed protocol for switch ctivtion cnnot introduce ny cycle becuse ll the links connecting to newly ctivted switch hve been ssigned down direction towrd the new switch. As consequence, the new switch is either brek node or flse brek node. Therefore, it cnnot close ny cycle in the directed grph. b d c e Figure 4. Switch dectivtion. After the dectivtion of switches nd c, the root node disppers. Then, three new root nodes, lbeled s b, d, nde, pper. 4. Prtil Progressive Reconfigurtion When switch detects chnge in the topology of the network, it triggers reconfigurtion process through control messge. This messge is propgted to every node in the network by flooding. Ech node will mrk its routing tble s invlid, lthough it will still use it. Before chnging the routing tbles ccording to the new topology, it would be necessry to modify the direction of severl links to evolve from n incorrect grph into correct grph. We sw in the previous section tht node ctivtion/dectivtion voided incorrect grph situtions in which there is n bsence of root node. We lso sw tht the grph cnnot contin cycles. However, the directed grph my contin severl root nodes. In wht follows, we will show how to correct incorrect grphs with severl root nodes. b d c e

6 It should be noted tht fter the topology hs chnged, sttic reconfigurtion lgorithm stops trffic nd computes the direction ssignment for every link in the network strting from scrtch, i.e., it discrds the previous configurtion. On the other hnd, dynmic reconfigurtion lgorithm should not stop trffic. It progressively chnges the direction of the links, until it reches correct grph ccording to the new topology. For this reson we cll this dt structure n Adptive Directed Grph (). Now we present five-step dynmic reconfigurtion lgorithm to chieve this purpose Step 1: Genertion of Correct Regions b e f g c h q n d l p j o i m k s r u t When directed grph contins severl root nodes, it is not possible to route messges between root nodes. In this cse, it is possible to split the directed grph into severl correct subgrphs. A correct region is mximl subgrph of n incorrect grph tht is correct. The reconfigurtion process must determine the correct regions in the network. The network hs s mny correct regions s root nodes (one root node in ech correct region). Figure 5 indictes the correct regions of the grph shown in Figure 3. In order to determine those regions in distributed wy, fter receiving the messge indicting the strt of new reconfigurtion, ech node checks whether it is cting s deep brek node, i.e., ll the links incident on tht node hve their down end connected to it. Deep brek nodes exist becuse the grph is cyclic. Those nodes strt the process by sending DRP MAKE REGION messge through ll their output ports. The rest of the nodes expect those messges from every input link with the up end connected to them. When node hs received ll the expected messges (one on ech up input link), it forms new messge, combining informtion from ll the received messges, nd dding its own informtion, s explined below. This messge is forwrded through every output link with the down end connected to tht node. Messge propgtion finishes t root nodes. The purpose of DRP MAKE REGION messges is to gther informtion bout network topology in order to determine the correct regions in distributed wy. The DRP MAKE REGION formt is shown in Figure 6. Once this informtion is ggregted t the root node for ech region, this node distributes the complete messge downwrds nd ll the nodes in the region get informtion bout the topology of tht region Step 2: Obtining the Virtul Inter-Region Grph We cll those switches for which every upwrd link crosses the limits of region contining them boundry nodes. In Figure 5, these switches re nodes q nd s. A Figure 5. Splitting the grph into correct regions. switch s-1 switch s switch s+1 ID s n p1 ID p1 p1 p2 ID p2 p pn ID pn pn ID of current switch n output ports with the down end connected to them port 1 of n ID of the switch t the other end of the link input port t tht switch port 2 of n ID of the switch t the other end of the link input port t tht switch port n of n ID of the switch t the other end of the link input port t tht switch Figure 6. DRP MAKE REGION messge formt. The current node fills shded cells dding informtion bout itself. Upwrd nodes will fill the other cells when the messge reches them. boundry node between two regions is the only vlid communiction pth between them. We define virtul inter-region grph s the grph composed of ll root nodes, ll boundry nodes, nd the pths interconnecting them. Virtul grph informtion is distributed mong the nodes belonging to it, i.e., root nd boundry nodes exchnge virtul grph position pckets. When the process strts, ech node in the virtul grph only knows the existence of its neighbours in the virtul grph. When the informtion distribution finishes, ech node in the virtul grph knows the entire virtul inter-region grph. A virtul grph is n bstrction of the incorrect informtion in grph in order to simplify the proposed dynmic reconfigurtion lgorithm. Figure 7 shows the virtul grph

7 corresponding to the exmple shown in Figure Step 3: Correcting the Virtul Grph q i s r q i s r A virtul grph resulting from n incorrect grph is lso incorrect. In prticulr, it hs s mny root nodes s the originl grph. Therefore, it is necessry to modify the virtul grph to mke it correct by chnging the direction of some links. Ech node in the virtul grph mkes the incorrect virtul grph evolve into correct one by pplying modifiction of Dijkstr s lgorithm [6]. The lgorithm selects one of the root nodes s the primry root node nd itertively updtes virtul link direction ssignments until the virtul grph hs been corrected. It consists of the following steps: 1. A set C of corrected virtul nodes is defined. Initilly, only the primry root node belongs to C. 2. Every other virtul node directly connected to virtul node in set C by virtul link with up direction is dded to C. 3. A virtul node not belonging to C is selected, such tht stisfies the following three requirements. First of ll, it should hve virtul link connected to virtul node in C (in down direction). Second, chnge in the direction of this virtul link should not produce cycle in the virtul grph. And third, the corresponding virtul link must hve the lowest cost. We define this cost s the number of brek nd root node movements tht its chnge of direction implies (virtul links without chnge of direction hve null cost). 4. This process (2 nd 3) continues until every virtul node belongs to C. A criterion to select primry root node could be the root node with lower UID. Another option could be to select s the primry root node the one tht minimizes the globl cost (defined s the sum of every virtul link cost). When the virtul grph hs been corrected, only one primry root node remins in it. The other ones re no longer root nodes, nd will be referred to s secondry root nodes. The primry root node does not modify its position in the virtul inter-region grph. On the other hnd, secondry root nodes reverse the direction of the pth to the boundry node close to the primry root node. If secondry root node cn be ttrcted by severl boundry nodes, the criterion mentioned in step 3 selects the best one. Figure 7b shows the virtul grph in Figure 7 once it hs been corrected. () Figure 7. ) Virtul inter-region grph. The direction of ech link in the virtul grph mtches the direction of the complete pth it represents (composed of severl links in the grph); b) Correct virtul grph. After distributing the virtul grph informtion nd pplying the correction criterion, this grph evolves into correct one completely connected ccording to the up*/down* routing lgorithm Step 4: Correcting the Rel Grph Once the virtul inter-region grph hs been corrected, virtul link direction chnges must be propgted to physicl links. Some conditions should be met when chnging the direction of the links in the network. Every chnge in link direction should be crried out without disconnecting the routing lgorithm. As consequence, only the direction of the links connected to root or brek nodes cn be chnged. Otherwise, some nodes my be unrechble fter chnging link direction. Figure 8 shows direction chnge in link tht is not connected to ny of these nodes. Tking into ccount the previous restriction, the rel grph is corrected s follows. Ech secondry root node begins its movement, exchnging its position with neighboring node; this one exchnges its position with the following node, nd so on. The movement of root node requires the individul nd sequentil direction reversl of the corresponding links. From now on, we will use the expression to move the root/brek node lthough in fct we re only reversing link direction ssignment. The movement of secondry root node does not produce ny chnge in the routing tbles nd therefore it does not cuse dedlocks, s shown in Figure 8b. In up*/down* routing, dedlocks re voided by the restrictions imposed on brek nodes. Therefore, the position of the root node is irrelevnt. For this reson, n direction chnge in link connected to root node does not ffect the routing tbles. We cn even perform the movement cross severl nodes in single step. There is specil sitution when root node reches brek node. As shown in Figure 8c, moving root node cnnot move over brek node becuse cycle rises nd

8 Brek node Root node Brek node Root node Root node Root node Root node Root node Brek node Brek node Brek node Brek node () (c) Figure 8. ) Direction chnge in link not connected to root or brek nodes. After the chnge in the lower link direction, the network is split into two disconnected regions. There re two brek nodes nd two root nodes in the sme cycle; b) Root node movement. The movement of the root node does not ffect the routing tbles; c) Movement of root node over brek node. A cycle ppers in the directed grph fter chnging the direction of the leftmost link. The resulting grph hs no root node. the root node disppers. With this restriction, it my hppen tht there is no vlid pth between the secondry root node nd the boundry node tht must be reched. In Figure 5, node must rech node q but brek nodes c, g, ndh void it. The solution consists of moving s mny brek nodes s necessry first, keeping them wy from the pth followed by the secondry root node. In [4], we show how brek node movements re performed without producing dedlocks. This movement hs high ssocited cost, requiring the synchroniztion of severl nodes. For this reson, the pth tht minimizes the number of brek node movements should be chosen. The secondry root node determines this pth. Figure 9 shows the initil rel grph once tht it hs been corrected. h q Figure 9. Corrected rel grph. Node cn be moved through severl pths. The selected pth only requires moving the brek node lbeled s h. i s r 4.5. Step 5: Updting the Routing Tbles This is not sepprte step. Indeed, routing tbles re updted through sequence of prtil tble updtes tht re embedded in steps 1 nd 4. Let us revise the reconfigurtion process, highlighting when routing tbles re updted. The reconfigurtion process defines the correct regions in the grph. Then, every node inside ech region cn generte its intr-region routing tble. The process finishes if the reconfigurtion process defines only one correct region in the grph. Otherwise, if severl correct regions pper within the grph, then it is necessry to determine the boundry nodes nd the virtul inter-region grph, nd to correct the virtul grph. A region disppers when the corresponding secondry root node reches boundry node. Then, the boundry node provides the topology of this region to the neighboring regions s shown in Figure 1. This process consists of brodcsting topology informtion to ll the nodes in the region by flooding. At this time, it is possible to generte the complete routing tble t ech node, becuse it knows the whole topology. This is done s usully for the up*/down* routing lgorithm. A finl considertion: when some components re removed from the network, routing tbles must be updted so tht messges re not routed towrd nonexistent components. If messge blocks becuse it requested nonexistent component before routing tbles were updted, it my hppen tht the only possible ction consists of discrding tht messge. Moreover, the updted routing function my not offer ny pth to some messges. In these cses, it could be necessry to send notifiction messge to the source node. This is the cse when link filure splits the network into two disconnected regions.

9 b e f g c h q n d l p j o Figure 1. Joining regions. Secondry root nodes must rech the corresponding boundry nodes (lbeled s q nd s, respectively). Then, boundry nodes exchnge topology informtion mong the regions in which they re included. At this moment, ech node inside the region cn complete its own routing tble. i m k s r u t the output chnnel for ech pcket s function of its destintion host, the input chnnel, the output chnnel sttus, nd the queue sttus t the next switch. Tble look-up routing is used. The routing unit cn only process one pcket heder t time. It is ssigned to witing pckets in round-robin fshion. When pcket cnnot be routed becuse the requested chnnels re busy, it must wit in the corresponding input buffer until its next turn. Ech input chnnel hs two sets of buffers: user nd control buffers. They hve cpcity for one or more pckets. Control buffers cn be smller thn user buffers. The reson is tht control buffers only hndle control messges generted in ech reconfigurtion process. User nd control buffers will only ccept new pcket if there is enough spce to store the whole pcket. We hve ssumed tht one clock cycle is required to ccess the routing tble nd provide the output link for messge, nd it tkes one cycle to trnsmit one byte cross the internl crossbr. Also, dt re injected into the physicl link t rte of one byte per cycle. As in current high-speed LANs, chnnel pipelining [17] hs been considered ssuming tht the fly time is 4 cycles for ech byte. For the results presented in this pper, we hve set the user buffer size to 4 pckets. User buffers 5. Performnce Evlution In this section, we evlute the performnce of the dynmic reconfigurtion technique (referred to s in the plots) proposed in this pper. We lso evlute sttic reconfigurtion technique for comprison purposes. This technique is bsed on the genertion of the propgtion order spnning-tree (), similr to the technique implemented in the Autonet network [16]. Insted of nlytic modelling, simultion ws used to evlute the reconfigurtion technique. Our simultors model the network t the phit 3 level. The evlution methodology used is bsed on the one proposed in [7] Switch Model Figure 11 shows the switch model. Ech switch consists of crossbr, routing nd rbitrtion unit, nd severl fullduplex links (composed of two unidirectionl chnnels in opposite direction). The crossbr llows multiple pckets to cross switch simultneously without interference. It is configured by the routing unit ech time successful route is estblished. The routing nd rbitrtion unit determines 3 Unit of informtion tht cn be trnsferred cross physicl chnnel in single step or cycle. control buffers Multiplexer Crossbr Routing nd rbitrtion unit Figure 11. Switch rchitecture Network Model The network is composed of set of switches nd hosts connected to them. Network topology is completely irregulr nd hs been generted rndomly. However, for the ske of simplicity, we imposed two restrictions to the topologies tht cn be generted. First, we ssumed tht there is one host connected to ech switch. Second, two neighboring switches re connected by single link. In generl, the number of ports connected to other switches is different for ech switch. This number vries between 2 nd 8 ports. The remining ports re idle. With respect to network size, we hve evluted networks with 8, 16, 24, 32, 48, nd 64 switches.

10 5.3. Pcket Genertion Messge destintion is rndomly chosen mong ll the hosts in the network. On the other hnd, for the genertion rte we hve considered two trffic models: rndom nd feedbck model. In the former, messges re rndomly generted but the verge genertion rte is constnt nd the sme for ll the hosts. In the ltter, we simulte the behvior of client/server pplictions in the sense tht when messge is delyed, the lte rrivl of this messge will lso dely its processing nd the injection of new messges into the network. For pcket length, 16-byte nd 64-byte pckets were considered. Nevertheless, we hve found tht pcket length does not ffect very much the reltive performnce of the reconfigurtion techniques. Thus, we will only show the results for short pckets Simultion Results In this section, we show the performnce evlution results. We compre the performnce of our dynmic reconfigurtion technique () with the sttic reconfigurtion technique (). Figure 12 shows the instntneous ltency versus simultion time for networks with 16 nd 32 switches, respectively, when uniform trffic is considered. Figure 13 shows the sme results but considering the feedbck trffic model. In these figures, the top plot corresponds to the results obtined when the technique is pplied, nd the bottom plot shows the results for the technique. Also, in ech plot, we cn observe the effects of two reconfigurtion processes: switch dectivtion first nd switch ctivtion lter. In this pper, we show the results corresponding to the worst sitution for the technique: dectivtion/ctivtion of the root switch. Pcket length is 16 bytes, corresponding to n verge ltency of pproximtely 5 cycles in the bsence of reconfigurtion. Figure 12 shows reconfigurtion process due to the dectivtion nd ctivtion of switch. For simultion time equl to cycles, we cn see time intervl of severl thousnds of cycles in which messges re not trnsmitted in the network. The bsence of messges is due to stopping user trffic during the reconfigurtion. After reconfigurtion, messges witing to be injected produce n importnt increse of the injection rte, sturting the network nd significntly incresing pcket ltency. After some time, this congestion dissiptes. The time tht the network cnnot provide suitble service is the sum of the reconfigurtion time nd the time it is sturted. On the other hnd, when the technique is pplied, user trffic is only slightly ffected during the reconfigurtion process. The network does not stop its ctivity, nd pcket ltency slightly increses during short period of time fter strting Instntneous ltency (cycles) Simultion time (cycles x 1) () Instntneous ltency (cycles) Simultion time (cycles x 1) Figure 12. Instntneous ltency versus simultion time for n irregulr network composed of () 16 switches nd 32 switches. Trffic is uniform nd pcket length is 16 bytes. the reconfigurtion. Note tht we considered the worst cse network reconfigurtion for the technique. Feedbck trffic models client/server pplictions. During the reconfigurtion process, the hosts do not receive messges nd, s consequence, they do not inject new messges into the network. When user trffic is llowed gin, congested sitution like the one described bove is not produced. Nevertheless, some pckets experience very high ltency when using the reconfigurtion technique. This cn be seen in Figure 13. The most importnt problem does not pper t the network level but t the ppliction level. During the time intervl just fter reconfigurtion, the trffic corresponds to the sme requests tht were discrded during the reconfigurtion. Discrding pckets prevents the network from gurnteeing QoS. Figures 14 nd 15 show the mount of pckets discrded when the reconfigurtion process strts. In cse of the sttic reconfigurtion technique, these pckets come from emptying the buffers when the reconfigurtion process is triggered. For tht reson, the number of discrded pckets increses when the network size increses. However, the mount of pckets discrded during dynmic reconfigurtion depends on the network size only in n indirect wy. The figures show tht the number of discrded pckets is much lower when dynmic reconfigurtion is used. Also, it cn be seen tht when the reconfigurtion consists of the ctivtion of new switches there re not ny discrded pckets when using dynmic reconfigurtion. The reson is tht no legl route between two hosts disppers. Only new routes re dded to the network. Finlly, it seems tht the kind of trffic (rndom or feedbck trffic) does not significntly influence the mount of discrded pckets.

11 Instntneous ltency (cycles) Discrded pckets Simultion time (cycles x 1) () Instntneous ltency (cycles) Simultion time (cycles x 1) Figure 13. Instntneous ltency versus simultion time for n irregulr network composed of () 16 switches nd 32 switches. Trffic follows the feedbck model nd pcket length is 16 bytes Network size (switches) () Discrded pckets Network size (switches) Figure 14. Discrded pckets versus network size for irregulr networks with rndom trffic; ) switch dectivtion; b) switch ctivtion. We hve nlyzed the benefits provided by the proposed reconfigurtion lgorithm to user trffic. Now, we nlyze how long it tkes to reconfigure the network. Figure 16 shows reconfigurtion time versus network size. In this figure, we cn see tht our pproch requires less time thn the methodology, nd this behvior remins s network size increses. Also, s our reconfigurtion lgorithm does not stop user trffic, it my be thought tht the reconfigurtion time will depend on network lod. Figure 17 shows reconfigurtion time versus trffic for network with 32 switches. In this figure, we cn see tht reconfigurtion time is not ffected by network lod. The reson is tht control messges re interleved with user messges by the switches whenever required. Therefore, reconfigurtion is not significntly delyed by user trffic disble disble Network size (switches) () enble enble Network size (switches) Figure 16. Reconfigurtion time versus network size for irregulr networks with rndom trffic; ) switch dectivtion; b) switch ctivtion. Discrded pckets Network size (switches) () Discrded pckets Network size (switches) Figure 15. Discrded pckets versus network size for irregulr networks with feedbck trffic model; ) switch dectivtion; b) switch ctivtion. Reconf. time (cycles) Trffic (flits/cycle/node) () Reconf. time (cycles) Trffic (flits/cycle/node) Figure 17. Reconfigurtion time versus rndom trffic for irregulr network composed of 32 switches; ) switch dectivtion; b) switch ctivtion.

12 6. Conclusions Severl distributed pplictions demnd high system vilbility. Additionlly, multimedi pplictions require some qulity of service (QoS) gurntees. In prticulr, udio nd video strems must be delivered within some dedline, lso minimizing jitter. When executed in locl environment, distributed multimedi pplictions require high-speed LANs with point-to-point links. Unfortuntely, high-speed LANs my suffer frequent reconfigurtions due to the ctivtion nd dectivtion of switches nd hosts, link rempping, nd component filures. In these cses, distributed reconfigurtion lgorithm nlyzes the topology, computes the new routing tbles, nd synchronously downlods them to the corresponding switches. Current reconfigurtion lgorithms stop user trffic during routing tble updte to void dedlocks. As consequence, these networks re not suitble to support certin multimedi pplictions becuse messge ltency my increse by three orders of mgnitude during the reconfigurtion, nd consequently QoS my no longer be gurnteed. In this pper, we hve proposed new distributed dynmic reconfigurtion lgorithm tht synchronously updtes routing tbles without stopping user trffic. This lgorithm is vlid for ny topology nd gurntees the bsence of dedlocks during the reconfigurtion process. After introducing some grph concepts, this pper nlyzes the ctivtion nd dectivtion of network components, showing how to build network grph nd updte routing tbles in such wy tht no dedlock cn rise. However, the routing lgorithm my be disconnected, i.e., some prts of the network my be unrechble. Then, we propose distributed protocol tht produces sequence of prtil routing tble updtes, which re ble to reconnect the routing lgorithm. We show tht the routing lgorithm t every intermedite step is dedlock-free, nd tht the finl routing lgorithm correctly routes messges between every pir of nodes. We lso show by simultion tht the negtive effects (high pcket ltency nd discrded pckets) produced by the reconfigurtion process re considerbly reduced if we pply dynmic reconfigurtion technique. In summry, we conclude tht the dynmic reconfigurtion technique proposed in this pper significntly outperforms previously proposed reconfigurtion techniques becuse it does not require stopping user trffic during the reconfigurtion. Our reconfigurtion technique is especilly suitble for systems tht hve to provide QoS gurntees. Without using our reconfigurtion technique, it is not possible to gurntee QoS when the topology chnges. References [1] B. Abli. A dedlock voidnce method for computer networks. In Lecture Notes in Computer Science. Springer, Februry Proceedings of the CANPC 97. [2] N. Boden, D. Cohen, R. Feldermn, A. Kulwik, C. Seitz, J. Seizovic, nd W. Su. Myrinet - gigbit per second locl re network. IEEE Micro, pges 29 36, Februry [3] R. Csdo, M. Cminero, P. Cuenc, F. Quiles, A. Grrido, nd J. Duto. A tool for the nlysis of reconfigurtion nd routing lgorithms in irregulr networks. In Lecture Notes in Computer Science, volume 1362, pges Springer, Februry Proceedings of the CANPC 98. [4] R. Csdo, F. J. Quiles, J. L. Sánchez, nd J. Duto. An efficient protocol for dynmic reconfigurtion in irregulr networks. Technicl Report DIAB , Universidd de Cstill-L Mnch, October [5] W. Dlly nd C. Seitz. Dedlock-free messge routing in multiprocessor interconnection networks. IEEE Trnsctions on Computers, C-36(5): , My [6] E. Dijkstr. A note on two problems in connection with grphs. Numericl mthemtics, Oct [7] J. Duto, A. Robles, F. Sill, nd R. Beivide. A comprison of router rchitectures for virtul cut-through nd wormhole switchin in NOW environment. In Procs. of the 13th Int. Prllel Processing Symposium, April [8] M. Schroeder et l. Autonet: A high-speed, self-configuring locl re network using point-to-point links. IEEE Journl on Selected Ares in Communictions, 9(8), October [9] A. Forum. ATM Forum trffic mngement specifiction. Version 4., My [1] R. Horst. TNet: A relible system re network. IEEE Micro, pges 36 44, Feb [11] G. Krlsson. Asynchronous trnsfer of video. IEEE communiction Mgzine, 24(8): , August [12] E. Knightly nd H. Zhng. D-BIND: An ccurte trffic model for providing QoS gurntees to VBR trffic. IEEE Trns. on Networking, 5(2), April [13] S. Owicki nd A. Krlin. Fctors in the performnce of AN1 computer network. Technicl Report 88, SRC reserch, June [14] G. Pfister. In serch of clusters. Prentice Hll, Englewood, NJ, [15] W. Qio nd L. Ni. Adptive routing in irregulr networks using cut-through switches. In Proceedings of the 1996 Int. Conference on Prllel Processing, August [16] T. Rodeheffer nd M. Schroeder. Automtic reconfigurtion in Autonet. Technicl Report 77, SRC reserch, September [17] S. Scott nd J. Goodmn. The impct of pipelined chnnels on k-ry n-cube networks. IEEE Trnsctions on Prllel nd Distributed Systems, 5(1):2 16, Jnury [18] F. Sill nd J. Duto. Improving the efficiency of dptive routing in networks with irregulr topology. In Proceedings of the 1997 Int. Conf. on Prllel Processing, August [19] F. Sill, M. Mlumbres, A. Robles, P. López, nd J. Duto. Efficient dptive routing in networks of worksttions with irregulr topology. In Workshop on Communictions nd Architecturl Support for Network-bsed Prllel Computing, Februry 1997.