Design and Evaluation of a Fault-Tolerant Adaptive Router for Parallel Computers

Transcription

1 Design and Evaluation of a Fault-Tolerant Adaptive Router for Parallel Computers Tsutomu YOSHINAGA, Hiroyuki HOSOGOSHI, Masahiro SOWA Graduate School of Information Systems, University of Electro-Communications, Chofu-shi, Tokyo, , Japan. yosinaga@is.uec.ac.jp Abstract In this paper, we propose a design methodology for faulttolerant adaptive routers for parallel and distributed computers. The key idea of our method is integrating minimal and non-minimal routing that is supported by independent virtual channels (VCs). Distinguishing the routing functions for each set of VCs simplifies the design of fault-tolerant algorithms. After describing the method, we show an application of a routing algorithm for two-dimensional mesh and torus networks. This algorithm, called Detour-NF, supports three routing modes: deterministic, minimal fully adaptive and non-minimal fault-tolerant operations. We also discuss the hardware cost and operational speed of minimal and non-minimal routers based on our design, which uses hardware description language (HDL). Communication performance and fault-tolerance are demonstrated by an HDL simulation. The experimental results show that supporting both minimal and non-minimal routing modes is advantageous for high-bandwidth and low-latency communication, as well as fault-tolerance. Keywords: fault-tolerance, adaptive router, nonminimal routing algorithm, hardware design, communication performance 1. Introduction Recent high-performance parallel and distributed computers consist of thousands of nodes or processing elements. For such large systems, communication performance as well as fault-tolerance is important. The adaptive routing technique is an approach to obtaining high bandwidth and low latency communication by its dynamic selection of message paths based on network state. By avoiding congestion areas or faults, messages can be routed more smoothly than with deterministic routing. In spite of such potential ability, adaptive routers have not been utilized widely in real computer systems. One reason for their limited use is that adaptive routers don t keep in-order message delivery. Since the software implementation for in-order delivery involves a large overhead, system designers often adopt a deterministic routing algorithm. However, if only a limited number of messages require in-order delivery and the rest of the messages can be routed without this constraint of delivery order, the deterministic routing decision for all messages is unnecessarily strict. A solution is to support both adaptive and nonadaptive routing modes. If a user could choose the preferred routing mode on a message or application basis, then the system could take advantage of both adaptive routing and in-order delivery. Adaptive routing algorithms can be classified into two categories: minimal and non-minimal path selections [4]. Minimal adaptive algorithms are usually used to increase performance by improving the utilization of network resources, such as channels. However, they don t provide fault-tolerance. Therefore, non-minimal adaptive routing is required for reliable communication against faults. Once a non-minimal adaptive routing algorithm is supported, it is not so difficult to use the router as a minimal router by restricting path selection to the minimal paths. Generally, non-minimal routers require more virtual channels (VCs) than minimal routers for deadlock prevention, even though the additional VCs may rarely be used when there are no faults. In the static fault model, (that is, a model in which faults can be detected before starting computation and dynamic faults do not appear during computation), these VCs can be more effectively utilized under the minimal routing constraint. Our goal is to design a fault-tolerant adaptive router which is able to switch its routing operation mode without wasting hardware resources such as VCs. In this paper, we assume the static fault model. First, we consider the design methodology of non-minimal routing algorithms. Then, we show an example design of a non-minimal adaptive router. We also analyze the hardware costs and performance of our design using hardware description language (HDL).

2 2. Related Works There have been several studies that combine deterministic and adaptive routing modes in a router. Fulgham and Snyder proposed the Triplex router, which supports oblivious, minimal and non-minimal fully adaptive routing classes [7]. This router basically provides a non-minimal class for congested networks. Millar and Najjar designed a hybrid deterministic and adaptive router [11]. This router tries to apply low-latency deterministic routing at low traffic with the flexibility of adaptive routing at high traffic. Therefore, fault-tolerance is not the main issue in either of these cases. Most fault-tolerant adaptive routing algorithms for direct networks have been discussed for -ary -cubes and wormhole switching [3]. Linder and Harden proposed an adaptive and fault-tolerant wormhole routing algorithm using the concept of a virtual network [9]. Their algorithm requires VCs per physical channel, and additional VCs to support faulty channels. Glass and Ni proposed the turn model, [8] which tolerates up to faults in an - dimensional mesh. The turn model presents partially adaptive routing algorithms that do not perform well in fault-free cases [1]. Boppana and Chalasani proposed fault rings and fault chains which show the boundaries of faulty components. The fault rings and fault chains can handle block faults on a two-dimensional mesh with four VCs per physical channel. Suh proposed software-based rerouting as a cost-effective alternative to the hardware solution [13], although the performance is relatively low. Duato proposed a methodology for the design of faulttolerant, fully adaptive routing algorithms [5]. He defined the redundancy level of a network, which represents the maximum number of simultaneous faults. In his methodology, a designer sets the redundancy level first. Then, a faulttolerant routing function is defined by repeating the steps for adding VCs to extend the non-minimal routing function and by removing redundant output channels by checking whether or not the extended channel dependency graph is acyclic. With Duato s methodology, we can generally design an appropriate, fault-tolerant, fully adaptive routing algorithm, although it is complex. In this paper, we propose an alternative methodology to design fault-tolerant, fully adaptive routing algorithms. This design method is simpler than Duato s method with respect to distinguishing minimal and non-minimal VCs. 3. Design Methodology To preserve simple switch-ability between minimal and non-minimal routing modes, our design method for faulttolerant, adaptive routing algorithms starts from an existing minimal fully adaptive routing function R. Normally, R requires a few VCs per physical channel to prevent deadlocks. Since R supplies minimal paths to messages, we identify these VCs as minimal VCs. The design steps adding fault-tolerance to R are as follows: 1. Add one or more VCs to each physical channel. Let us identify these additional VCs as non-minimal VCs. 2. Define a routing function R1 which specifies how to use the non-minimal VCs. Once a message enters the non-minimal VC, R1 supplies the non-minimal VCs only for a misrouting message until the message is delivered to its destination. Therefore, there is no channel dependency from the non-minimal VCs to the minimal VCs. The channel dependency graph of R1, which is constructed from the non-minimal VCs, should be acyclic to avoid deadlocks. 3. Combine the original minimal fully adaptive routing function R and the non-minimal routing function R1 such that a message selects a minimal path supplied by R as much as possible. When the message is blocked by a faulty channel and there are no alternative minimal paths to its destination, it is misrouted to a nonminimal path supplied by R1. 4. Finally, if they are necessary, 180-degree turns are added to support backtrack channels when messages switch their routing function from R to R1. The number of additional non-minimal VCs and the routing function R1 decide the fault-tolerance ability. A larger number of non-minimal VCs may handle a variety of faulty patterns, whereas fewer non-minimal VCs are costefficient for a network that is rarely faulty. The new routing function which is obtained by the combination of R and R1 is deadlock-free, because the original R and R1 are independently deadlock-free on the virtual networks organized by the minimal and non-minimal VCs, respectively. Livelock can be avoided by limiting the number of misrouting hops. The final step of the method is not always necessary, but it increases the misrouting flexibility when only the local faulty information is usable. 4. Routing Algorithm This section presents a fault-tolerant, adaptive routing algorithm for two-dimensional mesh and torus networks as an application of the method described in Section 3. For the minimal fully adaptive routing function R, we use Duato s Protocol, which requires two and three VCs per physical channel for meshes and tori, respectively [6]. One VC is an

3 adaptive VC, and the remaining one (in the mesh) or two (in the torus) are escape VC(s). By restricting the use of the adaptive VC as a non-adaptive one, it can easily operate as a deterministic router in order to keep in-order message delivery. For flow control, wormhole switching is used. The fault model that we consider here is a static one. Channel faults are assumed to be recognized only by their directly connected live routers, and no global faulty information is used Mesh algorithm As stated in the previous section, for the first step, we add one VC to each physical channel. Then, we define a non-minimal routing function R1. In order to maximize the fault-tolerance ability, which can be applied by a single non-minimal VC, we select the negative-first routing function [8]. Step 3 combines the minimal routing function R with the non-minimal function R1. To prevent performance degradation, misrouting is allowed for messages which have a single minimal path to their destination, and only when the path is blocked by a faulty channel. Figure 1 shows some examples of the allowed misrouting paths for messages that are going straight. Thick arrows represent the message paths supplied by the minimal fully adaptive function R, and thin U-shaped arrows represent the non-minimal paths for bypassing the faulty channels, which are drawn as dotted lines. Since R1 is a negative-first algorithm, it prohibits turns to the positive direction followed by the negative direction. However, as we show in Figure 1, any straight massage, except on the negative edge channels, can bypass the faulty channel. For example, when a message advancing from the bottom to the top is blocked by a fault, it turns to the left or negative direction in the x dimension, then it turns to the upward (positive) direction, and finally it turns to the right (positive) direction. In such a negative-positive-positive misrouting case, adaptive routing can be applied to the path selection of the positive-positive portion. On the other hand, a solitary misrouting path is applied for a negative-negative-positive bypass. This is because R1, the negative-first algorithm, is a partially adaptive routing algorithm. If R1 is defined by other variants of the turn model, such as north-last and west-first, we cannot guarantee the bypass for some directional messages because of the imbalance of prohibitive turns. We notice that all bypasses include at least one positive transmission followed by a negative transmission, and the bypass cannot start from a positive transmission. Therefore, when a message advancing to the positive direction is blocked by a fault at the final turn, it needs a 180-degree turn to start the bypass from a negative transmission. Otherwise, the misrouting path becomes longer or non-local faulty information is required. Here, we extend the routing function Positive direction Negative direction Routing path by R Misrouting path by R1 Faulty channel Positive direction Negative direction Figure 1. Misrouting examples for straight messages. Negative Positive Negative Routing path by R Misrouting path by R1 Faulty channel Positive Figure 2. Misrouting examples with 180- degree turns. to support 180-degree turns for messages that are moving to positive directions. Figure 2 shows examples of 180-degree turns when messages switch routing functions from R to R Torus algorithm The mesh algorithm is simple, but has the disadvantage that it cannot handle faults on the negative edge. We can design a torus algorithm, which provides support by utilizing wraparound channels that connect the nodes across the dateline. One way is preparing two non-minimal VCs that eliminate the occurrence of torus cycles. Another way is restricting the number of misrouting hops across the dateline to one hop, so that this does not cause the torus cycle even with a single non-minimal VC. This can be guaranteed in such a way that, once a message is misrouted from one dimension, it is not allowed to pass the wraparound channels on that dimension. In other words, the non-minimal routing function R1 would regard the network topology as a mesh. Figure 3 shows the misrouting paths of two messages in a two-dimensional torus. We assume that the dotted lines are datelines for each dimension. Message A is misrouted from the to the Y dimension, and it is not allowed to pass the wraparound channel on the dimension any more. So, the direction of message A is changed on the misrouting path of the dimension. Message B can be misrouted by only a

4 A S D West Port North Port East Port D B S... wire... PE S: source node D: destination node : faulty channels : dateline PE I/F South Port Figure 3. Misrouting paths on tori. single hop across the dateline of the Y dimension. Because messages which need more than two hops after passing the dateline do not exist, the torus cycle never occurs. We call the resulting routing algorithm Detour-NF. Its features are summarized as follows: It can support three routing modes: deterministic, minimal fully adaptive, and non-minimal fault-tolerant without wasting hardware resources. It suits networks where the fault rate is relatively low because it combines minimal fully adaptive and nonminimal fault-tolerant routing. It does not require global faulty information or routing table management. When there is no static faulty channel on the network, the user may use the Detour-NF router as a fully adaptive minimal router which has two escape VCs and two adaptive VCs per physical channel. Otherwise, non-minimal VCs may be reserved, even on a non-static fault network, in order to tolerate dynamic faults. 5. Router Design We have designed the Detour-NF router for twodimensional tori so that we could evaluate its hardware cost and operational speed. We first explain the hardware organization of Detour-NF, then compare the cost and speed with a dimension-ordered deterministic router and Duato s minimal fully adaptive router, based on our designs using Verilog-HDL Hardware Organization and Routing Logic Figure 4 shows the block diagram of Detour-NF. It consists of four network ports (North, East, West and South) Virtual Channel(VC) Address Decoder(AD) Output Channel Arbiter(OCA) Figure 4. Hardware Organization. and one processing element interface (PE I/F). All are connected to each other by wire. Instead of a central crossbar switch, separate multiplexers are placed in each port [14, 15]. Each network port consists of four VCs with address decoders (ADs) and an output channel arbiter (OCA). The PE I/F consists of two VCs with the ADs and OCA. One difference between Detour-NF and the pure Duato s minimal adaptive router is that Detour-NF has a connection from the non-minimal VC to the OCA in the west and south ports to support 180-degree turns. Messages generated by the PE are stored in one of the VCs in the PE I/F. The message header, containing its destination address, is decoded by the AD. The independent AD per VC enables parallel header decoding for multiple messages. The AD creates one or two output request signals based on the routing function. When there are two candidates for output ports, these two requests are simultaneously propagated to the OCAs in the selected output ports. The OCA arbitrates several output requests from the VCs which hold the messages, and returns an acknowledge signal to one of them. The acknowledged VC decides the output port based on some selection policy when it receives multiple acknowledgments, and injects the message into the network. Since there is no central crossbar switch in the router, the routing decision and intra-router data transmission are not serialized. We use a dedicated signal that is exchanged between two adjacent routers to detect the faulty channel. The routing functions R and R1 are automatically selected by the ADs, based on the possible paths for messages and faulty information. For fault-free networks, one of the routing modes (the dimension-order or minimal fully

5 Table 1. Synthesis results of three routers. Router Dimension-order Duato s protocol Detour-NF VCs / channel MA clock (MHz) 89 (1.05) 89 (1.05) 85 (1.00) 82 (0.96) Area (cells) 4974 (0.85) 6331 (1.08) 5882 (1.00) 9151 (1.56) Total FFs 4016 (0.99) 5133 (1.26) 4066 (1.00) 5245 (1.29) Numbers in ( ) show the ratios to the values of Duato s minimal fully adaptive router. adaptive mode) can be chosen based on a flag in the message header or by a static router setting. The entire routing function is realized by the hardware logic for fast routing decisions and the fact that no routing table lookup is required [2]. This router adopts an input buffer scheme. A message arriving from the network is stored to a requested VC in the network port. Then, the routing action is repeated until the message reaches its destination Hardware Cost and Speed Table 1 shows the speed and hardware cost of the three routers (dimension-order, Duato s protocol (DP) and Detour-NF). The values in this table were obtained from the synthesis results using the Synopsys FPGA Compiler II. We specified the target device as ilinks VirtexE V600EFG900-8 with a higher priority for speed. The maximum clock frequency reflects the complexity of the circuits. The required chip area, which is represented by the number of cells, and the total flip-flops (FFs) show the hardware cost. For the dimension-order router, we show the results for two patterns (namely, either three or four VCs per physical channel) to compare with the minimum number of VCs for DP and Detour-NF, which have three and four VCs, respectively. All of the routers were designed with 32-bit width physical channels and the buffer capacity per VC is eight 32-bit flits. We also show the ratios relative to the values of the DP router. The dimension-order router can be operated at the fastest clock frequency because of its simple routing logic. Detour- NF is 4% slower than DP in clock frequency because of its logic complexity and its increase of hardware quantities. When we increase the number of VCs, the area and total FFs are increased by the buffer space. The dimension-order router and Detour-NF with four VCs require 26% and 29% more FFs compared with the DP. The increase in area of the dimension-order router is smaller than Detour-NF since the wire area of the dimension-order router is smaller than that of Detour-NF. 1 1 The -Y dimension-ordered router does not require the wire from the north and the south ports to the east and west ports, because turns from the From logic synthesis, we can say that the Detour-NF router is more complex than the other two, but the speed degradation and the increase in hardware are not considerable when we take into account its fault-tolerant ability. 6. Communication Performance 6.1. Simulation Conditions In order to compare the performance characteristics of the adaptive and non-adaptive routing algorithms, we have simulated routers which have four VCs per physical channel. The simulation was executed on a 10 by 10 twodimensional torus network using an HDL simulator. We assume that the clock frequency of the routers is 100 MHz, and that each router takes three clock cycles to hop a message header. We also assume that the cable delay between two routers is less than a single clock cycle. Before showing the simulation results, we summarize the relationship between the routing algorithms and their usage of four VCs per physical channel. Dimension-order 4 non-adaptive VCs Duato s Protocol (DP) 2 non-adaptive (escape) VCs 2 minimal fully adaptive VC Detour-NF 2 non-adaptive VCs 1 minimal fully adaptive VC 1 non-minimal partially adaptive VC For the dimension-order router, all four VCs are used as non-adaptive VCs. On the other hand, DP requires two non-adaptive (escape) VCs and an additional two VCs can be used as minimal fully adaptive VCs. Finally, Detour-NF uses two non-adaptive VCs with one minimal fully adaptive VC and one non-minimal partially adaptive VC. vertical dimension to the horizontal are prohibited.

6 The simulated traffic patterns are random and hot-spot. In the random traffic, each node decides the destination randomly so that messages are spread uniformly on the network. In the hot-spot traffic, one-fourth of all messages are forced to a destination in the middle column of the torus, and the rest of the messages are sent randomly. For each simulation, we ignore the first 2000 messages, and the communication bandwidth is calculated for the following 5000 messages. We evaluated the network bandwidth by varying the message size. We evaluated the average message latency for 64-byte (16-flit) messages by changing the injection rate from the source nodes. For the latency evaluation, we measured the time for messages to reach their destinations after the source nodes created them Faulty models We simulated the Detour-NF router on fault-free and faulty networks. For the fault-free network, the nonminimal VC was not used in order to compare the performance with DP. For the faulty models, we assumed channel or node faults. Figure 5 shows the locations of the faulty channels and nodes. First, we will examine the bandwidth for random traffic with four and eight faulty channels. The four faulty channels are marked as x in the figure. In the case of the eight faulty channels, four additional channels are also set as faulty. Next, we will show the bandwidth for the cases of two and four faulty nodes. The faulty nodes are modeled in such a way that all four channels connected to the faulty node are marked as faulty. The faulty nodes do not take part in the communication. Namely, the faulty nodes never send or receive any messages Results (1) Random traffic Figure 6 shows the network bandwidth for random traffic with and without faulty components. This graph shows that DP achieves the highest bandwidth, although it does not have the fault-tolerance ability. The bandwidth of the Detour-NF router is plotted between the two minimal routing modes, DP and dimension-order. This tendency derives from the routing freedom, which increases according to the number of adaptive VCs. When we increase the number of faulty channels or nodes, the bandwidth of Detour-NF is degraded. Performance degradation by the separated faulty channels is larger than in the case of the faulty nodes. Even in the case of four faulty nodes, Detour-NF has a clear advantage over the dimension-order mode. One reason for these results is that the faulty node is never a message source or destination. This condition eases congestion around the faults compared to the faulty channels. : four faulty channels : additional four faulty channels : two faulty nodes : additional two faulty nodes Figure 5. Locations of the faulty channels and nodes. Figure 7 shows the average message latency for random traffic. The network is saturated at a certain bandwidth and the latency increases. The saturation points show the peak bandwidth of each routing algorithm for a 64-byte message. This graph shows that the saturation points and average latency of Detour-NF are mid-way between DP and dimension-order. The greater the number of faulty nodes that exist, the lower the bandwidth saturation. However, a fewer number of faulty nodes, such as two and four faults, can be tolerated by a single non-minimal VC without degrading communication performance as much. We obtained similar results for another uniform communication pattern, the all-to-all traffic pattern [10]. (2) Hot-spot traffic Figure 8 shows the network bandwidth for hot-spot traffic. We notice that DP and Detour-NF modes achieve a much higher bandwidth than the dimension-order router because of adaptive routing. The bandwidth of Detour-NF is degraded for larger messages because of the fewer number of adaptive VCs. However, the performance degradation caused by the faulty nodes is relatively small when we compare it with the random traffic performance. Figure 9 shows the average message latency for hot-spot traffic. The saturation bandwidth and latency of Detour-NF do not show large differences for the fault-free network and that with two or four faulty nodes. The reason is that the bottleneck of the hot-spot has a larger impact than the few faulty nodes. The Detour-NF router shows saturation bandwidth and latency that is closer to DP than the dimension-order mode due to adaptive routing.

7 22 20 Duato(no faults) Detour-NF(no faults) 10 Bandwidth [GB/s] Detour-NF(2 faulty nodes) Detour-NF(4 faulty nodes) Detour-NF(4 faulty channels) Detour-NF(8 faulty channels) Dimension-order(no faults) Message size [bytes] Bandwidth [GB/s] Duato(No faults) Detour-NF(No faults) Detour-NF(2 faulty nodes) Detour-NF(4 faulty nodes) Dimension-order(No faults) Message size [bytes] Figure 6. Bandwidth for random traffic. Figure 8. Bandwidth for hot-spot traffic. Average Latency [us] Duato(no faults) Detour-NF(no faults) Detour-NF(2 faulty nodes) Detour-NF(4 faulty nodes) Dimension-order(no faults) Average Latency [us] Duato(no faults) Detour-NF(no faults) Detour-NF(2 faulty nodes) Detour-NF(4 faulty nodes) Dimension-order(no faults) Bandwidth[GB/s] Figure 7. Average message latency for random traffic Bandwidth[GB/s] Figure 9. Average message latency for hotspot traffic. From these studies, we can conclude that supporting multiple routing modes such as deterministic, minimal fully adaptive, and non-minimal adaptive routing, has advantages. The Detour-NF router provides fault-tolerance ability and reasonable performance for cases with few faulty components. Since the deterministic and minimal fully adaptive routing modes do not allow any faults, they still are advantageous for fault-free networks. The deterministic mode is good for an environment requiring in-order message delivery, and the minimal fully adaptive routing mode provides good performance. 7. Conclusion We have considered the designs of fault-tolerant adaptive routers and proposed a method for constructing routing algorithms. Our method is simple in the sense that it integrates minimal and non-minimal routing algorithms for independent sets of VCs. We believe our method suits the router design in this era of VLSI technology and large-scale parallel and distributed systems. Based on our method, we showed an example design of Detour-NF. The Detour-NF design can be used not only as a non-minimal adaptive router but also as a minimal adaptive or non-adaptive router without wasting hardware resources. It suits environments where the faults rates are relatively low in both fault-free and faulty networks. The decoding of hardware messages without requiring routing table management is useful for fast operation and it simplifies the treatment of static network faults. However, the faulttolerance ability depends on the number of VCs. To support unconstrained fault regions with a few VCs, a routing algorithm for irregular networks, such as up*/down* routing [12], could be a candidate for the non-minimal routing function R1 in our design method. We are currently considering a flexible algorithm such as up*/down* routing for networks of short mean time between failure. Acknowledgments We would like to thank Prof. Takanobu Baba of

8 Utsunomiya University for his helpful comments. We also wish to thank Osamu Mitobe and Ta Quoq Viet, graduate school students at the University of Electro- Communications, for their help in our experiments. This research is supported in part by the Grants-in-Aid for Scientific Research of the Japan Society for the Promotion of Science (JSPS), No and No The study has been done using CAD tools provided by the VLSI Design and Education Center(VDEC) at the University of Tokyo. [14] T. Yoshinaga, M. Hayashi, M. Horita, Y. Yamaguchi, K. Ootsu, and T. Baba: A Cost and Performance Comparison for Wormhole Routers based on HDL Designs, Proc. IC- PADS 98, pp (1998). [15] T. Yoshinaga, M. Hayashi, M. Horita, S. Nakamura, K. Ootsu, and T. Baba: Recover-x: An Adaptive Router with Limited Escape Channels, Proc. ICPADS 2000, pp (2000). References [1] R.V. Boppana and S. Snyder: A Comparison of Adaptive Wormhole Routing Algorithms, Proc. 20th ISCA, pp (1993). [2] A.A. Chien: A Cost and Speed Model for k-ary n-cube Wormhole Routers, IEEE Trans. Parallel and Distributed System, vol.9, No.2, pp (1998). [3] W.J. Dally and C.L. Seiz: Deadlock-Free Message Routing in Multiprocessor Interconnection Network, IEEE Trans. Computers, vol.c-36, no.5, pp (1987). [4] J. Duato, S. Yalamanchili, and L. Ni: Interconnection Networks, an Engineering Approach, IEEE Computer Society Press, p.515 (1997). [5] J. Duato: A Theory of Fault-Tolerant Routing in Wormhole Networks, IEEE Trans. Parallel and Distributed Systems, vol.8, no.8, pp (1997). [6] J. Duato: A Necessary and Sufficient Condition for Deadlock-Free Adaptive Routing in Wormhole Networks, IEEE Trans. Parallel and Distributed Systems, vol.6, no.10, pp (1995). [7] M.L. Fulgham and L. Snyder: Triplex Router: A Versatile Torus Routing Algorithm, Technical Report UW-CSE , University of Washington (1996). [8] C.J. Glass and L.M. Ni: The Turn Model for Adaptive Routing, Proc. 19th ISCA, pp (1992). [9] D.H. Linder and J.C.Harden: An Adaptive and Fault Tolerant Wormhole Routing Strategy for -ary -cubes, IEEE Trans. on Computers, vol.40, no.1, pp.2 12 (1991). [10] H. Hosogoshi, O. Mitobe, T. Yoshinaga, and M. Sowa: Design of a Fault-Tolerant Fully Adaptive Router, Proc. Symposium on Advanced Computing Systems and Infrastructures, pp (2003, in Japanese). [11] D.R. Millar and W.A. Najjar: Preliminary Evaluation of a Hybrid Deterministic/Adaptive Router, Proc. Parallel Computing, Routing and Communication Workshop, Lecture Notes in Computer Science, vol.1417, pp (1997). [12] M.D. Schroeder, A.D. Birrel, M. Burrows, H. Murray, R.M. Needham, T.L. Rodeheffer, E.H. Satterthwaite, and C.P. Thacker: Autonet: A High-Speed, Self-Configurable Local Area Network using Point-to-Point links, IEEE J. Selected Areas Commun., vol.9, no.8, pp (1991). [13] Y.-J. Suh, et al.: Software Based Fault-Tolerant Oblivious Routing in Pipelined Network, Proc. ICPP, vol.1, pp (1995).