A Real-Time Communication Method for Wormhole Switching Networks

Transcription

1 A Real-Time Communication Method for Wormhole Switching Networks Byungjae Kim Access Network Research Laboratory Korea Telecom 62-1, Whaam-dong, Yusung-gu Taejeon, Korea Jong Kim, Sungje Hong, and Sunggu Lee y Dept. of Computer Science and Engineering y Dept. of Electronic and Electrical Engineering Pohang University of Science and Technology San 31 Hyoja Dong, Pohang , KOREA jkim@postech.ac.kr Abstract In this paper, we propose a real-time communication scheme that can be used in general point-to-point real-time multicomputer systems with wormhole switching. Real-time communication should satisfy the two requirements of predictability and priority handling. Since traditional wormhole switching does not support priority handling, which is essential in real-time computing, flit-level preemption is adopted in our wormhole switching. Also, we develop an algorithm to determine the message transmission delay upper bound to predict worst-case message delay. Simulation results show that the delay upper bounds calculated using the proposed algorithm are very close to actual average message transmission delays for messages with high priorities. 1. Introduction Real-time applications have two very stringent requirements, known as correct result and timing. Real-time tasks should honor the timing requirements of real-time applications to make the result effective. To determine whether real-time tasks satisfy their timing requirements, it is necessary to know the execution time of each real-time task. The execution time analysis of real-time tasks have been studied for a long time. One important factor which makes the analysis difficult is communication. Since communication is in general not bounded in time, it is hard to predict maximum message transmission delays and to guarantee that messages are delivered within their message transmission delay limit (or deadline) if it is given. Real-time communication enables us to either predict the maximum message transmission delay for a given message or to guarantee the This research was supported in part by KOSEF under Grant delivery of messages within their delivery timing requirements. In multiprocessor systems, several cooperating tasks running on different processing nodes have to communicate with each other, and if these tasks have timing constraints such as deadlines, unpredictable delay of message transmission can adversely affect the execution of the tasks dependent on the messages, possibly resulting in missed deadlines. Thus real-time multiprocessor systems must support real-time communication, which means the timely delivery of inter-task messages. A popular communication switching method used in multiprocessor systems is wormhole switching. In wormhole switching, it is very hard to guarantee the timely delivery of messages, since wormhole switching is based on a hold-and-wait technique. Therefore, real-time communication in wormhole networks has received growing attention in recent years. Real-time communication in multi-hop networks has been studied widely. Ferrari and Verma proposed the realtime channel concept, which is an enhancement of a traditional packet-switched network to provide delay bound guarantees to real-time message traffic [3, 4]. A real-time channel is a unidirectional virtual connection that traverses one or more network links. The necessary network bandwidth and other resources are reserved using real-time channel establishment procedure [1]. When real-time channels are to be established, the schedulability problem (given a set of real-time channels, can all packets in these channels be delivered before their requested delay bound[4]?) is essential. Kandlur et al. established the sufficient conditions for the schedulability problem [1] and Zheng and Shin derived the necessary and sufficient conditions for the same problem [4]. Research results on a fault tolerant version of the real-time channel [2] and a router architecture designed to support real-time channel have been reported recently [7]. Real-time communication in wormhole switching networks was addressed by Li [8], Mutka [9], and Song [1] recently. In Li s approach, there are as many virtual chan-

2 nels as priority levels and each virtual channel is assigned a unique priority level. A packet can request virtual channels which are numbered lower than or equal to its priority. By doing this, the probability that a higher priority packet can obtain a virtual channel is higher than that of lower priority packets [8]. Song proposed a new flow control method for real-time communications in wormhole networks. With a special router architecture, flit-level preemption of physical channels among prioritized packets can be implemented using a smaller number of virtual channels [1]. While Song s and Li s methods are concerned only with priority handling in wormhole switching networks, Mutka has addressed how to guarantee deadlines of packets in wormhole networks. By using previous research results on rate monotonic scheduling technology, he studied how to check the schedulability of a given set of real-time periodic traffic flows [9]. However, because of the blocking characteristic of wormhole networks, mere application of the rate monotonic algorithm to real-time message traffic is not appropriate. In this paper, we propose a communication structure and algorithm that provides message transmission delay upper bounds (U) in wormhole switching networks. For a given set of real-time communication requests, if all of their U values are less than or equal to the corresponding deadlines, the requests can be met. Traditional wormhole switching networks do not have a priority handling method, which is essential in order to make the communication have a bounded delay. Hence, we assume a flit-level preemptive wormhole switching method. Additionally, we show the relationship between the number of real-time communication requests and the number of possible priority levels using simulation experiments. The remainder of this paper is organized as follows. In Section 2, we define the problem and describe the system model considered in this paper. Section 3 treats priority handling in wormhole switching networks. Section 4 gives the proposed algorithm and an example. Section 5 provides simulation results. Finally, in Section 6, we summarize and conclude this paper. 2. System model and problem definition A general real-time multiprocessor system considered in this paper is shown in Figure 1. As shown in Figure 1, the system consists of a special processor, called the host processor, and several nodes for job execution. The host processor is in charge of overall system management such as job scheduling, node allocation, and schedulability testing of real-time jobs. Also, the host downloads jobs to allocated nodes. Each node in Figure 1 contains its own processor and local memory. Each real-time job is downloaded into a group of allocated nodes and these nodes communicate with Host processor Node Node 1 Node 2 Node 3 Node N-1 Node N The Interconnection Network Figure 1. A general multiprocessor system each other by sending messages through an interconnection network. The interconnection network connects the nodes in a topology, such as a hypercube or a mesh. We assume most message traffic which require timely delivery are periodic as in [9]. A real-time application consists of several cooperating jobs, and each job is executed on a different processing node. Real-time message traffic flows are required between such jobs. We assume that the minimum communication frequency between jobs is known. The host processor calculates the upper bound of message transmission delay for all messages and compares it with the required delay limit. Prior to the formal description of the problem discussed in this paper, we present a few definitions. Network latency [5] : The time taken to deliver a message when no other traffic is present. Message stream (M i ) : The continuous message traffic between a source and destination node pair. Each message stream is characterized by minimum message inter-generation time (T i ), maximum message length (C i ), deadline (D i ), and priority (P i ). L i : The maximum network latency of messages belonging to M i U i : The transmission delay upper bound of messages belonging to M i One message stream denotes a message traffic flow between a source and destination node pair. Each message stream M i has a priority value P i representing the importance of the message stream. Every message that belongs to the message stream M i inherits the corresponding P i value. D i is the requested delay limit and U i is the calculated delay upper bound. In other words, D i is the deadline within which the message should be delivered, and U i is the guaranteed delivery delay. The following is the formal definition of the problem. Instance : A multiprocessor system such as shown in Figure 1 with a wormhole switching interconnection network

3 and a set of message streams M = fm 1 ; M 2 ; M 3 ; :::; M n g. A message stream M i is characterized by a seven-tuple (S id ; R id ; P i ; T i ; C i ; D i ; L i ), where S id and R id are the source node and the destination node, respectively. The routing path of each message stream is statically determined by using a deterministic routing algorithm such as X-Y routing for meshes. Problem : Determine whether all message streams in M satisfy the condition that U i for a given M i is less than or equal to its corresponding D i (U i D i for all M i ). To be able to solve the above problem, we should have an algorithm that finds the delay upper bound of all message streams when all real-time message streams are defined for a given job. We will call the above problem message stream feasibility testing. The calculation of U i is the kernel of this problem. The mapping of real-time jobs to processing nodes should be preceded for this problem to be meaningful. The jobs which communicate each other frequently could be mapped to relatively nearby processing nodes. But job allocation is another problem and we do not consider in this paper. Predictability and priority handling are two cardinal issues in real-time computing. Determining the delay upper bound U for all M i s is the predictability issue. Without having a priority handling scheme, no difference in delay upper bound will be shown between higher priority messages and lower priority messages. Higher priority messages should be delivered as fast as possible at the cost of lower priority messages and the delay upper bound of higher priority messages should be closer to the actual message transmission delay than that of lower priority messages. In a classical wormhole switching network, there is no priority handling method. In the following section, we introduce a priority handling method for wormhole switching networks. 3. Priority handling in wormhole switching networks Early multi-hop networks used store-and-forward switching. In this approach, the entire packet is stored in a packet buffer when a packet reaches an intermediate node. On the other hand, cut-through switching is a switching technique which does not buffer packets in an intermediate node if the next output channel is available. It just forwards the received data to a selected neighboring node without buffering. Wormhole switching is a special case of cut-through switching. A packet (or message) is divided into a number of flits. The header flit governs the route. As the header advances along the specified route, the remaining flits follow in a pipeline fashion. If the header flit encounters a channel that is already being message n priority 3 message B is permanently blocked message 2 priority 3 message B priority 4 message 1 priority 3 message A priority 2 incoming channels SWITCH blocking occurred outgoing channels Figure 2. An example of priority inversion used, it is blocked until the channel becomes available [5]. This blocking is a major difference with other cut-through switching variations, such as circuit switching and virtual cut-through. Because of the blocking and waiting for channels, deadlock situation may occur in wormhole switching. Deadlock can be avoided by some deterministic path selection schemes, such as X-Y routing for meshes. Hence, we assume deadlock situation never occur. Priority inversion can occur in handling real-time communication messages in wormhole switching. Priority inversion [6] is a situation where low-priority messages block higher priority messages. Priority inversion occurs when the physical channel in the switch is arbitrated by priorities of messages and non-preemptable. An example of priority inversion is shown in Figure 2. In Figure 2, message B, which has the highest priority, is permanently blocked. We assume flit-level preemptive wormhole switching. If higher priority messages can preempt the blocking lower priority messages, the priority inversion problem can be easily resolved. This is an ideal priority handling method. However, flit-level preemption is not easy to implement in classical wormhole switching networks because only the header flit has the routing information. Preemption may cause messages to become lost in the network. As previously mentioned, Song proposed a real-time traffic flow control method based on flit-level preemption [1]. However, the proposed method requires special router development. Hence, we use a flit-level preemption method implemented by using virtual channels. Several virtual channels can exist in one physical channel. We assume there are as many virtual channels as priority levels. Each virtual channel is assigned a different priority level. The arbitration for the physical channel is based on priorities. A virtual channel V i associated with priority level i can obtain physical channel bandwidth if all virtual channels associated with priorities higher than i are free. A message with priority i can only request the virtual channel associated with priority value i. This method works identically with Song s scheme from the viewpoint of real-time message arrival [1]. Although more virtual channels guarantee a wider range of priorities, it is difficult to have too many virtual channels due to practical resource constraints. Hence, we simulate a multi-hop network with the assumed flit-level preemption

4 method to find the effect of the number of virtual channels (or the number of different priorities) on the U value. 4. Delay upper bound calculation algorithm In this section, we first present a delay upper bound calculation algorithm and then give a simple example of the algorithm Overview of the proposed algorithm Messages in a preemptive and prioritized network are blocked only when higher priority messages use a part of the path to be used by the blocked messages. To determine the maximum delay, known as the delay upper bound, of the blocked messages, information on higher priority messages that may affect the blocked messages is required. So the first step is determining which message streams affect the given message stream. Next, we calculate the delay upper bound by considering the message generation interval, maximum message size, and blocking relationships between message streams. Let us describe how to determine which message streams affect the given message stream. The host processor has all the information about message streams. For each message stream from higher priority to lower priority, the set of all affecting higher priority messages, denoted as the HP set, is constructed. The HP set of the given message stream is constructed by combining the HP sets of higher priority message streams that pass through a part of the path to be used by the given message stream. An example is shown in Figure 3. In Figure 3, the message D which has the highest priority cannot be blocked or preempted by any other message. Thus its HP set is empty. In the case of messages B and C, both messages can be blocked or preempted by the message D and they are mutually influential. So the HP set of the message B consists of messages D and C, and the HP set of the message C consists of messages D and B. All elements in these sets are marked as direct. In the case of the message A, it can be blocked or preempted by messages B or C. But the message D can block the message A indirectly through messages B or C. So the HP set of message A consists of messages B and C which are marked as direct and the message D which is marked as indirect. For each indirect element, the list of intermediate message streams are maintained. Messages B and C are intermediate message streams of message D. Using dependencies between message streams we can draw a blocking dependency graph for each message stream. We define the following terms. Direct blocking : the case that paths of two message streams are overlapping Indirect blocking : the case that paths of two message streams do not overlap but there is(are) intervening message stream(s) between given two message streams Blocking chain : the list of intervening message streams when an indirect blocking occurs More than one blocking chain can be defined for an indirect blocking. For example, the indirect blocking between message A and message D in Figure 3 defines two blocking chains. Message D can indirectly block message A through message B or message C. So the two blocking chains are (M B ) and (M C ). Using HP sets and derived blocking dependency graphs, we can calculate how frequently direct or indirect blockings occur. Then we can determine the worst case transmission delay (or delay upper bound). Every message has network latency. Network latency is defined as the time to complete transmission when no other traffic is present. If there is only one message, the delay upper bound of that messages equals the network latency. But we must consider the case where there are more than one message that can be mutually influential. The basic idea in calculating the delay upper bound for a message stream is shown in Figure 4. In Figure 4, there are four message streams. Message streams are listed in decreasing order of their priorities, i.e., (M 1, M 2, M 3, M 4 ). The shaded area shows the time slot occupied by a message and dotted area shows the time slot preempted by higher priority messages. For messages M 1, M 2, and M 3, message generation interval T and message length C are given. Let us assume that the HP set of M 4 is (M 1, M 2, M 3 ) and all elements are marked as direct. As time advances, there are free time slots that are not used by message streams in the HP set of M 4. These time slots can be used by M 4. The delay upper bound of M 4 is the time when the summation of all these free time slots equals the network latency. For example, if the network latency of M 4 is 6, then time 26 is the delay upper bound of M 4. If M 1 and M 2 in HP set of M 4 are marked as indirect and their intermediate message streams are M 2 and M 3 respectively, then their blocking dependency graph is like Figure 5. Figure 6 presents a timing diagram for delay upper bound calculation. As shown in Figure 6, the second and the third instance of M 1 in Figure 4 are removed since M 2, which is the intermediate message stream of M 1, does not exist in that time period. Thus the delay upper bound of M 4 is reduced to time Data structure The proposed algorithm constructs the set HP i for each message stream M i. It also creates GList[], which is a set

5 message A from 6 to 9 priority 1 period T = message B from 7 to 19 priority 2 period T = 4 message D from 14 to 24 priority 3, period T= 5 message C from 8 to 19 priority 2, period T = 45 Figure 3. HP set construction example M 1 T = 1, C = 2 M 2 T = 15, C = 3 M 3 T = 13, C = 4 M Figure 4. U calculation for an direct blocking TIME of message streams that have a same priority. HP i is a set of the structure variable that has following fields. M id field : stores message stream id Mode field : can have one of the values (DIRECT, INDIRECT) IN field : stores the set of message stream id The M id field stores the message stream id which affects the given message stream, the M ode field indicates whether M id affects it directly or indirectly, and the IN field keeps the list of intermediate message stream(s) when the M ode field is set to INDIRECT. After generating HP sets, the proposed algorithm creates the blocking dependency graph(bdg) in the form of adjacency matrix and the timing diagram in the form of twodimensional array structure for each message stream. The column index of timing diagram corresponds to time and the row index corresponds to message streams. Cells of timing diagram can have one of the values (FREE, BUSY, WAIT- ING, ALLOCATED). FREE means that time slot is usable. BUSY indicates that a higher priority message stream is using that time slot, so this time slot is not usable. WAITING means preempted state and ALLOCATED means that time slot is allocated to a message stream specified by the row index Algorithm M4 M3 M2 M1 Figure 5. The blocking dependency graph for an indirect blocking The input to the algorithm is the instance of message stream feasibility testing and the output is either success or f ail, which indicates whether the given real-time communication requests can be satisfied or not. The proposed algorithm is as follows. Determine-Feasibility(M) M 1 T = 1, C = M 2 T = 15, C = M 3 T = 13, C = M4 TIME 22 Figure 6. U calculation for an indirect blocking 1 plevel = the number of priority level; 2 N = the number of message streams; 3 FOR i = TO N? 1 4 GList[P i ] ( M i ; / add a element to a set / 5 FOR i = plevel? 1 DOWNTO 6 FOR each element M j in GList[i] 7 Generate HP(i, j); 8 FOR each element M j in GList[i] 9 U j = Cal U(j); 1 IF U i < for any i 11 RETURN fail; 12 ELSE RETURN success;

6 Global variables and GList[] are initialized from line 1 to line 4. Then HP sets are constructed. Because the host processor has all traffic information, generating HP set for each message stream is not difficult. So, we omit the detail of Generate HP() function code. Delay upper bound for each message stream is calculated by Cal U() function and the determination of the feasibility result follows. Cal U(j) 1 HP j -= an element that has M j as an M id field; 2 required = L j ; dtime = D j ; gtime = ; 3 ne = the number of elements in HP j ; 4 T diagram = allocate [..ne][1..dtime]; 5 Generate Init Diagram(T diagram,j,dtime); 6 IF there is an INDIRECT element in HP j 7 Generate BDG G of HP j ; 8 Modify Diagram(T diagram,j,g,ne); 9 FOR i = 1 TO dtime 1 IF T diagram[ne][i] == FREE 11 IF ++gtime == required RETURN i; 12 RETURN -1; Generate Init Diagram(T d,j,dtime) 1 mi = ; 2 Sort HP j in non-increasing order of priority 3 FOR each K in HP j 4 FOR i = TO ddtime=t K:Mid e 5 alloctime = ; 6 FOR l = TO T K:Mid 7 IF T d[mi][i T K:Mid + l]==free 8 alloctime++; 9 T d[mi][i T K:Mid + l]=allocated; 1 T d[mi + 1..ne][i T K:Mid + l]=busy; 11 ELSE IF T d[mi][i T K:Mid + l]==busy 12 T d[mi][i T K:Mid + l]=waiting; 13 IF alloctime == C K:Mid BREAK; 14 mi++; Modify Diagram(T d,j,g,ne) 1 G = G T ; /transpose/ 2 s = pre = the node M j in G; 3 Allocate vc[ne] and initialize to ; 4 DO 5 cur=bfs(pre,s,g); /breadth first search/ 6 ni = the order of M cur in sorted HP j ; 7 IF M cur INDIRECTly affects 8 IF ++vc[ni]==(indegree of M cur in G) 9 FOR i = 1 TO dtime 1 IF T d[ni][i]==allocated or WAITING 11 AND all T d[r][i]==free or BUSY 12 (r is the indexes of intermediate M. of M cur ) 13 T d[ni][i]=free 14 Update T d consistently; 15 MARK M cur ; 16 ELSE MARK M cur ; /DIRECT/ 17 pre = cur; 18 UNTIL all nodes in G MARKed; The function Generate Init Diagram() creates the timing diagram of a given HP set assuming all elements are direct. When one or more indirect elements exist, after generating blocking dependency graph, the proposed algorithm calls Modify Diagram(). A time slot used by an indirect element can be freed if all of the intermediate message streams do not request that time slot. A released time slot can be reused by other message streams. Free slots at the last row of timing diagram can be used by the given message stream and the time at which the summation of these time slots is equal to the network latency of the message stream is the delay upper bound of that message stream An example of the algorithm We show a simple example of the proposed algorithm in this subsection. We assume that nodes are interconnected in a two-dimensional mesh topology and deadlock is avoided by using X-Y routing. The given set of message streams are as follows. Each message stream is presented in the form M i = (S id ; R id ; P i ; T i ; C i ; D i ; L i ) M = fm ; M 1 ; M 2 ; M 3 ; M 4 g, where M = ((7; 3); (7; 7); 5; 15; 4; 15; 7) M 1 = ((1; 1); (5; 4); 4; 1; 2; 1; 8) M 2 = ((2; 1); (7; 5); 3; 4; 4; 4; 12) M 3 = ((4; 1); (8; 5); 2; 45; 9; 45; 16) M 4 = ((6; 1); (9; 3); 1; 5; 6; 5; 1) After calling Generate HP(), HP i s for all M i s are constructed as follows. HP = f(; DIRECT; )g HP 1 = f(1; DIRECT; )g HP 2 = f(; DIRECT; ); (1; DIRECT; ); (2; DIRECT; )g HP 3 = f(1; DIRECT; ); (3; DIRECT; )g HP 4 = f(; INDIRECT; (2)); (1; INDIRECT; (2; 3)); (2; DIRECT; ); (3; DIRECT; ); (4; DIRECT; )g During execution of Cal DUB(), timing diagrams of all HP sets are constructed and U values are determined. Among the HP sets, only HP 4 has indirect elements.

7 Figure 7 shows the initial timing diagram of HP 4 generated M M1 M2 M3 result : ALLOCATED : WAITING : BUSY : FREE Figure 7. The initial timing diagram of HP 4 assuming all elements are direct. There are 7 free time slots at the last row. Because the network latency of M 4 is 1, deadline can not be guaranteed. Using blocking M2 M Table 1. 1 priority level, 2 message streams P : tual latency changing the number of priority levels. In this simulation study, we assume the following environment. PNs are interconnected in a 1 1 two dimensional mesh and X-Y routing is used. Each PN is a source of at most one message stream and the corresponding destination node is selected using a spatial uniform distribution. The total simulation time is 3 flit time, omitting 2 start-up time units. One flit time unit is the time for a flit to be transmitted to a selected neighbor node. M4 M3 M1 is uniformly dis- The maximum message size C i tributed between 1 and 4. Figure 8. The blocking dependency graph of HP 4 dependency graph of HP 4 as shown in Figure 8, the algorithm modifies the initial timing diagram considering indirect elements. The final timing diagram of HP 4 is shown in Figure 9. As Figure 9 shows, the second and the third instance of M and the fourth instance of M 1 are removed. Because of the released time slots, the first instance of M 3 is compacted and there exist enough free time slots to guarantee the deadline of M 4. The U i s for all M i s are determined as follows. U = 7, U 1 = 8, U 2 = 26, U 3 = 2, U 4 = 33 Since all U i s are smaller than their corresponding D i s, it returns success. M M1 M2 M3 result delay upper bound of M : BUSY : ALLOCATED : WAITING : FREE Figure 9. The final timing diagram of HP 4 5. Simulation In this section, we show the ratio between the delay upper bound found using the proposed algorithm and the ac- All message streams are periodic. Minimum message inter-generation time T i is uniformly distributed between 4 and 9. If the calculated U i is larger then T i, we increased T i to accommodate all generated traffics. We performed simulation by changing the number of priority levels and total number of message streams. Each message stream has a priority value P i with probability 1 the number of prioriy levels. Table 1 shows the result when only one priority level is allowed and the total number of message streams is 2. The ratio between the calculated delay upper bound and the actual latency is less than.5. If more message streams are generated, the ratio is extremely exacerbated as shown in Table 2. The results of simulation when more than one priority level is allowed are presented in Tables 3, 4, and 5. As we can see from the tables, the more priority levels are allowed, the better the result we can get. From simulation results including not presented here, we found that at least 1 jmj priority levels are needed to have the ratio of the 4 highest priority level be higher than.9. jm j represents the total number of message streams. In addition, when more priority levels are allowed, the ratio value of the lowest priority one also increases as shown in Tables 1, 3, and Conclusion In this paper, we presented a real-time communication method that can be used in general point-to-point realtime multicomputer systems with wormhole switching. We

8 Table 2. 1 priority level, 6 message streams P : Table 3. 4 priority levels, 2 message streams P : P : P : P : Table 4. 5 priority levels, 2 message streams P : P : P : P : P : addressed two important issues in real-time communication: predictability and priority handling. For priority handling, we assume a flit-level preemptive wormhole switching method. We also provided an algorithm to compute message transmission delay upper bounds assuming that this priority handling method is used. Using simulation, we show that reasonable number of priority levels are required to obtain meaningful delay upper bounds. References [1] D. D. Kandlur, K. G. Shin, and D. Ferrari, Realtime communication in multi-hop networks, Proc. 11- th Int l. Conf. on Distributed Computing Systems, 1991, pp [2] Qin Zheng and Kang G. Shin, Fault-Tolerant Real- Time Communication in Distributed Computing Systems, 22nd International Symposium on Fault Tolerant Computing, Boston, MA, Jul., [3] D. Ferrari and D. C. Verma, A Scheme for Real-Time Channel Establishment in Wide-Area Networks, IEEE J. Selected Areas Communication, 199, pp [4] Qin Zheng, and Kang G. Shin, On the Ability of Establishing Real-Time Channels in Point-to-Point Packet- Switched Networks, IEEE Tran. on Communications, Vol. 42, No. 2/3/4, February/March/April [5] Lionel M. Ni and Philip K. McKinley, A Survey of Wormhole Routing Techniques in Direct Networks, IEEE Computer, February, 1993, pp Table priority levels, 6 message streams P : P : P : P : P : P : P : P : P : P : P : P : P : P : P : [6] K. Toda, K. Nishida, S. Sakai, and T. Shimada, A Priority Forwarding Scheme for Real-Time Multistage Interconnection Networks, Proc. of Real-Time Systems Symp., pp , December [7] Jennifer Rexford, John Hall, and Kang G. Shin, A Router Architecture for Real-Time Point-to-Point Networks, Int l. Symp. on Computer Architecture, [8] J. P. Li and M. W. Mutka, Priority Based Real-Time Communication for Large Scale Wormhole Networks, Proc. of the IPPS, [9] M. W. Mutka, Using Rate Monotonic Scheduling Technology for Real-Time Communication in a Wormhole Network, technical report, Department of Computer Science, Michigan State University, 1993 [1] H. Song, B. Kwon, and H. Yoon, Throttle and Preempt: A New Flow Control for Real-Time Communications in Wormhole Networks, Proc. of the ICPP, 1997