Fault-Tolerant Real-Time Communication in FDDI-Based Networks. Biao Chen, Sanjay Kamat and Wei Zhao. Texas A&M University

Transcription

1 Fault-Tolerant Real-Time Communication in FDDI-Based Networks Biao Chen, Sanjay Kamat and Wei Zhao Department of Computer Science Texas A&M University College Station, Texas Abstract FDDI-Based Recongurable Networks [] have an architecture that is suitable for delivering messages that have hard real-time constraints as well as certain fault-tolerance requirements. This architecture uses multiple FDDI networks to connect hosts and provides for automatic reconguration to maintain high network bandwidth in spite of faults. An important open problem is how resources in such networks should be managed in order to guarantee that the fault-tolerant real-time requirements of messages are met. This paper presents an ecient and practical solution to this problem. Our solution consists of o-line and on-line components. On-line management deals with run-time manipulation of messages and network resources. A message grouping approach simplies on-line management. O-line management deals with message grouping, bandwidth allocation and schedulability verication. Three approaches are investigated: spatial redundancy, temporal redundancy and an integrated approach. It is shown that the integrated approach has the best performance. Our solution is compatible with the FDDI and SAFENET standards. Introduction High-speed networks are being increasingly deployed in mission-critical systems. These networks have stringent timing and fault-tolerance requirements. The dual requirements of providing real-time communication and fault-tolerance means that such networks must guarantee the delivery of critical messages on time even in some faulty situations. Designing such networks is a challenging task. In this paper, we report our work aimed at providing fault-tolerant real-time communication using an FDDI-based network. A high bandwidth of 00 Mbps, support for realtime communication, and built-in fault-tolerance features make FDDI a good candidate for mission-critical applications. FDDI uses a timed token medium access control protocol that guarantees an upper bound on the medium access time of the stations. These features of FDDI MAC protocol enable transmission of real-time messages before their deadlines under normal, fault-free operational conditions. Because of its built-in provisions for fault-tolerance and real-time communications, FDDI has been adopted as the underlying backbone network by the SAFENET (Survivable Adaptable Fiber Optic Embedded Network) standards committee. It must be emphasized here that the built-in features of FDDI are not enough to meet the stringent requirements of the SAFENET standard. The SAFENET standard requires that the network be able to survive multiple faults. However, an FDDI network may be partitioned by just 2 trunk faults. Further, it is necessary to ensure that certain critical messages are still transmitted by their deadlines even during the periods of fault detection and recovery. Our objective is to design an FDDI-based network that can meet the requirements of systems such as the SAFENET. We extend the previous work by researchers in the area of fault-tolerance and real-time aspects of FDDI networks. FDDI-based network architecture designs for providing enhanced fault-tolerance are proposed in [, 2]. In [2], an implementation of network architecture that uses two FDDI networks to connect nodes is discussed. In [], this idea is generalized to an architecture called FBRN (FDDI-Based Recongurable Network) that consists of r FDDI rings and an implementation for r = is proposed. An FBRN achieves a high degree of fault-tolerance by ) using multiple FDDI ring networks to connect the hosts and 2) using ecient fault detection and network reconguration algorithms. Ecient reconguration algorithms for FBRNs are studied in [3, ]. Since an FBRN consists of FDDI rings as its building blocks, we can use established results on bandwidth allocation in FDDI for real-time messages. These results may be found in [5, 6]. We assume some familiarity with these results, since they provide a foundation for our work. An important problem that remained to be solved was how to manage network resources in order to provide fault-tolerant real-time communication with the FBRN architecture. This paper presents an ecient and practical solution to this problem. In this paper, we seek to satisfy the combined fault-tolerant realtime requirements of messages in an FBRN. In order to achieve this objective, we propose an integrated fault-tolerant real-time management mechanism for the FBRNs. This mechanism consists of on-line and o-line components. On-line management deals with system run-time activities such as network initialization, fault detection, network reconguration, message migration from faulty rings to non-faulty ones, etc. It is important to minimize the overheads in on-line management tasks.

2 O-line management plays a signicant role in achieving our objective. O-line management partitions messages into groups. Each group is intended for assignment to separate FDDI ring at run time. Further, it allocates bandwidth to messages within each group. Analytical tests are used to verify that this grouping and bandwidth allocation can satisfy the fault-tolerant real-time requirements. This approach reduces the overheads of many on-line management tasks since they deal with message groups rather than individual messages. One of our primary focuses in this paper is on designing ecient o-line management algorithms that lead to a high system performance. This paper is organized as follows. In Section 2 we present a fault-tolerant real-time model appropriate for our system including message model and relevant features of FBRN architecture. Section 3 describes our group-based approach that supports fault-tolerant real-time communication management for FBRN. Section and 5 deal with o-line and on-line management issues in detail. Section 6 presents the performance results and observations. We conclude in Section 7 with a summary of our contributions. 2 Fault-Tolerant Real Time System Model 2. Message Model Traditionally, the messages with hard real-time deadlines are considered synchronous or periodic. Such a message stream is characterized by the following three parameters: the message length (C), the message period (P ), and the message deadline (D) relative to message arrival. In other words, messages from such a stream are assumed to arrive at a network node at times t, t + P, t + 2P, : : :, etc., with each message requiring a total of C units of time for its transmission. Further, the message that arrives at time t must be sent out by time t + D. Due to the modal nature of system operation, the traditional real-time communication requirement for hard real-time systems is the following: given a set of message streams, the transmission of every message from each message stream must be completed by its deadline. This problem denition presumes a fault-free operation of the system. The consideration of faults requires us to re-examine the specication of message sets. Often, some of the messages are more critical than others, in the sense that they must be successfully transmitted by their deadlines in spite of some faults that may be detected at run time. Thus, the message specication must be augmented with an additional parameter that species the desired level of fault-tolerance for the message. We consider a message source to be characterized by the four parameters: C, P, D, and F, where C and P are message length and period as dened earlier. D and F together specify the fault-tolerant real-time communication requirement of this message as follows: If a message arrives at time t, the network must complete the transmission of this message by time t+d as long as the the number 2 3 Node Node 2 Node 3 Node Node Bunch Bunch Bunch 2 Bunch 3 Bunch An FBRN with n=, r=. Figure : FBRN Architecture: the Default Conguration of ring faults during the interval [t; t + D] is less than F (F ). F includes the number of faulty rings that already exist at t and those that are detected during the interval. Formally, we dene f(t) to be the number of faulty rings at time t and h(t; t + D) to be the number of ring faults detected during (t; t + D]. Then the faulttolerant real-time requirement can be stated as: If a message of stream M k arrives at time t and f(t) + h(t; t + D k ) < F k then the deadline of this message should be guaranteed. Let M = fm ; M 2 ; : : : ; M i ; : : : ; M n g denote a set of n hard real-time message streams, where stream M i is characterized by (C i ; P i ; D i ; F i ). Our problem is to guarantee that the fault-tolerant real-time requirements of all the message streams in M are met. In the rest of the paper, we discuss how we solve this problem in the context of FBRNs. 2.2 FBRN Architecture In this section, we discuss the features of FBRN architecture that are relevant to our problem. We provide only a brief sketch of FBRN architecture and the reader is referred to [] for details. An FBRN consists of r FDDI trunk rings, each trunk ring being composed of two counter-rotating ber loops. Only one of these two loops is used for transmission and the other is reserved as back up. Figure shows an FBRN with FDDI trunk rings and nodes. For conciseness, the network is shown in a at form. That is, node is repeated on both ends of the gure to indicate the ring topology. As shown in the gure, each node has r left ports and r right ports. The ports on either side are numbered ; 2; : : :; r starting from the top. In a fault-free situation, each left port i is connected to the corresponding right port i in order to form r FDDI trunk rings. Each node can transmit and receive messages on any of these r rings. The section of an FDDI trunk ring between two neighboring nodes is called a trunk link. Since an FDDI trunk ring consists of two loops, a trunk link is made of two links. Figure shows only the default conguration of an FBRN. The FBRN architecture possesses a certain reconguration capability. First, each FDDI trunk ring can wrap up the dual loops to isolate a faulty trunk link. Figure 2(a) shows the same FBRN as was shown in Figure under a fault situation. As shown

3 2 3 Node Node 2 Node 3 Node Node Group : M, M 2 Ring Group 2: M 3, M Ring 2 Bunch Bunch Bunch 2 Bunch 3 Bunch (a): Reconfiguration using FDDI s wrap up mechanism Ring 3 cp cp 2 cp 3 cp 2 3 Bunch Node Node 2 Node 3 Node Bunch Bunch 2 Bunch 3 Bunch (b): Enhanced reconfiguration by connecting fault-free segments. Node Figure 2: FBRN Architecture: Modied Congurations in Figure 2(a), the wrap up capability leads to recovery of one FDDI trunk ring that has single trunk fault. The other faulty rings have multiple faults and cannot be recovered by FDDI's built-in wrap up mechanism. However, FBRN nodes possess additional reconguration capability. Each node can reconnect any of its left ports to any of its right ports as long as certain validity constraints are met. As shown in Figure 2(b), this reconguration capability leads to recovery of an additional FDDI trunk ring. 3 Our Strategy The objective of fault-tolerant real-time management in FBRN is to manipulate the network resources and messages so that the communication requirements of all of the messages are met. We divide the functionalities of this management into two parts, viz., o-line mangement and on-line management. O-line management involves message assignments, bandwidth allocation and verication that the fault-tolerant realtime requirements can be met. On-line management is responsible for detecting faults, reconguring the network in the event of faults, migrating messages from faulty rings to non-faulty ones if necessary, and so on. For our solution to be practical, we have to minimize the overheads involved in on-line management. Imagine a set of messages already assigned to dierent rings. At run-time some ring may be detected as faulty. It is the task of on-line management to decide what needs to be done about the messages that were being transmitted on that ring. One approach is to dynamically revise all the message-to-ring assignments whenever a ring fault is detected. With this approach, one may be able to fully utilize the network resources while attempting to meet the message requirements. Clearly, this method involves a large runtime overhead and is not practical for mission-critical applications. We adopt a group-based management approach which deals with message groups rather than individual messages. This approach greatly reduces overheads associated with on-line management. In groupbased approach, messages are grouped together based on certain criteria. All messages belonging to a group are assigned to a single ring. For message group G and ring A, we use G(A) to denote the group that cur- F = 3, F 2 = 2, F 3 =, F = Figure 3: Temporal Redundancy Approach rently occupies ring A and r(g) to denote the ring to which G is currently assigned. With the group-based approach, o-line management is responsible for message grouping, bandwidth allocation and schedulability verication while on-line management handles network initialization, fault detection, network reconguration, message group migration and reinitialization. In the rest of this section we rst explain grouping strategies to provide messages with certain redundancy then we introduce the notion of group rank which plays an important role in o-line and on-line management. 3. Temporal and Spatial Redundancy We shall illustrate our general strategy for message grouping with an example. Consider an FBRN consisting of 3 FDDI rings (i.e., r = 3). Imagine a message set M = fm ; M 2 ; M 3 ; M g, where the faulttolerance specications of the messages are given by F = 3; F 2 = 2; F 3 = ; F =. Let M and M 2 constitute Group and M 3 and M constitute Group 2. We consider an assignment of Group to Ring and Group 2 to Ring 2, with Ring 3 reserved as a back up as shown in Figure 3. Note that the highest F values of messages associated with Group and Group 2 are 3 and respectively. It is evident that Group 2 does not require migration in the event of faults. This is because, even when a single ring fault occurs, Group 2 can be dropped without violating the fault-tolerant real-time specication of its messages. On the other hand, message M in Group can be dropped only if there are 3 or more ring faults; i.e., it must tolerate at least 2 ring faults. So Group should be allowed to migrate a certain number of times before it gets dropped. This approach to assignment of message to rings can be termed as temporal redundancy. With temporal redundancy, a message can survive multiple faults if sucient time is available to migrate it from faulty rings to non-faulty ones. Sometimes, temporal redundancy may not work due to small message deadline. An alternate approach may be termed as spatial redundancy. This approach relies on sending multiple copies of messages rather than migrating the message to tolerate faults. For the message set mentioned in the above example, we send 3 copies of M on each of the three rings and 2 copies of M 2 on Ring and Ring 3 as shown in Figure. With spatial redundancy, a message can survive multiple faults if sucient number of copies of the message are assigned to distinct rings. In this approach a group consists of message copies.

4 Group : M,, M Ring 2, Ring 2 Group 2: M,2, M 3,, M, Ring 3 Group 3: M,3, M 2,2 F = 3, F 2 = 2, F 3 =, F = M i,j = the j-th copy of message M i Figure : Spatial Redundancy Approach The specifications of messages and network Selection of Temporal and Spatial Redundancy Factors Transformation of Message Set Failed The message set is rejected Spatial redundancy may not work for some cases. In the above example, although the bandwidth requirement of a single copy of M is now lower than with the temporal redundancy approach, we may overload Ring 2. This is because the combined bandwidth requirements of M, M 3, and M assigned to Ring 2 may exceed the bandwidth available on the ring. We adopt an integrated approach. That is, we use temporal as well as spatial redundancy to meet the fault-tolerant real-time requirements. Our approach is to transform a message M i into S i + copies, where S i 0. Each of these copies is assigned to a distinct group so that no two copies may appear on the same ring. Only one of the copies may provide temporal redundancy through migration. This copy is allowed T i? migrations, where T i. S i and T i are called the spatial and temporal redundancy factors. The S i copies, by themselves, ensure that at least one copy will survive after S i? faults. In addition to these S i copies, the single copy that is allowed to migrate T i? times can survive T i? faults. Thus, taken together, the S i + copies ensure that at least one copy of M i will survive a total of S i + T i? faults. Hence, choosing T i and S i such that T i + S i = F i () ensures that at least one copy of M i will be transmitted if the number of faults is less than F i. Since this is the desired level of fault tolerance for M i, we must choose S i and T i such that they satisfy (). 3.2 The Concept of Group Rank Associated with each group G is an integer called its rank, denoted by R(G). A group's rank reects the number of migrations that the group may require at a given instant of time. Each group is assigned an initial value that may be updated several times. We illustrate the rank concept using the example in Figure 3. In order to tolerate at least ring fault Group 2 is assigned a rank 0 to reect that it does not need to be migrated. On the other hand, message M in Group can be dropped only if there are 3 or more ring faults; i.e, it must tolerate at least 2 ring faults. Thus Group may have to be migrated twice (when two ring faults occur). Hence, Group is assigned a rank value of 2. The reader may note that the initial rank of a group has implications on the bandwidth allocation for messages in the group. Since transmission of a message is interrupted during its migration, excess bandwidth must be allocated to messages in order to ensure that Group Assignment Verification of Satisfaction of Fault-Tolerant Real-Time Requirements Passed message set accepted Figure 5: Flow-chart for O-line Management their transmission will be completed by their deadlines even when the group to which they belong migrates from one ring to another at run time. This will be taken care of in the algorithm for message grouping and bandwidth allocation that will be presented in Section. Before proceeding further, we would like to reemphasize that the rank value of a group is dynamic. After being initialized, the rank of a group may be updated while certain on-line management tasks are carried out. O-line Management We now focus on the most important tasks carried out by o-line management. The primary objective of this part is to transform the message set into groups so that each group can be assigned to an operational ring and the fault-tolerant real-time requirements of all messages are met. Based on the discussion of redundancy in Section 3 we identify three important tasks that need to be carried out during o-line management. They are as follows:. choosing the spatial and temporal redundancy factors (i.e., S i and T i values) for each message M i 2 M, which amounts to transforming M to an augmented message set M 0 obtained by replacing M i 2 M by its S i + copies; 2. partitioning the augmented message set M 0 into groups; and 3. verifying that the fault-tolerant real-time requirements of messages will be met. The reader is referred to Figure 5 for a owchart depicting the interactions among the above tasks.. Selection of S i and T i There are two stages in this task. In the rst stage, we choose initial values of S i and T i. In the second

5 stage, we iteratively update these values until all the fault-tolerant real-time communication requirements are met or the selection process fails to nd any values for S i and T i that can satisfy all the requirements. We concentrate on selecting T i. Since T i + S i must equal F i, once T i is determined, S i is obtained as F i? T i... Initial Selection Given the fact that a message with a larger temporal redundancy is likely to require less total bandwidth, we would like to select an initial value of T i as large as possible. Although F i is an upper bound for T i, we will see that a lower value of an upper bound on T i can be obtained by accounting for certain constraints that any selection of S i and T i must satisfy. We now dene these constraints. Migration Overhead Constraint With the temporal redundancy approach, a copy of message may need to migrate T i? times. In each migration, there is certain time overhead involved. The migration overhead constraint is needed to ensure that this overhead will not cause missing of deadlines. First, with FBRN architecture a ring failure can be detected in 2 TTRT time. Second, once a group migrates to another ring, the distributed mode change protocol [7] is invoked to drop the on-going transmissions and resume the transmission of the newly assigned group. Suppose K T T RT is the time taken for failure detection and mode change, which is also the maximum interruption time taken by a group migration caused by one ring failure. If a group is migrated T i? times during its transmission, then in the worst case, its transmission will be interrupted by K (T i? ) TTRT time. An ecient mode change protocol with K = 3 has been introduced in [7]. According to the timing property of timed token protocol used by FDDI network, in order to guarantee the message deadline it is necessary that D i 2TTRT. Accounting for the migration overhead, we need to modify this condition to D i? K (T i? ) TTRT 2TTRT : (2) This constraint must be satised by any selection of T i. Constraint on Available Bandwidth For each copy of the message, sucient synchronous bandwidth has to be allocated in order to meet its deadline requirement. Generally speaking, the smaller the deadline the larger is the bandwidth requirement. We use the local bandwidth allocation method proposed in [8]. With this method, the bandwidth for a message M i is given by H t i (T i ) = U i D i b Di?K(Ti?)TTRT TTRT c? (3) C where U i = i min(p i;d i) denotes the eective utilization or the demand placed by this message on the network. Note that in this formula, we already take into account the migration overhead that may occur. However, this bandwidth should not exceed the available bandwidth on a single ring. That is, H t i (T i ) T T RT? : () This is the second constraint that any selection of T i must satisfy. Constraint on Potential Bandwidth Saving To justify the temporal redundancy, one should show that using it would require lesser total bandwidth than using pure spatial redundancy to achieve the same level of fault tolerance. Imagine that we do not take the temporal redundancy approach but instead use T i copies of the message with no migration allowance for any copy. Clearly, the message fault-tolerance requirement is still met. The bandwidth for each of the T i copies of M i is given by H s i = U i D i b Di TTRT c? : (5) To justify the copy with temporal redundancy, we should ensure that the following inequality holds. H t i (T i) T i H s i (6) where H t i (T i) is as dened in (3) and H s i as in (5). This is the third constraint that any selection of T i has to satisfy. In summary, the selection of T i is subject to constraints (2), (), and (6). Thus, we want to nd the maximum value of T i that satises (2), (), and (6). By solving (2), () and (6) we nd that the desired value of T i is given by T i = bmax(; min(a; A? D i U i D i K (T T RT? ) + )c where A = b T T RT?c K. Thus, we use this as the initial assignment for T i. The corresponding initial value of S i is obtained as S i = F i? T i...2 Updating S i and T i It is possible that the message set cannot pass the verication test for a particular choice of S i s and T i s. That is, with this selection, the fault-tolerant realtime requirements are not met. In this case, we need to update the values of the redundancy factors. Once again, we focus on T i s. Since the initial value of each T i was chosen to be the largest possible one, the update process only involves decrementing the T i s. We use a simple update procedure. During the update

6 procedure, we select a message M i that has the maximum temporal redundancy value (T i ) among all the messages. We reduce T i by one and test again to see if a message grouping can satisfy the fault-tolerant real-time requirements. Note that T i. Hence, the procedure terminates if some T i is decremented to zero during the update procedure. In such a case, our selection process declares the message set to be unschedulable..2 Transformation of the Message Set Given the selected values of temporal and spatial redundancy factors for messages, the original message set M is the collection of all (C i ; P i ; D i ; F i ; T i ; S i ) tuples where C i ; P i ; D i, and F i are message length, period, deadline and fault-tolerance level respectively. T i is the temporal redundancy factor and S i is the spatial redundancy factor. To facilitate the group assignment, we further transform the message set to obtain an augmented message set M 0 as follows. Each message M i 2 M, is replaced by its S i + copies denoted as M i; ; M i;2 ; : : : ; M i;si+. The rst S i copies (M i; ; M i;2 ; : : : ; M i;si ) provide spatial redundancy alone, and the last copy (M i;si+) employs temporal redundancy. Formally, the transformation of message M i is de- ned as follows: M i! fm i; ; M i;2 ; : : :; M i;si+g where the message copy M i;j is specied as M i;j = (i; j; H i;j ; R i;j ): We now explain the meaning of each parameter. The rst parameter, i, indicates that this is a copy of the original message M i. The second parameter, j, indicates that this is the jth copy of the message. H i;j is the bandwidth that has to be allocated to the copy and is given by H i;j = 8>< >: b U id i D i T T RT c? if j 6= S i + U id i b D i?k(t i?)t T RT T T RT c? if j = S i + : Finally, R i;j is the migration factor for this copy which is 0 if j 6= S i + and T i? if j = S i +. From these denitions, it is easy to see that M i;, M i;2, : : :, M i;si correspond to the copies for spatial redundancy purpose while M i;si+ is the one that provides temporal redundancy. Once this transformation is done for every message in the original set, we obtain the transformed set M 0 as the set of all M i;j. Set M 0 will be used as input for the group assignment algorithm that is described next..3 Group Assignment The function of the group assignment procedure is to partition the augmented message set M 0 into groups. Before we describe the procedure, we need to dene some notations. Formally, a group G is a subset of the augmented message set. Each group G is initially assigned its maximum rank value denoted R max (G) which is given by R max (G) = max M i;j2g (R i;j ): (7) R max (G) represents the maximum number of times G may have to migrate in order to meet the faulttolerance requirements of all the message copies in the group. Let H(G) denote the total bandwidth required by the message copies that belong to G. Thus, X H(G) = H i;j : M i;j2g The assignment of messages to a group is subject two constraints: Total Bandwidth Constraint. Because each group is going to occupy one ring at run time, the total bandwidth needed by the messages in the group should not exceed the total bandwidth available on a ring. That is, H(G) T T RT? : (8) Mutual Exclusion Constraint. Two copies of the same message should not be assigned to the same group. Formally, If M i;j 2 G, then for j 0 6= j, M i;j 0 62 G: (9) A copy of message is admissible into a group if adding the copy to the group will not violate (8) and (9). Assignment of messages into groups is a NP-hard problem. To see this, consider a special case when no fault tolerance is required for any message, i.e., 8i F i =. In this case, there is no need for either spatial or temporal redundancy. For the local bandwidth allocation scheme chosen, we only need to verify that the output of the group assignment algorithm satis- es the total bandwidth constraint (8). Hence, in this case, the message set is schedulable if and only if it can be partitioned into r or less number of groups such that the total bandwidth requirement of each group is less than or equal to T T RT?. This partition problem is NP-hard since the bi-partition problem is a special case of it. For the reason of eciency and simplicity, we design an algorithm based on the rst-t approach. For each copy of the message, we will try to nd a group to which it is admissible. If no such group exists, a new group will be generated to accommodate it. The pseudocode of the algorithm is shown below. Input: augmented message set M 0 Output: GS which is a set of groups Begin GS = ;; I = 0; (* I is the number of groups generated *) while (M 0 is not empty) do

7 pick an element M i;j = (i; j; H i;j; R i;j ) from M 0 ; M 0 = M? 0 fm i;jg; Initialization On-Line Management assigned = FALSE; (* Mi;j is not assigned yet *) k = ; while (k I && assigned == FALSE) if (Mi;j is admissible into group Gk ) then (* Assign M i;j to G k *) G k = G k [ fm i;jg; assigned = TRUE; else k = k + ; endif endwhile if (assigned == FALSE) then (* generate a new group and admit M i;j *) I = I + ; generate a new group GI; G I = fm i;jg; GS = GS [ GI; endif endwhile End Once the membership of a group is determined, the maximum rank value of the group is found by (7). The rank will be adjusted at run time to reect the system status. Since each group consists of message copies instead of original messages, we dene M(G) be the set of message ids corresponding to the copies assigned to G. That is, M(G) = fi j M ij 2 Gg.. Schedulability Verication The verication algorithm should check whether current assignment can satisfy all fault-tolerant realtime requirements of given message set. The algorithm guarantees that once an assignment passes schedulability verication, all fault-tolerant real-time requirements will be met. A trivial verication algorithm simply rejects any assignment. Reasonable verication algorithm should accept as many assignments as possible. Since real-time requirements are already satised through bandwidth allocation we only need to develop a criteria to check fault-tolerance requirements. Let GS be the output of the above group assignment algorithm. For group G, dene GS(G) = fg 0 2 GS j M(G) \ M(G 0 ) = ;g. That is, GS(G) contains those groups that do not share any copy of messages with G. We then dene a partition on GS(G) as follows: for i = 0; : : :; r?, GS i (G) = fg 0 2 GS(G) j R max (G 0 ) = ig: For each G 2 GS the verication criteria is to check if for any 0 j R max (G) the following inequality holds X j? (r? j GS j) + i=0 j GS i (G) j j: (0) Group Migration Fault Detection Reconfiguration Reinitialization Figure 6: Flow-chart of On-line Management The verication algorithm simply applies this criteria to each group in GS. If all groups of GS pass the above criteria then the current group assignment passes the verication and the message set is accepted. The correctness of the above verication algorithm depends on both o-line and on-line management and cannot be proved until on-line management is presented. Therefore, we will rst describe on-line management and address the correctness issue in Section 6. 5 On-line Management In this section we will briey discuss the activities involved in on-line management. Figure 6 shows the sequence of management tasks of on-line management. 5. Network Initialization The initialization of each FDDI trunk ring is similar to single FDDI ring with two additional tasks. The rst task is to assign groups to rings with at most one group per ring. The other task is to initialize rank of each group to be the maximum value of its rank calculated by o-line management. That is R(G) = R max (G) where R max (G) is dened in (7). 5.2 Fault Detection Recall that an FBRN is composed of multiple FDDI token rings. Fault detection within FBRN is a decentralized process with each FDDI ring performing its standard fault detection operations. In FDDI, each station is guaranteed to receive the token at least once in a time interval of length 2 TTRT where TTRT is the target token rotation time. If a station does not receive the token in this time, then it initiates the fault location and recovery procedure. Once a fault is detected, the on-line management concurrently carries out two tasks: viz., network reconguration and message migration. These tasks are described next. 5.3 Network Reconguration Network reconguration is part of FBRN's two-tier fault management system. Each FDDI trunk ring has its fault recovery mechanism which enables the ring to recover from node faults using the bypass switch and from single point trunk faults using the wrap up operation. FBRN also has a network-level fault recovery mechanism which monitors the available bandwidth in

8 the network and invokes a reconguration algorithm when the lower level FDDI-based recovery methods are inadequate []. The problem of designing ecient reconguration algorithms for FBRN has been addressed in [3, ]. The reconguration algorithm developed in [3] is optimal in the sense that it always produces a conguration that has the largest number of functional rings for the given fault pattern. One disadvantage of the optimal algorithm is that it requires global information regarding the system fault status. In mission-critical systems, the communication overheads incurred in collecting this information may be intolerable. The local reconguration algorithm described in [] does not suer from this problem. The local algorithm operates in a fully distributed fashion utilizing only locally available information at each node. Further, with this algorithm, the reconguration process is transparent to the fault-free rings; the ongoing trac on these rings is unaected by the reconguration process. Although the local algorithm is not optimal, it is demonstrated to have a near-optimal average case performance []. Hence, we employ the local reconguration algorithm. 5. Message Migration Once a fault condition is detected on a ring, the transmission of messages on that ring is disrupted. The messages on the aected ring may have to be migrated to some other ring. To illustrate the message migration process, reconsider the previous example shown in Figure 3. The FBRN in the example consists of 3 FDDI rings and the assignment of message groups to rings is as shown. If Ring 2 is detected as faulty, then there is no need to migrate messages in Group 2. This is reected in the value of Group 2's rank which is 0. On the other hand, consider a scenario where Ring 2 and Ring 3 are operational but Ring becomes faulty. Now, Group can be migrated to Ring 3. Ring 3 is considered assignable to Group since it has no group currently assigned to it. Further, if Ring and ring 3 were both faulty, we can still migrate Group from Ring to Ring 2. In this situation, Ring 2 is considered assignable to Group because dropping Group 2 that is currently assigned to it does not violate any of the fault-tolerant real-time requirements. The decisions such as whether to migrate a group or not and which ring to migrate to will make use of the notion of group ranks. Formally, the task of message migration is given by the following algorithm.. Let R be the rank of the group assigned to the faulty ring; 2. If R = 0 then drop this group; go to Step 6; 3. Find an operational rings that is assignable to this group;. If assignable ring has a group assigned to it then drop the group; 5. Migrate the group from faulty ring to the selected operational ring; 6. Update the group ranks. In Step 2 of the above algorithm, R = 0 implies that there is no reason to migrate the group, and hence this group is dropped. If R > 0, then we need to migrate the aected group. In Step 3, we look for an operational ring that is assignable to this group. Clearly, an operational ring that does not have any group currently assigned is assignable to the aected group. The formal denition of assignable is as follows: ring A is assignable to group G if one of the following conditions holds: ) Ring A has no group currently assigned to it. Formally, G(A) = ; ; 2) The group currently assigned to ring A can be dropped because it does not need to tolerate any more ring failures. That is, R(G(A)) = 0. However, dropping G(A) and then migrating G to ring A does not make sense if G(A) and G share copies of some message. Thus, the second condition is as follows: R(G(A)) = 0 and M(G) \ M(G(A)) = ;: The reader may wonder what happens when there is no operational ring that can be assigned to the affected group. The message assignment algorithm of o-line management (in conjunction with the group manipulations performed by on-line management) ensures that this situation never arises. The schedulability verication tests derived in Section. ensure that whenever R(G) (that is, the aected group requires further migration) there must exist an operational ring assignable to this group. O-line management rejects a message set if it does not satisfy this test. Step 6 updates the group ranks to reect the change in network status due to the ring fault. It decrements the ranks of certain groups which now require one less migration than before. Let G drop be the group that will be dropped. G drop is considered to be the empty set when no group needs to be dropped. The ranks of the groups that are currently assigned are modied as follows: R(G) = 8>< >: R(G) R(G)? if R(G) = 0 or M(G) \ M(G drop ) 6= ; otherwise. The fact that the rank of group is zero implies that the group has no need to tolerate any fault. Thus, there is no need to change its rank when some other group is dropped. Also, if group G shares some copy of a message with G drop, the group to be dropped, then dropping G drop has no impact on the rank of G. This justies the above formula. 5.5 Reinitialization Recall that in the event of a ring fault, the group assigned to that ring may have to be migrated to another ring. As discussed earlier, the ring chosen for migration may have a previously assigned group. The faulttolerant real-time management ensures that dropping that group does not violate any message requirements.

9 However, it is desirable that we reassign such dropped groups later when more rings are recovered as a result of network reconguration. This objective is accomplished by the reinitialization task. The main function of the reinitialization task is to choose a group for reassignment from those that are currently unassigned to any ring. The notion of group rank simplies this task. A high initial value of rank signies a high level of desired fault-tolerance for the group. Hence, when the network recovers a ring, the reinitialization task selects the unassigned group with the highest initial rank value for reassignment. Because an additional ring is now available as a result of reconguration, the ranks of some of the groups are incremented by one. Let G recv be the selected group. If no group is selected then G recv = ;. The ranks of some groups that are currently assigned need to be modied, reecting the fact that they can now tolerate more faults. The rule is as follows: R(G) = 8>< >: R(G) + R(G) if R(G) < R max (G) and M(G) \ M(G recv ) = ; otherwise. This is essentially the reverse of the rank modication performed when ring failure is detected. 6 Performance In this section we frist complete the correctness proof of the schedulability verication algorithm introduced in Section.. Then we compare the performance of pure temporal redundancy and pure spatial redundancy approaches with that of our integrated approach. 6. Correctness of Verication Algorithm To justfy the correctness of the verication algorithm given in Section. we need to prove the following theorem. THEOREM 6. 0 j R max (G) (r? j GS j) + If for every G 2 GS and any j? X i=0 j GS i (G) j j () where r is the number of rings, then the fault-tolerant real-time requirements of all the messages are satised. Sketch of proof: There are two parts of the proof: satiscation of fault-tolerance requirement and satiscation of real-time requirement. The rst part can be proved by showing that for G 2 GS if for any 0 j R max (G) inequality () is always satis- ed then at run time whenever R(G), there is an assignable ring for group G. The second part is already guaranteed by local bandwidth allocation scheme. Proof details are omitted due to space limitation. 6.2 Performance Comparison We believe that using proper temporal and spatial redundancies is the key to realizing a highly ecient system. A high temporal redundancy may require too much overhead for migration of messages from one ring to another, resulting in missing of deadlines. On the other hand, too much spatial redundancy may overload the network, making the message assignment infeasible. Here we compare the performance of pure temporal redundancy and pure spatial redundancy approaches with that of our integrated approach. We evaluate the performance through simulation. In the simulation we let K = 3 based on the mode change protocol described in [7]. We choose guarantee ratio, dened as the number of message sets guaranteed divided by total number of message sets generated, as our metric. The simulation program was written in C programming language and run in a Sun/Solaris environment. We consider an FBRN with FDDI rings. TTRT for each ring is taken as 0 milliseconds. We investigate three dierent loading conditions given by the average utilization per ring being 25%, 0%, and 50% respectively. Each message stream is characterized by the tuple (C i ; P i ; D i ; F i ). D i s are chosen from an exponential distribution with a mean that varies from TTRT to 52 TTRT. F i s are uniformly distributed between 0 and 3. In each run, the program generates 0,000,000 instances of message sets. Figure 7 shows the results. From this gure, we can make the following observations: The performance of all three approaches is sensitive to the average deadline and average utilization. The performance of pure temporal redundancy approach is very sensitive to message deadlines. On the other hand, the performance of pure spatial redundancy approach is much less sensitive to deadlines. However, the spatial redundancy approach is very sensitive to utilization. In all the cases tested, it is seen that our integrated approach performs far better than pure temporal or pure spatial approach. This is because our algorithm achieves a good balance between the two redundancies in order to meet the fault-tolerant real-time communication requirements. 7 Conclusions In this paper we addressed the problem of satisfying the combined fault-tolerant real-time requirements of messages with an FDDI-based recongurable network. Our approach to this problem involved message groupbased o-line and on-line management components. This approach is eective and ecient and can fully explore the reconguration capability provided by FDDI-based recongurable network. We observed that the performance of the system critically depends on the message grouping strategies used in o-line management. We considered three approaches: pure

10 Guarantee Ratio Guarantee Ratio Guarantee Ratio (A) Average Ring Utilization = Average Deadline (B) Average Ring Utilization = 0.0 Pure Spatial Pure Temporal Integrated Pure Spatial Pure Temporal Integrated Average Deadline Pure Spatial Pure Temporal Integrated (C) Average Ring Utilization = Average Deadline Figure 7: Performance Results temporal redundancy, pure spatial redundancy and intergrated approach. The intergrated approach strikes a balance between bandwidth requirement and migration overheads of pure spatial and pure temporal redundancy approaches and hence outperforms both of them. Our solution fully exploits the established results on recongurability of FDDI-based networks and the real-time capabilities of FDDI. It is practical because it is modular and easily scalable and implementable. References [] G. Agrawal, S. Kamat, and W. Zhao, \Architectural Support for FDDI-Based Recongurable Networks," Workshop on Architectures for Real-Time Applications (WARTA), 99. [5] G. Agrawal, B. Chen, W. Zhao, and S. Davari, \Guaranteeing Synchronous Message Deadlines with the Timed Token Protocol," IEEE Transactions on Computers Vol. 3, No. 3, March 99. [6] Q. Zheng and K.G. Shin, \Synchronous Bandwidth Allocation in FDDI Networks," Proc. ACM Conf. on Multimedia, 993. [7] B. Chen, \Fault-Tolerant Real-Time Communications," PhD Thesis, Texas A&M University, under preparation. [8] N. Malcolm, S. Kamat, and W. Zhao, \Real-Time Communication in FDDI Networks," to appear in Journal of Real-Time Systems, nal revision in Feb [9] L. Sha, S. S. Sathaye, and J. K. Strosnider, \Scheduling real-time communication on dual-link networks," Proc. IEEE RTSS, 99. [0] J. K. Strosnider, T. Marchok, and J. Lehcozky \Advanced real-time scheduling using the IEEE token ring," Proc. IEEE RTSS, 988. Glossary Term Location F i 2. G(A) 3. GS.3. GS(G). GS i (G). K... M(G).3. M i;j.2. M 2.. M 0.2. R i;j.2. R max (G).3. R(G) 3.2. S i 3.. T i 3.. r(g) 3. f(t) 2.. h(t; t + D) 2.. [2] S. Ralph, O. Ukrainsky, R. Schellack, and L. Weinberg, \Alternate Path FDDI Topology," Proc. IEEE LCN, 992. [3] S. Kamat and W. Zhao, \An Ecient Optimal Reconguration Algorithm for FDDI-Based Networks," To appear in IEEE Transactions on Parallel and Distributed Systems. [] S. Kamat, G. Agrawal, and W. Zhao, \A Local Reconguration Algorithm for FDDI-Based Recon- gurable Networks," Submitted for publication.