184 J. Comput. Sci. & Technol., Mar. 2004, Vol.19, No.2 On the other hand, however, the probability of the above situations is very small: it should b

Transcription

1 Mar. 2004, Vol.19, No.2, pp J. Comput. Sci. & Technol. On Fault Tolerance of 3-Dimensional Mesh Networks Gao-Cai Wang 1;2, Jian-Er Chen 1;3, and Guo-Jun Wang 1 1 College of Information Science and Engineering, Central South University, Changsha , P.R. China 2 School of Computer and Electrical Information, Guangxi University, Nanning , P.R. China 3 Department of Computer Science, Texas A&M University, U.S.A. wanggaocai@yahoo.com.cn Received May 15, 2003; revised August 29, Abstract In this paper, the concept of k-submesh and k-submesh connectivity fault tolerance model is proposed. And the fault tolerance of 3-D mesh networks is studied under a more realistic model in which each network node has an independent failure probability. It is first observed that if the node failure probability is fixed, then the connectivity probability of 3-D mesh networks can be arbitrarily small when the network size is sufficiently large. Thus, it is practically important formulticomputer system manufacturer to determine the upper bound for node failure probability when the probability of network connectivity and the network size are given. A novel technique is developed to formally derive lower bounds on the connectivity probability for 3-D mesh networks. The study shows that 3-D mesh networks of practical size can tolerate a large number of faulty nodes thus are reliable enough for multicomputer systems. A number of advantages of 3-D mesh networks over other popular network topologies are given. Compared to 2-D mesh networks, 3-D mesh networks are much stronger in tolerating faulty nodes, while for practical network size, the fault tolerance of 3-D mesh networks is comparable with that of hypercube networks but enjoys much lower node degree. Keywords 1 Introduction interconnection network, 3-D mesh network, fault tolerance, parallel processing With the size of computer networks increasing rapidly, dealing with networks with faulty components has become unavoidable. In particular, the problem of keeping all non-faulty nodes in a network connected and developing efficient and reliable routing algorithms in a network with faults has been extensively studied in the last two decades. In this paper, we will concentrate on the fault tolerance of 3-D mesh networks. Mesh networks are among the most important and popular interconnection network topologies for large multicomputer systems. The advantages of mesh networks include their simplicity, regularity and good scalability. A number of large research and commercial multicomputer systems have been built based on mesh topologies, including Illiac IV, Goodyear MPP, Intel Paragon, Intel Touchstone Delta, Stanford DASH, MIT Alewife, MasPar series, Tera Computer System, Cray T3D, and Blue Gene Supercomputer. In particular, Tera Computer System [1], Cray T3D [2], and Blue Gene Supercomputer in IBM Research [3] are based on 3-D meshes. Fault tolerance of mesh networks has been extensively studied. Some researches focused on how to design processor allocation schemes to find a subset of nodes of a given size in the mesh networks such that all nodes in the subset are nonfaulty [4 7]. It is also an interesting research topic to consider how to make the communication between non-faulty nodes available and efficient in a mesh network with faulty nodes [8 10]. There is an extensive research line that studies fault tolerant routing algorithms in mesh networks with faults (see, for example, [11 14]). We say that a network M with faulty nodes is disconnected (by faulty nodes) if there are nonfaulty nodes u and v in M such that no fault-free path from u to v can be found in M. Following the common definition [15], define the fault tolerance of a network to be the maximum number k such that any k faulty nodes in the network do not disconnect the network. According to this definition, the fault tolerance of mesh networks is very poor in the view of worst-case analysis: two faulty nodes in a 2-D mesh network and three faulty nodes in a 3-D mesh network are already sufficient to disconnect the networks. This trivial fact has made it uninteresting to consider the traditional worst-case analysis on mesh network fault tolerance. Λ Correspondence This research is supported in part by the National Natural Science Foundation of China for Distinguished Young Scholars under Grant No , the Major Research Plan of National Natural Science Foundation of China, Grant No , and by the National Science Foundation of USA under Grant No.CCR

2 184 J. Comput. Sci. & Technol., Mar. 2004, Vol.19, No.2 On the other hand, however, the probability of the above situations is very small: it should be very rare that the only two or three faulty nodes in a mesh network are just the neighbors of a corner node. It should be much more realistic to consider the mesh network fault tolerance based on a somehow uniform" fault distribution in the networks. The difficulty for this approach is that a mesh network may contain a large variety of connected subparts. Enumeration and characterization of these connected subparts can be very difficult. In this paper, we introduce new techniques to analyze the fault tolerance of 3-D mesh networks. We assume a realistic fault tolerance model in which each node in a 3-D mesh network has an independent failure probability, and study the probability of network connectivity in terms of node failure probability. We first observe that if the node failure probability is fixed, then the connectivity probability of 3-D mesh networks can be arbitrarily small when the network size is sufficiently large. Therefore, in order to keep a high connectivity probability for 3-D mesh networks of large size, the network node failure probability must be controlled. This then raises practically a very important problem for multicomputer system manufacture: suppose we need to build a multicomputer system based on a 3-D mesh network of given size, what probability for node failures we must achieve in order to satisfy a required probability for network connectivity? We have developed new techniques to deal with this problem. We first introduce the concept of k- submesh and k-submesh connectivity fault tolerance model, which divides a 3-D mesh network into ksubmeshes" and requires a special connectivity on each k-submesh (the precise definition will be given in Section 3). We prove that if all k-submeshes are connected in this sense, then the entire 3-D mesh network is connected. The advantage of this concept is that we reduce the difficult task of determining the connectivity of the entire large network to the easier task of determining the connectivity of much smaller submeshes. Now we calculate the probability of k-submesh connectivity for each k- submesh. Since the k-submeshes are small, this can be done either by a simple combinatorial analysis or by an exhaustive enumeration. From the probability of k-submesh connectivity, we will be able to derive lower bounds of the probability for the connectivity of the entire 3-D mesh network. From the above analysis, we will be able to derive upper bounds of the node failure probability when the size and the connectivity probability of a 3-D mesh network are given. Our results show that the 3-D mesh networks of practical size under this realistic fault model can actually tolerate a large number of faulty nodes. For example, we present formal proofs showing that as long as the node failure probability is bounded by 0:5%, a 3-D mesh network of up to a million nodes can remain connected with a probability larger than 99%. Our techniques also enable us to demonstrate a number of advantages of 3-D mesh networks over other popular network topologies. For example, compared with 2-D mesh networks, 3-D mesh networks are much stronger in tolerating faulty nodes, while for practical network size, the fault tolerance of 3-D mesh networks is comparable with that of hypercube networks but enjoys much lower node degree in the networks. 2 On Disconnectivity of 3-D Meshes In this paper, M m n q denotes the 3-D mesh network of size m n q with mnq nodes. Each node in M m n q is given by a coordinate triple (x; y; z), where 1 6 x 6 m, 1 6 y 6 n, and 1 6 z 6 q. To simplify our discussion, we assume without loss of generality that the values of m, n, and q are all divisible by k. Two nodes v = (x; y; z) and v 0 = (x 0 ;y 0 ;z 0 ) in M m n q are neighboring if jx x 0 j + jy y 0 j + jz z 0 j = 1, i.e., if v and v 0 are identical in two coordinates while the values for the other coordinate differ by 1. Definition 1. A k-submesh M k (or sometimes more specifically M (k) ) in the 3-D mesh network M m n q is determined by three integers a, b and c, 0 6 a < m=k, 0 6 b < n=k, and 0 6 c < q=k such that M (k) contains the following k3 nodes (x; y; z) in M m n q, where ak +16 x 6 (a +1)k, bk +16 y 6 (b +1)k, and ck +16 z 6 (c +1)k. We assume that each node in a 3-D mesh network fails independently with a probability p. We first show that when the node failure probability p is a fixed constant, then the probability that a 3-D mesh network is disconnected can be arbitrarily close to 1 when the network size is sufficiently large. Theorem 2. If the node failure probability p > 0 is a fixed constant, then the probability that the 3-D mesh network M m n q is disconnected can This model has been studied on the hypercube networks [16;17] and 2-D mesh networks [18].

3 Gao-Cai Wang et al.: On Fault Tolerance of 3-Dimensional Mesh Networks 185 be arbitrarily close to 1 when the network size mnq is sufficiently large. Proof. For a 3-submesh M 3 in M m n q, if the node v in the center of M 3 is non-faulty and all six neighbors of v are faulty, we call M 3 a bad 3- submesh. Obviously, a bad 3-submesh disconnects the center node from the rest of the mesh network. Since the nodes fail independently, the probability that a 3-submesh is bad is equal to p 6 (1 p). Since the 3-D mesh M m n q contains mnq=27 disjoint 3-submeshes and the node failure probability is independent, the probability that there is at least one bad 3-submesh in M m n q is at least 1 (1 p 6 (1 p)) mnq=27. Now the fact that p > 0 is a constant implies that 1 p 6 (1 p) < 1 is also a fixed constant. Thus, when mnq=27 (thus mnq) is sufficiently large, 1 (1 p 6 (1 p)) mnq=27 can be arbitrarily close to 1. Moreover, the probability that M m n q contains at least two non-faulty nodes can obviously be arbitrarily close to 1 when the mesh size mnq is sufficiently large. Now the conditions that M m n q contains at least one bad 3-submesh and that M m n q contains at least two non-faulty nodes imply immediately that M m n q is disconnected. That is, the probability that the 3-D mesh network M m n q is disconnected can be arbitrarily close to 1 when the network size mnq is sufficiently large. 2 Theorem 2 reveals that, in order to keep a high connectivity probability for 3-D mesh networks of large size, the network node failure probability must be controlled. To build a multicomputer system based on a 3-D mesh network of a given size, what probability for node failures we must achieve in order to satisfy a required probability for network connectivity? The rest of this paper will develop techniques to answer this question. 3 The k-submesh Connectivity According to the definition of k-submeshes, the 3-D mesh network M m n q can be divided into (m=k) (n=k) (q=k) k-submeshes. Two k- submeshes M (k) (k) and M a 0 ;b 0 ;c are neighboring if 0 ja a 0 j + jb b 0 j + jc c 0 j = 1. Each k-submesh M k is a 3-D mesh of size k k k. If we regard a k-submesh M k as a 3-D cube, then naturally, M k has six faces", each consisting of k 2 nodes. Definition 3. The 3-D mesh M m n q is k- submesh connected if for each k-submesh M k in M m n q, all non-faulty nodes in M k make a connected graph, and each face of M k has more than k 2 =2 non-faulty nodes. We have the following theorem. Theorem 4. The non-faulty nodes in a k- submesh connected 3-D mesh network M m n q make a connected graph. Proof. Suppose that the 3-D mesh network M m n q is k-submesh connected. We first prove that all non-faulty nodes in any two neighboring k-submeshes in M m n q are connected. Each face of a k-submesh has k 2 nodes, so k 2 pairs of neighboring nodes are formed at the two adjacent faces of two neighboring k-submeshes. According to the definition, more than k 2 =2 nodes on each face of each k-submesh are non-faulty, so one pair of non-faulty neighboring nodes must exist in the k 2 pairs of neighboring nodes on the adjacent faces of the two neighboring k-submeshes, which provides a bridge" from one k-submesh to the other k-submesh. Since the non-faulty nodes in each k- submesh are connected, we derive that all nonfaulty nodes in the two neighboring k-submeshes are connected. Now we prove that the non-faulty nodes in the k-submesh connected 3-D mesh M m n q make a connected graph. For this, we only need to show that for any two non-faulty nodes u and v in M m n q, there is a fault-free path between u and v. Assume that the node u lies in the k-submesh M (k) and the node v lies in the k- submesh M (k) a 0 ;b 0 ;c 0. Without loss of generality, we assume a 6 a 0, b 6 b 0, and c 6 c 0 (other cases can be proved similarly). We consider the following k- submesh sequence: M (k) ;M(k) a+1;b;c ;:::;M(k) a 0 ;b;c ;M(k) a 0 ;b+1;c ;:::; M (k) a 0 ;b 0 ;c ;M(k) a 0 ;b 0 ;c+1 ;:::;M(k) a 0 ;b 0 ;c 0 Any two consecutive k-submeshes in the above sequence are neighboring k-submeshes. As proved above, all non-faulty nodes in any two consecutive k-submeshes in the sequence are connected. In consequence, all non-faulty nodes in the above sequence are connected. Since the node u lies in the k-submesh M (k) while the node v lies in the k- submesh M (k) a 0 ;b 0 ;c, there must be a fault-free path 0 starting from u in the k-submesh M (k) and ending at v in the k-submesh M (k) a 0 ;b 0 ;c. 2 0 According to the definition, the k k k 3-D mesh M k k k is k-submesh connected if all nonfaulty nodes in M k k k are connected, and each face of M k k k has more than k 2 =2 non-faulty

4 186 J. Comput. Sci. & Technol., Mar. 2004, Vol.19, No.2 nodes. In particular, a 3-D mesh M m n q is k- submesh connected if and only if every k-submesh in M m n q is k-submesh connected. For the 3-D mesh M k k k, we define the following event: Event C(M k ): The 3-D mesh M k k k is k- submesh connected. Lemma 5. Suppose that the node failure probability is p, then the probability of the event C(M k ) is Pr[p; C(M k )] = Xk 3 i=0 N k;i (1 p) k3 i p i where N k;i is the the number of ways by which we remove i nodes from M k k k and the remaining graph is connected and each face of M k k k has more than k 2 =2 nodes left in the remaining graph. Proof. Fix i nodes in M k k k such that removal of these i nodes from M k k k leaves a connected graph and each face of M k k k has more than k 2 =2 nodes left. Note that the probability that exactly these i nodes are faulty while all other nodes are non-faulty is (1 p) k3 i p i (recall that M k k k has exactly k 3 nodes). Note that in this situation, the mesh M k k k is k-submesh connected. If we enumerate all these possible situations for all i = 0; 1;:::;k 3, and add the probabilities together, we get the probability that M k k k is k-submesh connected. By the assumption, there are N k;i ways to remove i nodes from M k k k while maintaining the conditions. Thus, the probability that M k k k is k-submesh connected is P k 3 i=0 N k;i(1 p) k3 i p i. 2 4 Fault Tolerance of 3-D Meshes Theorem 4 seems simple but in fact is very powerful for the following reason: each k-submesh M k has much fewer nodes than the original 3-D mesh network M m n q. Thus, analysis and discussion on each k-submesh are much easier than that for the original 3-D mesh network M m n q. On the other hand, Theorem 4 says that if every k-submesh is k-submesh connected, then the original 3-D mesh network M m n q is connected. Therefore, using Lemma 5 we can compute the probability of k- submesh connectivity for each k-submesh. Then by Theorem 4 and from the probability of k-submesh connectivity for each k-submesh, we will be able to derive a lower bound on the connectivity probability for the original 3-D mesh network M m n q. We implement this idea in the following theorem. Theorem 6. Suppose the node failure probability is p. Then the probability that the 3-D mesh network M m n q is connected is at least (Pr[p; C(M k )]) mnq=k3, where Pr[p; C(M k )] is given in Lemma 5. Proof. There are mnq=k 3 k-submeshes in the 3-D mesh M m n q. Since all k-submeshes in M m n q are disjoint and the nodes in M m n q fail independently, the probability that all k-submeshes in M m n q are k-submesh connected is equal to (Pr[p; C(M k )]) mnq=k3. Therefore, the probability that the 3-D mesh M m n q is k-submesh connected is equal to (Pr[p; C(M k )]) mnq=k3. Now by Theorem 4, k-submesh connectivity of M m n q implies that all non-faulty nodes in M m n q are connected, and we conclude that the probability that the 3-D mesh network M m n q is connected is at least (Pr[p; C(M k )]) mnq=k3. 2 To apply Theorem 6, we let k = 3 and consider the probability Pr[p; C(M 3 )]. The 3-D mesh M has 27 nodes. Note that the values N 3;i can be computed in a straightforward way: we pick every possible selection of i faulty nodes in the 3-D mesh M 3 3 3, and record the number of selections by which M is 3-submesh connected (i.e., the remaining nodes make a connected graph and each face of M has at least d3 2 =2e = 5 nodes left). This process can be easily programmed and executed by a computer. Thus, we wrote a computer program for this purpose that computed all values N 3;i. The computational results are summarized in the following lemma. Lemma 7. Suppose that the node failure probability is p, then the probability of the event C(M 3 ) is Pr[p;C(M 3)] = (1 p) p(1 p) p 2 (1 p) p 3 (1 p) p 4 (1 p) p 5 (1 p) p 6 (1 p) p 7 (1 p) p 8 (1 p) p 9 (1 p) p 10 (1 p) p 11 (1 p) p 12 (1 p) p 13 (1 p) p 14 (1 p) 13 +4p 15 (1 p) 12 Combining Theorem 6 and Lemma 7, for each node failure probability p, we can derive a lower bound (Pr[p; C(M 3 )]) mnq=27 for the connectivity probability for the 3-D mesh M m n q. Moreover, suppose we are given a requirement that the connectivity probability of the 3-D mesh M m n q be at least p 0. Then by letting p 0 6 (Pr[p; C(M 3 )]) mnq=27

5 Gao-Cai Wang et al.: On Fault Tolerance of 3-Dimensional Mesh Networks 187 we can derive an upper bound p on the node failure probability such that as long as the node failure probability p is bounded by p, the connectivity probability of the 3-D mesh M m n q is at least p 0. Note that all results are derived based on the above formal mathematical analysis. We apply the above method to 3-D mesh networks of different sizes. We obtain two groups of results: one is under the condition that the node failure probability is p = 1% and we compute the corresponding connectivity probability for the 3-D mesh networks; and the other is under the required connectivity probability p 0 = 99% for the networks, we compute the corresponding node failure probability p that meets the requirement for network connectivity probability. The results are given in Table 1. Table 1. Results Based on Pr[p; C(M 3 )] 3-D mesh #nodes connectivity prob :98% ; :95% ; :83% ; :61% ; :39% ; :71% (a) Network connectivity probability when node failure probability is p = 1%: 3-D mesh #nodes node failure prob :46% ; :63% ; :77% ; :89% ; :44% ; :32% (b) Node failure probability when network connectivity probability is p 0 = 99%. Table 1 provides us with the bounds that are mathematically proven when we consider 3-D mesh network connectivity. For example, from Table 1(a), we conclude that as long as the node failure probability is bounded by 1%, 3-D mesh networks of up to 46,656 nodes (e.g., the 3-D mesh network M ) remain connected with a probability larger than 98:5%. On the other hand, from Table 1(b), we can see, for example, that if we are building a 3-D mesh network M , and want to keep a probability of network connectivity of at least 99%, we only need to bound the individual node failure probability under 0:89%. For further larger 3-D mesh networks, such as M of around one million nodes, the results given in Table 1 are not very satisfying. For example, when the node failure probability p is equal to 1%, Table 1(a) can only claim a connectivity probability of 74:71%. We indicate that the bounds given in Table 1(a) are only lower bounds, which are based on 3-submesh connectivity. Thus, a more thorough analysis may further improve the bounds presented in Table 1. In the following, we describe a method to achieve such an improvement. It is intuitive to see that for two integers k and k 0 with k < k 0, the probability of k 0 -submesh connectivity is at least as large as the probability of k-submesh connectivity. In fact, it is easy to see that for any integer k, if the 3-D mesh network M m n q is k-submesh connected, then M m n q is also 2k-submesh connected. Note that by Theorem 4, the k-submesh connectivity of M m n q for any k would imply the connectivity of the mesh M m n q. Therefore, it is reasonable to consider the probability of k-submesh connectivity for k > 3, in order to improve the results given in Table 1. However, there is a technical challenge for this approach. Consider 5-submesh connectivity. The 3-D mesh M has nodes. In order to compute the values N 5;i for all i, 1 6 i 6, we need to enumerate all 2 ways of removing a subset of nodes from the 3-D mesh M 5 5 5, which would not be very feasible even using a computer program. Therefore, we need to develop new analyses and computational techniques to overcome the difficulties. In the following, we take 5-submesh connectivity as an example and propose a new technique to solve the problem. We observe that when the node failure probability p is small, the probability that the 3-D mesh M contains many faulty nodes is also very small, thus can be neglected in our discussion. This observation is formulated in the following lemma. Lemma 8. Suppose that the node failure probability is p, then the probability that any 5-submesh in the 3-D mesh M m n q contains more than h faulty nodes is bounded by (mnq=) h+1 ph+1. Proof. A 5-submesh has nodes. Thus, the probability that the 5-submesh contains more than h faulty nodes is equal to X p i (1 p) i i i=h+1 i=h+1 From Theorem C.2 of [19], we get X p i (1 p) 6 p i h+1 i h +1 Now since the 3-D mesh M m n q contains mnq= 5-submeshes, the probability that any

6 188 J. Comput. Sci. & Technol., Mar. 2004, Vol.19, No.2 of these 5-submeshes contains more than h faulty nodes is bounded by (mnq=) h+1 ph+1. 2 Corollary 9. Suppose that the node failure probability is p, then the probability that any 5-submesh in the 3-D mesh M m n q contains more than 7 faulty nodes is bounded by (mnq=) p8. 8 Therefore, as long as the value (mnq=) 8 p 8 is very small, we can neglect the cases in which there are more than 7 faulty nodes but concentrate on the cases where there are at most 7 faulty nodes in a 5-submesh M 5. Note that this will significantly reduce the number of enumerated cases. We define the following event: Event [C7(M 5 )]: The 5-submesh M 5 has at most 7 faulty nodes, and all non-faulty nodes in M 5 are connected. Note that event C7(M 5 ) implies event C(M 5 ). In fact, since each face of the 5-submesh M 5 has 5 2 = 25 nodes, if the total number of faulty nodes is bounded by 7, then automatically each face of M 5 has at least 13 non-faulty nodes. Lemma 10. Suppose that the node failure probability is p, then the probability of the event C7(M 5 ) is Pr[p; C7(M 5 )] = 7X i=0 N 5;i (1 p) i p i where N 5;i is the same as defined in Lemma 5, i.e., N 5;i is the number of ways by which we remove i nodes from M and the remaining graph is connected. Proof. The proof is similar to that of Lemma 5. Note that here the condition that each face has more than 5 2 =2 nodes is not required, because the condition is implied by the fact that M 5 has no more than 7 faulty nodes. 2 Now the computation becomes feasible and we have written a computer program to compute all the values N 5;i for 0 6 i 6 7. The computational results are given as follows. N 5;0 = 1 N 5;1 = N 5;2 = 7750 N 5;3 = N 5;4 = N 5;5 = N 5;6 = N 5;7 = : Since Pr[p; C7(M 5 )] 6 Pr[p; C(M 5 )], combining the above numerical data with Theorem 6, we get the following theorem. Theorem 11. Suppose that node failure probability is p, then the connectivity probability of the mesh network M m n q is at least (Pr[p; C7(M 5 )]) (mnq)=, where Pr[p;C7(M 5 )] = 7X i=0 N 5;i (1 p) i p i =(1 p) + p(1 p) p 2 (1 p) p 3 (1 p) p 4 (1 p) p 5 (1 p) p 6 (1 p) (1 p) 118 6Pr[p; C(M 5 )] = X i=0 N 5;i (1 p) i p i As we did in Table 1, we apply Theorem 11 to 3-D mesh networks of different sizes. We again obtain two groups of results: one is under the condition that the node failure probability is p = 1% and we compute the corresponding connectivity probability for the 3-D mesh networks; the other is under the required connectivity probability p 0 = 99% for the networks, we compute the corresponding node failure probability p that meets the requirement of network connectivity probability. The results are given in Table 2. Table 2. Results Based on Pr[p; C7(M 5 )] 3-D mesh #nodes connectivity prob ; :99% ; :96% ; :87% ; :38% ; :92% ; 000; :14% (a) Network connectivity probability when node failure probability is p = 1%. 3-D mesh #nodes node failure prob ; :50% ; :87% ; :57% ; :15% ; :77% ; 000; :56% (b) Node failure probability when network connectivity probability is p 0 = 99%. Compare Table 1 with Table 2, we observe that the results in Table 2 provide us with more precise information. For example, when the node failure probability is 1%, a 3-D mesh network of one million nodes (i.e., M ) remains connected with a probability larger than 95%, instead of the

7 Gao-Cai Wang et al.: On Fault Tolerance of 3-Dimensional Mesh Networks 189 probability 74:71% claimed in Table 1(a). Moreover, the data in Table 2 now seem to be sufficiently good and useful for practical multicomputer system manufacturing. For example, suppose we are building a multicomputer system based on 3-D meshes of up to one million nodes (e.g., the 3-D mesh M ). From Table 2(b), we are guaranteed that the network connectivity probability is 99% if we can control the node failure probability under 0:56%. Note that according to today's VLSI technology, building network nodes with failure probability under 0:5% is achievable. Thus, Table 2 provides formally proven and achievable bounds on node failure probability, in terms of the currently available technology. This shows the feasibility of building highly fault tolerant 3-D mesh networks of very large size. 5 Comparing 3-D Mesh with Other Networks 3-D mesh networks seem to have a number of advantages over other popular network topologies. In this section, we compare 3-D mesh networks with 2-D mesh networks and hypercube networks. Based on similar techniques, lower bounds have been derived on the network connectivity for 2-D mesh networks [18]. We set node failure probability p = 0:1%, and use the formula in [18] to compute the network connectivity for 2-D meshes of various sizes. The results are given in Table 3 (compare it with the table in Fig.3 in [18]). Table 3. Results for 2-D Meshes 2-D mesh #nodes connectivity prob ; :8% ; :4% ; :5% ; :4% 1; 001 1; 001 1; 002; :9% Network connectivity probability when node failure probability is p = 0:1%. Consider 2-D meshes and 3-D meshes of similar size, and compare the data in Table 2 and Table 3, we can see a significant difference between the power of fault tolerance of 2-D meshes and that of 3-D meshes. For example, consider mesh networks of around one million nodes, then with node failure probability p = 1%, the 3-D mesh network M can maintain a connectivity probability larger than 95%. On the other hand, in order to maintain the similar connectivity probability, the 2-D mesh network M of similar size requires that the node failure probability be bounded by 0:1%, a much tougher condition on the robustness of network nodes. In fact, using Theorem 11, we can verify that for the node failure probability 0:1%, the connectivity probability ofthe3-d mesh M is larger than 99:99%. We conclude that compared with 2-D mesh networks, 3-D mesh networks seem to be much stronger in tolerating faulty nodes. This result matches the intuition. Moreover, our discussion and conclusion are based on more precise and quantitative calculation and analysis. Now we compare 3-D mesh networks with the popular hypercube networks. Based on the results in [16], we set a probability 99% for the network connectivity, and calculate the required node failure probability for hypercube networks of various sizes. The results are given in Table 4. Compared with 3-D mesh networks of similar sizes, we can see that hypercube networks are stronger in tolerating faulty nodes while maintaining non-faulty nodes connected. However, the results seem to be comparable. For example, for a network of around one million nodes, in order to achieve a network connectivity probability 99%, the results in [16] requires the hypercube network (i.e., the 20-dimensional hypercube H 20 ) to keep the node failure probability bounded by 2:49% (see Table 4), while the results of the current paper requires a 3-D mesh network (e.g., the M mesh) to keep the node failure probability bounded by 0:56% (see Table 2(b)). Our impression is that having the node failure probability bounded by 0:5% is already quite achievable by today's technology. Thus, allowing much higher node failure probability seems not to be among the most critical issues in the current development of multicomputer systems. On the other hand, the 3-D mesh networks enjoy a much lower node degree compared with the hypercube networks: the node degree of a 3-D mesh network is always bounded by 6, while the large size hypercube networks such as the 20-dimensional hypercube H 20 would have a much higher node degree in the network. Table 4. Results for Hypercube hypercube degree #nodes node failure prob ; :71% ; :33% ; :28% ; 048; :49% Node failure probability when network connectivity probability is p 0 = 99%.

8 190 J. Comput. Sci. & Technol., Mar. 2004, Vol.19, No.2 6 Conclusions The mesh network is a very popular interconnection topology for multicomputer systems. Many research and commercial multicomputer systems have been built based on mesh networks of many thousands of processors. For this kind of multicomputer systems with a large number of nodes, fault tolerance is among the most important and interesting issues. In this paper, we have studied the fault tolerance of 3-D mesh networks under the probabilistic model in which each node has an independent failure probability. Our research provides formally proven results that show the possibility of building highly fault tolerant 3-D mesh networks of very large size. To the authors' knowledge, these are the first group of results for 3-D mesh network fault tolerance obtained by formal and thorough mathematical proofs. References [1] Alverson R. The Tera computer system. In Proc. Int. Conf. Supercomputing, 1990, pp.1 6. [2] Cray T3D System Architecture Overview. Technical Report, Cray Research Inc. HR-04033, March, [3] Allen F, Almasi G, Andreoni W et al. Blue Gene: A vision for protein science using a petaflop supercomputer. IBM Systems Journal, 2001, 40: [4] Chuang P, Tzeng N. Allocating precise submesh in mesh-connected systems. IEEE Trans. Parallel and Distributed Systems, 1994, 5(2): [5] Liu T, Huang W, Lombardi F et al. A submesh allocation scheme for mesh-connected multiprocessor systems. In Proc. Int. Conf. Parallel Processing II, 1995, pp [6] Chang C, Mohapatra P. An efficient method for approximating submesh reliability of two-dimensional meshes. IEEE Trans. Parallel and Distributed Systems, 1998, 9(11): [7] Yoo B, Das C. A fast and efficient processor allocation scheme for mesh-connected multicomputers. IEEE Trans. Computers, 2002, 51(1): [8] Almohammand B F A, Bose Bella. Fault-tolerant communication algorithms in toroidal networks. IEEE Trans. Parallel and Distributed Systems, 1999, 10(10): [9] Cang S, Wu J. Time-step optimal broadcasting in 3- D meshes with minimum total communication distance. Journal of Parallel and Distributed Computing, 2000, 60: [10] Wu J. A simple fault-tolerant adaptive and minimal routing approach in 3-D meshes. Journal of Computer Science and Technology, 2003, 18(1): [11] Boppana R, Chalasani S. Fault-tolerant wormhole routing algorithms for mesh networks. IEEE Trans. Computers, 1995, 44(7): [12] Chen C, Chiu G. A fault-tolerant routing scheme for meshes with nonconvex faults. IEEE Trans. Parallel and Distributed Systems, 2001, 12(5): [13] Kim S, Han T. Fault-tolerant wormhole routing in mesh with overlapped solid fault regions. Parallel Computing, 1997, 23: [14] Wu J, Chen X. Fault-tolerant tree-based multicasting in mesh multicomputers. Journal of Computer Science and Technology, 2001, 16(5): [15] Leighton F T. Introduction to Parallel Algorithms and Architectures: Arrays, Trees, Hypercubes. Morgan Kaufmann Publishers, San Mateo, CA, [16] Chen J, Kang I, Wang G. Hypercube network fault tolerance: A probabilistic approach. In Proc. Int. Conf. Parallel Processing (ICPP'2002), 2002, pp [17] Najjar W, Gaudiot J. Network resilience: A measure of network fault tolerance. IEEE Trans. Computers, 1990, 39(2): [18] Chen J, Wang T. Probabilistic analysis on mesh network fault tolerance. In Proc. 14th International Conference onparallel and Distributed Computing and Systems (PDCS'02), 2002, pp [19] Cormen T H, Leiserson C E, Rivest R L et al. Introduction to Algorithms. 2nd Ed., McGraw-Hill, Gao-Cai Wang received the M.S. degree in geographic information system from Central South University, China, in Currently, he is a Ph.D. candidate in the Department of Computer Science, College of Information Science and Engineering at Central South University. His research interests include computer networks, routing algorithms, computer fault tolerance. He has published more than 15 papers in these areas. Jian-Er Chen received the Ph.D. degree in computer science from the Courant Institute of Mathematical Science, New York University (NYU), in After graduation from NYU, he went to the Department of Mathematics at Columbia University, where he received the Ph.D. degree in mathematics in Since then, he has been with the Department of Computer Science at Texas A&M University, where he is currently a professor. He also holds a ChangJiang Scholar Professorship at Central South University, China. His research interests include computational complexity and optimization, graph theory and algorithms, parallel processing and networks, and computer graphics. He has published more than 100 papers in these areas. Guo-Jun Wang received the M.S. degree and Ph.D. degree in computer science from Central South University, China, in 1996 and 2002, respectively. Currently, he is an associate professor in the Department of Computer Science, College of Information Science and Engineering at Central South University. His research interests include computer networks, routing algorithms, computer fault tolerance, and software engineering. He has published more than 40 papers in these areas.