Fault Tolerance in Hypercubes

Transcription

1 Falt Tolerance in Hypercbes Shobana Balakrishnan, Füsn Özgüner, and Baback A. Izadi Department of Electrical Engineering, The Ohio State University, Colmbs, OH 40, USA Abstract: This paper describes different schemes for tolerating falts in hypercbe mltiprocessors. A stdy of hypercbe algorithms reveals that in many cases, the comptations that reqire local commnication are mapped onto topologies sch as meshes or rings and the hypercbe topology is sed for global data commnication. Therefore, a falty hypercbe needs to be reconfigred to perform both local and global commnication as reqired by the algorithm, effectively and with minimal performance degradation. Two general approaches can be identified. The first approach looks into ways of tilizing the healthy processors and links of a hypercbe with falty nodes/links, for embedding topologies sch as lower dimensional hypercbes, rings, meshes and trees for performing commnication. The second approach makes se of hardware redndancy in the form of spare nodes and/or links and sally reqires modifications in the commnication hardware. Agmented hypercbes and spare allocation schemes are described. Keywords: Hypercbe, falt tolerance, embeddings, global commnication, spare allocation, reconfigration Introdction The area of reconfigrability in the presence of falts is becoming increasingly important with the emergence of massively parallel architectres. Therefore, falt tolerance is an important isse that needs to be addressed in the design of node architectres, commnication hardware and software design as well as parallel algorithm development. A d dimensional hypercbe mlticompter consists of n = d processors (nodes) interconnected as a Boolean d-cbe, with each processor having only local memory. Inter processor commnication is done by explicit message passing. Each processor in a d cbe can be represented by a d tple (b d; b i b 0 ) where b i f0 g, and a sbcbe in a d-cbe can be represented by a d tple f0 xg d : Coordinate vales 0 and can be referred to as bond coordinates and x as free. A (d ; k) dimensional sbcbe ((d ; k) sbcbe) in a d cbe is represented by a d tple with k bond coordinates

2 and d-k free coordinates. Two general approaches can be identified for falt tolerance in hypercbe mltiprocessors. The first approach looks into ways of tilizing the healthy processors and links of a hypercbe with falty nodes/links, to identify embedded topologies sch as lower dimensional hypercbes (sbcbes), meshes, etc. reqired by the comptations. With this approach, some performance degradation is expected, however special hardware design for falt tolerance is not necessary. Embeddings are discssed in Section and an algorithm for finding falt-free sbcbes is described in Section. A distribted falt diagnosis algorithm for identifying the set of falty processors and links, when the nmber of falty processors and links r d is given in []. Once the falty elements are identified, the mltiprocessor and the distribted algorithm rnning on the mltiprocessor[] mst be reconfigred to rn on the available processors. A falty hypercbe needs to be reconfigred to perform both local and global commnication as reqired by the algorithm, effectively and with minimal performance degradation. A stdy of hypercbe algorithms reveals that in many cases, the comptations that reqire local commnication are mapped onto topologies sch as meshes or rings (grid problems for example []) and the hypercbe topology is sed for global data commnication. Therefore, both local and global commnication schemes mst be considered in designing algorithms for falty hypercbes. The problem of commnication in a falty hypercbe has been addressed by many researchers. A srvey of some of these schemes as well as new reslts will be presented in Section 4. The second approach to falt tolerance makes se of hardware redndancy in the form of spare nodes and/or links and sally reqires modifications in the commnication hardware. Althogh a d-dimensional hypercbe with falty nodes still contains a nmber of lower dimensional sbcbes and therefore can be sed to rn hypercbe algorithms withot any modification [4, 5, 6, 7], one falty node redces the dimension of the largest falt-free sbcbe to (d ; ) and falty nodes can redce it to (d ; ). This has motivated researchers to investigate hardware redndancy and spare allocation schemes [8, 9, 0, 7,,,, 4, 5]. Agmented hypercbes and spare allocation schemes will be described in Section 5. Embeddings Embedding problems are concerned with finding mappings between two graphs that preserve certain topological properties. Let G a = (V a E a ) be an application graph and G h = (V h E h ) be the host graph that represents the interconnection topology of the parallel compter. An embedding of a graph G a into a graph G h is a fnction from V a to V h [6]. is an isomorphic embedding iff G a G h and for all v V a d a ( v) = d h (() (v)) where d( v) is the distance between vertices and v. The dilation of is Max(d(() (v))) for all v V a. Note that an isomorphic embedding is a nit dilation embedding.

3 Previos research has shown that a nmber of sefl graphs can be embedded isomorphically into healthy hypercbes. A graph G is cbical if there exists an isomorphic embedding of G in the d-cbe graph. The focs of the research has been to determine the largest size of the graph G that can be embedded in a d-cbe. For example, in [7] and [8] it is shown that trees are cbical, and that a (d ; ) height tree can be embedded in a d-cbe isomorphically. In cases where adjacency cannot always be preserved, embeddings of dilation greater than one have been determined [9, 0,, ]. In this section, we present an overview of algorithms for embedding trees, rings and meshes in falty hypercbes. We discss embeddings that handle node failres only, nless otherwise stated. The direction of the research in falt tolerant embeddings has followed two different paths depending on the motivating problem. The first approach [9, 0] is to find embedding fnctions that primarily minimize the cost of reconfigration; i.e., the nmber of node state changes dring reconfigration is minimized. Generally, sch research has focsed on distribted algorithms so that the neighbors of the falty node can detect the falty node and locally decide how to remap the falty node to a healthy node with minimm reconfigration cost. Remapping in this way may not always preserve adjacency, ths increasing the commnication cost. Frthermore, the nmber of nodes tilized in the embedding is not always maximized. Sch algorithms also do not maximize the nmber of falts that can be tolerated and exist for single or doble falt cases only. The second direction taken by researchers [, 4] in falt tolerant embeddings has been to preserve the adjacency of nodes; i.e., prodce an isomorphic embedding. Hence, for nit dilation embeddings, the focs has been to determine the maximm size of the embedding so that fewest nmber of healthy nodes are wasted (nsed). Bonds on the maximm nmber of falts that can be tolerated are also obtained.. Rings In this section we review the approaches taken in [0] and []. We elaborate on the idea sed in [] since it handles mltiple falts and creates the falt tolerant embedding at rn time in a distribted manner. The key idea in [0] is to embed the ring so that nsed (healthy) nodes are close to all nodes in the embedding, whereby, an nsed node can replace a falty node in the embedding easily with a few reconfigration steps. Fig. (dark lines) illstrates a ring of length eight embedded in a 4-cbe. In general, the algorithm tilizes a small percentage of the nodes where the maximm is 50%. It can be seen from Fig. that sch an embedding can tolerate two falts and the reconfigration algorithm reqires two state changes i.e., nodes 00 and 000 are added in place of the falty nodes 0000 and 000 respectively. Embedding a ring in a d-cbe so that the size of the ring is maximized is the approach taken

4 Falty nodes Original Embedded Ring Dim Dim Spares sed in falt tolerant embedding Reconfigred Ring State Changes Dim0 Dim Figre : Ring embedding minimizing reconfigration cost by []. The athors devise a distribted algorithm that embeds a ring of size d ;f in a d-cbe where the nmber of falts f b d+ c. They se cyclic Gray codes to embed the ring as is done in healthy hypercbes [8] and vary this basis ring to skip over the falty nodes. Fig. shows a 4-cbe with the basis ring embedded (shown by tracing the nmber in the angle brackets i.e., <> starting at node 0000). The embedding dimension seqence for the basis ring in a 4-cbe denoted as DM 0 = f0 g is ( ) where following this dimension seqence from the sorce node reslts in the basis ring. Let s also denote the rank of the processor as the location of the processor in the embedded basis ring. The nmber in the angle bracket in Fig. represents the rank of the processor. For example, the sorce has rank, and for the embedding seqence f0 g the neighbor of the sorce along dimension 0 has rank while the neighbor of the sorce along dimension has rank 6. The first dimension in DM 0 is the alternating dimension (i.e., the dimension traversed most freqently), and is denoted as d f (d f =0in this example). When a falty node is encontered along the embedded ring, the algorithm in [] exectes one of the following two cases. In the first case, if the dimension along which the falty node is reached is the most freqently occrring dimension d f (i.e., dimension 0 in the example), then the dimension sb-seqence [d f d 0 d f ] is replaced by [d 0 ] where d 0 6= d f. For example, as in Fig. (a), if node 0 is falty, then at node 00 (which is the fifth processor in the ring) the next dimension along the embedding seqence is changed from 0 to i.e., the dimension sb-seqence [0 0] is replaced by []. In the second case, if the dimension along which the falty processor is reached is not dimension d f bt some other dimension d 0, then the dimension sb-seqence [d f d 0 d f ] is replaced by d 0, i.e., the ring

5 00 <5> 0 <6> 0 <> <> 000 <8> <7> <9> <0> <4> <> <> <4> <> <> <6> <5> 00 (a) 00 <5> 0 <6> 0 <> <> <8> 000 <7> <9> <0> <4> <> <> <4> <> <> <6> <5> 00 (b) Dim Dim Dim0 Dim Falt Tolerant Embedding Falty node Skipped node Figre : Ring embedding maximizing size of ring: (a) Falty processor reached along dimension 0 (b) Falty processor reached along dimension so far constrcted is backtracked along d f and the ring constrction is contined by the previos processor in the embedding. For example, as in Fig. (b), if processor 00 is falty (which is the processor accessed along dimension ), the algorithm backtracks to processor 00 and replaces the dimension sb-seqence [0 0] by [] with the next dimension pointer pointing to dimension. The algorithm is garanteed to find a ring embedding (where for each falty processor, a healthy processor is removed from the original embedding) provided that the nmber of falts does not exceed b d+ c. In the first case, when the falty processor is reached along d f, the falty processor has an even rank in the embedding seqence. The algorithm skips the falty processor and the processor after it in the falt tolerant embedding. In the second case, when the falty processor has an odd rank in the original embedding seqence, the algorithm skips the falty processor and the processor before it in the falt tolerant embedding. To tolerate mltiple falts the algorithm handles falt distribtions where () no two falts are distance two apart and () if two falts are distance one apart then the first falty processor encontered in the original basis

6 ring mst have an even rank. The athors fix the nmber of falts f, and permte the dimension seqence of the basis ring f ; times, by defining f ; embedding dimension seqences DM 0 DM ::: DM f ; where DM i is DM 0 with dimension 0 and dimension i swapped. The nmber of falts is fixed to not exceed b d+ c since the nmber of permtations cannot exceed d. It is shown that there exists at least one dimension seqence for which the basis ring satisfies the above two conditions and is garanteed to find a falt-tolerant embedding of size d ; f.. Meshes In a falty hypercbe, an m m mesh can be obtained by removing falty rows and/or colmns in a d d mesh, where d dlog m e, d dlog m e, and (d + d ) d, and finding a Gray code ordering of the m healthy rows and m healthy colmns [4]. An example of embedding a 5 5 mesh in a falty 6-cbe is illstrated in Fig.. An initial embedding of an 8 8 mesh shown in Fig. is done by generating a -D Gray code for the coordinates b 5 b 4 b and b b b 0. A 5 5 mesh is obtained by removing the falty rows and/or colmns in Fig. and then by rearranging the rows and colmns in Gray code ordering. For this example, the rows wold be arranged as [0, 00, 000, 00, 0] and the colmns as [00, 000, 00, 0, ]. However, the search for a Gray code ordering of healthy rows and colmns is comptationally complex. Pasting of the healthy rows/colmns for constrcting either side of the mesh is eqivalent to embedding a ring in a falty d -sbcbe or a falty d -sbcbe respectively and can be constrcted sing the ring embedding techniqes so that a falty node is skipped along with a node adjacent to it. In the worst case this reslts in two rows or colmns of nodes being skipped for every falt, reslting in an embedded mesh of size ( d ; x) ( d ; y), where x + y = f. However, if the hypercbe has Direct-Connect Capability [5] and assming that only processors can fail, then a linear array of any length m can be formed by first forming a linear array of length d in a d-cbe sing a Gray code seqence and then connecting throgh the Direct- Connect Modles (DCM s) of the falty nodes. An m m mesh can be formed similarly by pasting the healthy rows and colmns throgh the DCM s of falty rows/colmns (Fig. ). In roting messages, if the e-cbe roting algorithm [5, 6] is sed (dimension order roting starting with dimension 0), commnication from the nodes of sbcbe xxx00 (colmn 00) to the nodes of sbcbe xxx0 (colmn 0) will go throgh the DCM s of sbcbe xxx000 and sbcbe xxx00; on the other hand, commnication from the nodes of sbcbe xxx0 to the nodes of sbcbe xxx00 will go throgh the DCM s of sbcbe xxx and sbcbe xxx0. Ths e-cbe commnication is not sitable for two reasons: one is that roting is throgh a different set of processors in each direction. The more important reason is that a link connecting processors in the mesh (sch as xxx000 and xxx00) may be sed, which is also sed for local (nearest neighbor) commnication between those processors. In order to garantee reglar and ninterrpted local commnication, messages between two processors in

7 b b b 0 b 5 b 4 b : falty nodes : removed colmns and rows Figre : A 55 mesh embedded in a 6-cbe pasted rows/colmns mst go throgh the DCM s of the deleted rows/colmns. In the example shown in Fig., messages between the nodes of sbcbe xxx00 (colmn 00) and the nodes of sbcbe xxx0 (colmn 0) shold go throgh the DCM s of sbcbe xxx0 and sbcbe xxx00 in both directions. Ths the DCM mst be designed to rote messages in both directions in accordance with the coordinate seqence sed in defining each side of the mesh. In a specific Karnagh-map illstration of a d d mesh, the grop of nodes in a colmn or a row form a d -sbcbe or a d -sbcbe, and are represented by a niqe d-tplef0 xg d. In algorithm MESHDCM [4], we ignore all the free coordinates x and consider the bond coordinates only in deleting falty rows/colmns and ths the grop of nodes in a row or a colmn are identified by a niqe d -tple or a niqe d -tple respectively. The falty and healthy nodes of a row/colmn have the same representation when they are indicated by considering the bond coordinates only. Therefore, deleting any one falty node in a row/colmn is eqivalent to deleting the entire row/colmn. MESHDCM given below, first chooses d coordinates among the d coordinates to constrct the m side of the m m mesh, where d = dlog m e, and then constrcts the m side of the mesh by sing the remaining (d-d ) coordinates. Algorithm MESHDCM: Step : If (d ; d ) <d or ( d ; f ) < (m m ), then exit. Step : Calclate the nmber of extra rows N e = ( d ; m ), that can be deleted for forming the m side of the mesh.

8 Step : Choose d coordinates among the d coordinates to constrct the m side of the mesh. Then constrct the list F d by representing the f falty elements in the falt list F by the d coordinates only (j F d j d ), and deleting identical representations of the falty nodes in the d coordinates. Step 4: If j F d j>n e then constrct F m by choosing N e elements from the falt list F d. Else let N e =j F d j and F m = F d, and choose any m colmns and exit. The N e elements in F m correspond to the rows to be deleted represented in the d coordinates. Step 5: Constrct F m, the list for constrcting the m side, by deleting from F the falty nodes in the rows in F m, then represent the falty elements by the d coordinates only deleting identical representations. Calclate the nmber of healthy colmns N c = d ; j F m j. If N c <m, then go to Step 6, else exit. Step 6: Check if all the possible choices of N e nodes have been sed exhastively in Step 4. If yes, then go to Step 7, else go to Step 4 for the next trial. Step 7: Check if all the combinations of choosing d coordinates among the d coordinates are examined exhastively. If yes, then exit, else go to Step for the next trial. d The complexity of Algorithm MESHDCM is analyzed as follows. In the worst case, combinations of coordinates will be tried to constrct the m side of the mesh and for each jf choice, there are d j N e ways of deleting rows. Step has a complexity of flog f and Step 5 has a complexity of (fn e + flog f ). Therefore, the time complexity is O( d j jfd Ne d d [flog f + (flog f + fn e )]). Iff < d, (N e < d) a pessimistic worst case complexity of O(P d ) can be derived since d d <P, and jf d j Ne <P, where P = d. An example of embedding a mesh in a 4-cbe with falty nodes (f =000, f =00, and f =00) by Algorithm MESHDCM is shown in Fig. 4. Since d=4, m =, and m =, (d ;dlog m e)=, dlog m e =, and ( d ; f )=> (m m ) = 9, the mesh embedding is possible. Then we have N e = ( d 4 -m )=. Since d = dlog m e =, there are ways of choosing coordinates among 4 coordinates. As shown in Fig. 4, b b is selected first and the falty nodes represented in b b (elements of F d ) are F =00, F =0, F =. In trial, F is deleted and F m is f0 00g. Since N c =< m, trial is reqired. In trial, F is deleted and F m is f0 00g. Since N c =< m, trial is reqired. In trial, F is deleted, F m is fg, F m is f0g and N c ==m.. Binary Trees Another topology that is often sed in algorithms is a binary tree. Two significant reslts regarding tree embeddings have emerged. It was shown in [] that a tree of height d, T d can be

9 Illstration of Trial : b b b b 0 Delete mmmm f mmmm f fmmmm mmmm - b b 0 b b 0 mmmm f fmmmm 0 mmmm 6 6 Delete healthy colmns < Illstration of Trial : b b b b 0 00 Delete mmmm f mmmm f fmmmm mmmm - b b 0 b b 00 mmmm f fmmmm 0 mmmm 6 6 Delete healthy colmns < Illstration of Trial : b b b b Delete mmmm f mmmm f fmmmm mmmm - b b 0 b b 00 mmmm f 0 mmmm f 0 mmmm 6 Delete - b b b b mmm mmm mmm Figre 4: Illstration of a x mesh embedding sing algorithm MESHDCM

10 0 Root 0 L 00 R L R Root (a) (b) Tree Nodes Intermediate Nodes L Left child R Right child Falty Node Figre 5: Tree embedding maximizing the size of the embedding (a) T with dilation - basis embedding (b) T with dilation - falt-tolerant embedding embedded in a d-cbe with maximm dilation, i.e., the adjacency of nodes in a tree are not preserved. Here, the motivation is to tilize the maximm nmber of nodes ths trading off the increased dilation for a maximm embedding size. The other approach taken in [7, 8] was to show that a binary tree of height d ;, T d;, can be embedded in a d-cbe with nit dilation. Here the trade-offs are jst the reverse of the previos approach, i.e., preserving adjacency reslts in not maximizing the nmber of nodes sed. Falt tolerant embeddings have been considered sing one of the above two approaches as the basis embedding. The factors considered in the falt tolerant embedding are: () the nmber of falts tolerated, () the dilation of the falt tolerant embedding, () the reconfigration cost i.e., the nmber of nodes in the new embedded graph that do not have the same state as in the basis embedding and (4) the size of the embedding. In [] the basis embedding is T d where the maximm dilation is as shown in Fig. 5 (a) by the dark lines. The approach as seen in Fig. 5 (b) is to se the intermediate node i.e., node (in the mlti hop edge of the tree) to replace a falty node (node 00) and tilize another tree node (node 00) in the basis embedding as an intermediate node as well as a tree node in the falt-tolerant embedding. Here the objective is to minimize the dilation of the falt tolerant embedding (i.e., keep it constant at ) and not degrade the size of the embedding (i.e., embed T d ). Only one falt is tolerated by this scheme. Note also that the new falt tolerant embedding reslts in the states of all the nodes being changed as shown in Fig. 5 (b). For example, the root of the tree is mapped to node 0 instead of 0, the left child to instead of 00, the right

11 child to 000 (via 00) instead of 0 (via ) and so on. Improvements to this reslt have been made in [0]. Here, a falt tolerant embedding with T d; as the basis embedding is sed so that the dilation is minimized (i.e., kept constant at ) and falts are handled. This approach has the same penalty as before i.e., a reconfigration cost of d; ;. Frthermore, the height of the embedding is d ; de to the approach taken for the basis embedding. The other approach taken is to minimize the reconfigration cost [7, ]. Here the basis embedding has nit dilation bt the falt tolerant embedding has a dilation of which is the penalty paid to obtain a reconfigration cost of. In [] the reconfigration algorithm involves two state changes if the falty node is at height in the basis embedding. Both these algorithms se a mirror image of the falty node along some dimension as the spare to replace the falty node. The athors in [] se a distribted algorithm where the falty node is remapped and only its parent and children are aware of the change. The athors also extend their algorithm to tolerate mltiple falts. Here they se a form of backtracking where if a node cannot embed its children, it declares itself falty to its parent who mst handle the reconfigration. In the worst case, the algorithm will backtrack p to the root and the embedding will fail. The reslts discssed so far have not considered maximizing the nmber of falts that can be tolerated. In [8] the athors show that in a d-cbe with d ; ;dlog de node and/or link falts, there exists a d ; tree which avoids the falts. Then the falt avoiding (d ; ) tree is constrcted in two steps. First, a falt-free tree T 0 is constrcted in which each node has exactly two or no children and the leaves of T 0 are at level d ; k or level d ; k ;, where the level of a node is the height of the node from the root and k = dlog de +. Second, falt avoiding (k ; ) trees (i.e., of height k ; ) are constrcted to attach onto the leaves of T 0 at level d ; k and falt avoiding (k) trees to attach onto the leaves at level d ; k ;. It is shown that a k-sbcbe can be associated with the leaves at level d ; k anda(k +)-sbcbe with the leaves at level d; k ;. These sbcbes which partition the d-cbe have the following properties. Exactly one of the leaves of T 0 and at most two internal nodes of T 0 belong to each of the k;cbes and each of the k-cbes has at most one falt. Exactly one of the leaves of T 0 and at most one internal node of T 0 belong to each of the (k +);cbes and each of the (k +)-cbes has at most two falts. Within sch sbcbes falt avoiding (k ; )-trees and k-trees can be constrcted. Finding Falt-Free Sbcbes Parallel algorithms for hypercbes can be formlated with the dimension d of the hypercbe as a parameter of the algorithm[5] and therefore can be rn on a falt-free sbcbe of the falty hypercbe. A simple distribted procedre to find the maximm dimension m d of a falt-free sbcbe is given in [5]. However, as indicated in [5], this procedre does not always find m d and frthermore, it does not constrct the set of falt free m d sbcbes. An algorithm which always finds m d and also the complete set of falt-free m d -sbcbes is presented in [4]. The

12 reslts are briefly explained below. The algorithm is formlated to rn on a single processor which wold typically k be the host or the resorce manager in a commercial hypercbe system. d Note that there are k different (d ; k) sbcbes and a total nmber of P d i d i= = d ; i different sbcbes in a d hypercbe. A commtative and associative intersection operation Id = fig d on the falty processors is defined as follows where I denotes the intersection of individal coordinates; 0I0 =0, I =, 0I = q, and qi0 = qi = qiq = q. The bond coordinates ( 0 s and s) in the intersection of all falty processors indicate the fixed coordinates among them. Hence, the nmber of fixed coordinates, n f, gives the nmber of (d ; )-dimensional falt-free sbcbes. Finding (d ; k) dimensional falt-free sbcbes for k > is more complex and will be described later in this section. The algorithm given in [4] is based on sing the Inclsion-Exclsion[9] principle to cont the nmber of sbcbes of a given dimension (d ; k), that can be formed in the presence of falty processors and links. d k. Lemma [4]: The nmber of (d ; k) sbcbes destroyed by a falty processor is D d;k = Let S d;k F i denote the set of (d ; k) sbcbes that falty processor F i belongs to. Sbcbes jointly destroyed by ` falty processors F i F i ::: F i` or sbcbes in S d;k F i \ S d;k F i \S d;k F i` can be conted withot explicit enmeration of sbcbes in each set by sing the following lemma. Lemma [4]: The nmber of (d ; k) dimensional sbcbes jointly destroyed by ` falty processors F i F i ::: F i` is K d;k (F i F i ::: F i`) = n f () k where n f is the nmber of fixed coordinates in I i i = F i` i IdF i Id IdF i`. For example, in a 5-cbe, if the falty processors are F = (000) and F = (0), then I = F IdF =(0qq). Fork =, d ; k =the sbcbes (xxx), (0xxx) and (0xxx) are destroyed by both F and F. Note that F and F each destroy ten -cbes. In conting the nmber of distinct (d;k) sbcbes destroyed by falty processors F F :::: F r, the common (d ; k) sbcbes destroyed shold not be inclded more than once in the cont and shold be exclded by sing the Inclsion-Exclsion[9] principle of conting given below. Theorem [9]: Principle of Inclsion-Exclsion. If N is the nmber of elements in a set S, the nmber of elements of S not having any of the properties p p ::::: p r is given as: N(p 0 p0 p0 r ) = N ; rx i= N(p i )+ X i6=j! N(p i p j ) ; X i j k N(p i p j p k ) + +(;) r N(p p p r ) () Here, N(p i ) denotes the nmber of objects having property p i, N(p i p j ) denotes the nmber of objects having both properties p i and p j, and so on. Objects having the same property p i

13 are elements of the set S i. The second smmation on the right hand side of the eqation is over all pairs of sets S i S j (i 6= j) and therefore enmerates the nmber of elements in pairwise intersections of sets S i and S j, the third smmation is over triples S i S j S k and finally the last term enmerates the nmber of elements in the intersection of sets S S r. This principle can be applied to conting the nmber of falt-free or available sbcbes of a given dimension (d ; k), G d;k, as described by the following theorem. Theorem [4]: The nmber of falt-free (available) (d;k) sbcbes, G d;k, in the presence of r falty processors F F ::::F r is G d;k = k d k! ; r d k! + X i6=j K d;k (F i F j ) ; X i j k K d;k (F i F j F k ) + +(;) r K d;k (F F F r ) () The steps of the algorithm for finding m d are given as:. Constrct the intersection I r and compte n f for this intersection. If n f then exit with m d = d ; and G d; = n f.. Constrct all pairwise intersections I i i and compte n f for these intersections.. Constrct intersections (I i i`; )I df`, < ` < r, for which n f (I i i`; ) and n f (I ij i`) for all i j fi i ::: i`; g, and compte n f for these intersections. 4. for k = :::do (a) compte K d;k (F i F i F i`) from eqation(), for <`<r. (b) compte G d;k from eqation. (c) exit the loop if G d;k 6=0with m d = d ; k and G md = G d;k. The complexity of the algorithm is O(k r ) which occrs only when all of the intersections have to be constrcted and n f k for each intersection. With the assmption that r d[], the complexity can be expressed as O(k d ). Once m d = d ; k and G md are fond, an m d -dimensional sbcbe can be fond as follows. There are no (d ; k) dimensional falt-free sbcbes, if all the k combinations of f0 g k d are exhastively covered at each k different k-coordinate position combinations by the set of falty processors. If any one of the k combinations of f0 g k d, at any one of the k- k coordinate position combinations, is fond to be missing in the list of falty processors, then the d tple constrcted by assigning that missing k tple combination to those particlar k coordinate positions and also assigning the remaining d ; k coordinates as free ( x ), defines a (d ; k) dimensional falt-free sbcbe. The search for missing combination(s) for each falty

14 element can be avoided by encoding each k tple at a k-coordinate position combination of falty elements with a -ot-of- k code and then simply sing the logic OR operation on these encoded k tples. Consider the following example: b b b b 0 F : F : F : 0 0 F 4 : 0 0 Eqation is sed to find the nmber of falt-free -cbes as follows: For k= (d-k=); I =(0q00) n f =, I =(0qq0) n f =, I = (0q0) n f =, I 4 =(q00q) n f =, I 4 =(qq0q) n f =, I 4 =(qqqq) n f =0, I =(0qq0) n f =, I 4 =(qq0q) n f =, I 4 = I 4 = I 4 =(qqqq) n f =0, Pi6=j K d;k (F i F j )= Pi j k K d;k (F i F j F k )= G = 4 ; = + + =8 +8; =7. Since G 6=0, the maximm dimension m 4 =. b b 0 = is one of the missing combinations and therefore the -cbe xx is fond as one of the falt-free -cbes. (rk + k ) d k, The pper bond for the complexity of finding the missing combination is O when the nmber of falts r is not restricted. A simpler bond is derived in [4] for r d. 4 Commnication in Falty Hypercbes 4. Global Sm/Global Broadcast Algorithms for Healthy Hypercbes Efficient commnication algorithms [0, ] have been devised for healthy hypercbes. The standard algorithm for broadcasting a message to all nodes from a single sorce was presented in [6]. This algorithm embeds a Spanning Binomial Tree (SBT) in the hypercbe. The Global Sm (GS) and Global Broadcast (GB) algorithms are the dals of each other. In the GS operation, data residing in the nodes of a d-cbe is collected in a Final Collecting Node (FCN). The GS algorithm reqires each node to know the address (c d; ::::c 0 ) of the FCN. For each step i, (i = d ; ::: 0), all nodes with address (c d; ::::c i x i ) receive accmlated

15 Dim Dim C Final Collecting Node Dim0 S Initial Sorce Node C S (a) (b) 00 <00> 0 <000> <> 00 S 0 <000> <00> 00 0 <000> 000 <0> (c) 00 <000> Figre 6: Commnication algorithms for a healthy -cbe (a) GS with dimension seqence ( 0) (b) GB with dimension seqence (0 ) (c) GB initiated by ISN data from nodes with address (c d; ::::c i x i ) along dimension i. Hence the data residing in the d-cbe is collected in an i;sbcbe at the end of each step. Here, powers of fx 0 g refer to concatenation of fx 0 g and x 0 is the empty string. The commnication steps are shown in Fig. 6 (a) for a commnication dimension seqence ( 0) where the FCN is 00. Note that all nodes reqire the address of the FCN to determine whether to send or receive the data, or to not participate at each step. In the GB operation, data residing in an Initial Sorce Node (ISN) is distribted to all nodes in the d-cbe. Each node reqires to know the address (s d; ::::s 0 ) of the ISN. For each step i, (i = 0 :::: d ; ), all nodes with address (s d; ::::s i x i ) send data to nodes with address (s d; ::::s i x i ) along dimension i. Data resides in an (i +);sbcbe at the end of i steps. The commnication steps are shown in Fig. 6 (b) for a commnication dimension seqence (0 ) where the ISN is 00. The GB operation is initiated by an ISN. If a d-bit control word is sent by a sorce node to a receiving node indicating how the receiver shold contine broadcasting the message, then nodes other than the ISN do not need to know the address of the sorce. If an intermediate node receives a message with the i th bit of the control word set to (scanning from left), then

16 0 <> 0 <00> 00 <000> 0 <00> 00 <> 0 <> 00 <000> 0 <0> 00 <000> 4 0 <00> 000 (a) 00 <00> 000 (b) 00 <0> Dim Dim Dim0 Falty Node Unsafe Node Active Node Figre 7: Global broadcast in a falty hypercbe from (a) active ISN (b) nsafe ISN it sends a copy of the message along dimension i. Each node repeats this step as many times as the nmber of s in the control word. Fig. 6 (c) illstrates this algorithm where the word in the angle brackets (<>) is the control word received by the node. The ISN is 00 and the commnication dimension seqence is ( 0). 4. Falt-Tolerant Global Broadcast Algorithms The motivation in falt tolerant commnication algorithm design has been to devise schemes that take the same nmber of commnication steps as the algorithms for healthy hypercbes since an increase in the nmber of commnication steps is a heavy penalty to pay in message passing systems. Several researchers [, 7] have investigated GS/GB algorithms for falty hypercbes. In [] the GB is performed by constrcting a family of d link-disjoint (a link is nidirectional and an edge consists of two directed links) spanning trees of height d rooted at the sorce node. This algorithm tolerates d d e; edge falts and takes d commnication steps. Broadcasting in falty hypercbes sing a d-bit control word was considered in [7], where an nsafe node is defined as one that has at least two falty or nsafe neighbors. A node that is not nsafe is defined as an active node. An algorithm that takes O(d ) commnication steps is sed to determine the nsafe nodes in a d-cbe. The athors show that the GB algorithm takes d commnication steps if the sorce node is not nsafe or falty (i.e., active) and d +steps if the sorce is nsafe provided that the nmber of node falts does not exceed d d e. At each step of the broadcast an active node is assigned to broadcast in a sbcbe, the sbcbe size indicated by the nmber of s in the control word received by the active node. The control word is scanned from left to right. The algorithm sends a message from an active node to an active neighbor

17 along dimension i, if the corresponding bit i of the control word is ; it sends a message to an nsafe neighbor along dimension i only if bit i of the control word is, and all other bits are 0 s. In other words, it broadcasts to all nsafe nodes only in the d th step of the GB algorithm. Fig. 7 (a) illstrates a -cbe with two falts performing the GB from an active ISN in steps. The nmber in the sqare denotes the commnication step nmber and the seqence in the angle brackets (<>) is the control word received by the node. The ISN generates a control word of < > and first sends along dimension with the control word altered to < 0 > to reflect the sbcbes xx and 0xx in which nodes and 0 mst contine to broadcast. If the ISN is an nsafe node, then the ISN sends the message to the first active neighbor obtained by scanning the control word from left to right. This active node acts as a new ISN and performs the GB as before except that no message is sent back to the original nsafe ISN. Fig. 7 (b) illstrates the cbe performing the GB from an nsafe ISN i.e., node 00. Note that in step 4 node 0 does not send the message to 00 since 00 is the original ISN. 4. Global Sm for Falty Hypercbes In Section 4. a d-step algorithm for the GS operation in healthy hypercbes was described. The GS [4] can also be collected by performing a partial Global Sm operation in i disjoint (d ; i)-sbcbes in parallel, collecting the partial global sm in nodes that form an i-sbcbe. For example, in a 4-cbe, for i =, splitting the 4-cbe along dimension, reslts in two disjoint -cbes 0xxx and xxx. Each sbcbe can perform a partial GS in steps collecting in nodes 0000 and 000 respectively, which are the nodes of the -cbe x000. Then, these i = nodes of the -cbe x000 perform a GS operation in step. Note that we can split the cbe into disjoint sbcbes of different sizes and yet apply the same principle. For example, the -cbe xxx obtained earlier can be split along dimension into two -cbes 0xx and xx. Now, the partial GS can be performed in steps in the three sbcbes S = 0xxx, S =xx, and S = 0xx in parallel, with nodes (0000 and 000 S ), node (00 S ) and node (000 S ) collecting the partial reslts. These are nodes of the -cbe xx00 and can perform agsinsteps. Note that in general, the dimension seqence of the partial GS can be different for the disjoint cbes. This techniqe can be sed in falty hypercbes where the d-cbe is split into good and falty partners as explained next. Consider the 4-cbe shown in Fig. 8(a) with falty nodes at f = 000, f = 00, and f =. The idea is to try to split the 4-cbe into two -cbes, one healthy and the other containing all falty nodes. If sch a split can be fond, then the healthy nodes in the falty cbe will send to the adjacent nodes in the good cbe in one step and the GS will be performed in the good cbe in d ; commnication steps. A (d ; );cbe containing all of the falty nodes can be fond by applying the intersection operation Id = fig d defined in Section (0I0 =0 I = 0I =q qi0 =qi =qiq = q). The bond coordinates in the intersection

18 Dim Dim Dim0 Dim Healthy nodes Falty nodes Collecting Cbe 0x0x (a) Collecting Cbe 0xx 0xxx xxx 0xx F 0x0x 0xx G G (b) xx F Figre 8: (a) Performing 4 step GS in a falty 4-cbe (b) Split tree for the GS operation shown in (a) of all falty processors indicate the fixed coordinates among them, and can be sed to split the cbe. For the 4-cbe considered, I = f If If = qqqq, and therefore a good-falty sbcbe split cannot be fond. Hence the 4-cbe is split along any dimension into two falty -cbes. Fig. 8(b) shows the first split along dimension reslting in two -cbes xxx and 0xxx where f and f are contained in xxx and f is contained in 0xxx. Since 0xxx contains only one falt, we can choose any dimension, for example dimension, to split it into goodfalty partners 0x0x and 0xx respectively. xxx has to be split again and dimension which is a bond coordinate of the intersection (I = qq ) is chosen. This reslts in good-falty partners 0xx and xx respectively. Now, the healthy nodes in the falty sbcbes can send their data to their partner in the good sbcbes in parallel in step. The partial GS s for the sbcbes 0x0x and 0xx can be performed in parallel in steps. In order to collect the reslts of the partial GS s in one node in the 4th step, the nodes collecting the partial GS s in each - cbe mst be adjacent, i.e., the collecting nodes containing the reslt of the partial GS s mst be in the same -cbe, so that the final GS can be collected in step. Sch collecting nodes can be determined by performing a consistency operation on the healthy (collecting) sbcbes fond in the last split. For the two collecting cbes 0xx and 0x0x in Fig. 8(a), the collecting nodes can be fond by performing a consistency operation & d = f&g d defined as follows: 0&0=0&x=0,

19 0xxx xxx 00xx F 0xx G 0xx G xx F Figre 9: Other possible way to split the falty 4-cbe &=&x=, 0&= &0= &= &x=, x&x=x. The candidate nodes for each collecting cbe are obtained by replacing the ; coordinates of the reslt with the corresponding coordinate vales in the collecting cbe. For this example, 0x0x&0xx = ;00x. The candidate collecting nodes (000x 0x0x) and (00x 0xx) form a -cbe called the Final Collecting Cbe (FCC). Fig. 8(b) shows the splitting operation. The split tree is a binary tree where each node represents a split and has associated with it a dimension d i sed at that split to partition the cbe. Note that the split tree does not inclde the edges of the tree corresponding to the last splitting operation to obtain the good-falty partners (shown in dotted lines) since these edges do not have a node at either end. A leaf of the split tree is a node of degree, while an internal node is a node of degree (root) or. Now consider the same example bt with dimension chosen to split both sbcbes 0xxx and xxx into good-falty partners. The split tree in Fig. 9 shows that the 4-cbe can be split reslting in two good -cbes 0xx and 0xx, bt partial collecting nodes that are adjacent cannot be fond. 4.4 d-step Falt-Tolerant GS/GB Algorithm By sing a niqe dimension at each split, the GS/GB operation can always be performed in d steps if the nmber of falts is d d e. Having split the cbe, the data from each healthy node in the falty sbcbe is sent to its partner in the good sbcbe along a dimension i, where i is the dimension sed at the final split to obtain a good-falty pair. This is done in one step. Now all the data resides in the good sbcbes. The following Depth First Search (DFS) operation on the split tree yields the dimension set D c of the FCC. Initially all the nodes of the tree are ncolored and D c is empty. When a node is visited, if it is not colored, and it is not a leaf node, then it is colored and the dimension associated with that node is added to set D c. The dimension of the FCC, which is the nmber of internal nodes in the split tree is denoted as k where k = j D c j. Note that the dimension set of the FCC can also be obtained by the & operator applied to the good sbcbes. The data in each good sbcbe is collected in parallel by a partial GS performed in d ; k ; steps. Let the set of free coordinates of the collecting good sbcbes be denoted by D gi where i is the i th good sbcbe. The dimension seqence along which to perform the

20 partial GS operation in the i th good sbcbe is determined by deleting from D gi the dimensions in the intersection of D gi and D c. By constrction we can show that the collecting node(s) in the good sbcbes form a k dimensional Final Collecting Cbe (FCC). The GS operation can then, be performed on the FCC in k steps to collect the data in one node. Hence, the entire operation takes d steps. The formal proofs can be fond in []. The following example illstrates the methodology sed to determine the FCN and the dimension seqence along which to perform the first commnication step from falty to good sbcbes, and thereafter the partial GS in each good sbcbe. Fig. 0 shows a 5-cbe with falts f at 0000, f at 0000, and f at 000. Fig. shows the split tree for a chosen split. The FCN that collects the reslt of the GS operation is determined from the constrcted split tree. Each node (circle) represents a sbcbe and the nmber inside the circle represents the dimension along which that cbe is split. First the cbe is split along dimension 4, giving rise to two falty cbes xxxx and 0xxxx. Next, the first falty sbcbe (xxxx) is split along dimension and the other falty sbcbe (0xxxx) along dimension. The first split gives rise to a good-falty pair where the good sbcbe is denoted by G = 0xxx. The second falty sbcbe is split into two falty sbcbes 0xxx and 0x0xx. Finally, the falty sbcbes 0xxx and 0x0xx are split along dimensions and 0 respectively to give good sbcbes G = 0xx and G = 0x0x respectively. Note that this is only one of the d! ways to split the 5-cbe into good-falty partners where a niqe dimension is chosen for each split. Using the DFS algorithm on the split tree, fd c g = f4 g. The dimension of the FCC, is k = j D c j =. The FCC is given by x0x where the fixed dimensions correspond to the dimensions that reslt in the good-falty splits, and the free dimensions correspond to the internal nodes of the split tree. Any node contained in the FCC can be chosen as the FCN. Fig. 0 shows the GS operation for the 5-cbe with the FCN being 00. The nmber i in the sqare denotes the commnication step i involved in the operation. In the first step, each node in the good sbcbes receives data from its healthy partner in the falty sbcbes along the dimension sed to perform the final split. Nodes in G receive from their partners along dimension, while nodes in G and G receive from their partners along dimensions and 0 respectively. This takes one commnication step. The partial GS in each good sbcbe is performed in d ; k ; steps i.e., in this case. The dimension seqence D i along which to perform the partial GS operation in the i th good sbcbe mst satisfy the condition D i \D c =. Hence, in G the GS is performed along dimensions 0 and, ing along 0 and, and in G along and. The action to be taken by a node i.e., send, receive, or not participate is determined as in the healthy GS algorithm sing the address(es) of the collecting node(s) in each sbcbe. The FCC is x0x. The collecting nodes in G are 00 and 0, the collecting node in G is 00, and the collecting node in

21 Dim Dim Falty nodes L Local collecting node in G Dim0 Dim Healthy nodes L Local collecting node in G L Local collecting node in G Dim4 000 Collecting Cbe G 0xx FCN L L Collecting Cbe G 0x0x 00 L L Collecting Cbe G 0xxx Figre 0: Hypercbe of dimension 5 with three falty nodes performing GS operation

22 4 xxxx 0xxxx F xxx 0xxx 0x0xx G 0 0xxx G G F G F 0xx 0x0x 0x0x 0x0x0 G G Figre : Split tree for the falty 5-cbe in Fig.0 G is 000. After the partial collection in the collecting nodes, the GS operation is performed on the FCC in steps collecting the data in the FCN. The total nmber of steps taken is 5 ( step for sending from falty to good sbcbes +steps to perform the local partial GS in each good sbcbe +steps to perform the GS in the FCC). 5 Falt-Tolerant Hypercbes As mentioned in Section, falty nodes can sbstantially redce the dimension of the largest falt-free available sbcbe. To preserve the dimensionality of the falty hypercbe, hardware redndancy and spare allocation schemes [8, 9, 0, 7,,,, 4, 5] have been investigated. A hardware scheme was first proposed by Rennels [8]. Here, N = d processors are groped into S = s clsters of M = m processors each, where d = m + s and one spare is assigned per clster. Crossbar switches are employed to add spare nodes via the additional port in dimension (d +)(Fig. ). If a processor fails in a clster, the spare replacing it mst be able to commnicate in d dimensions. Therefore, it has to be linked to the m neighboring processors in that clster as well as one processor in each of the s external sbcbes to which the failed processor is connected. This is accomplished sing two crossbars, namely the CCB (Connection Crossbar) and the RCB (Relay Crossbar). The CCB allows a spare to be connected to each of the m processors in its designated clster via their spare port. Using the RCB, the spare is linked to each of the other s clsters to which the host clster is connected. This approach can tolerate only a single falty node per clster with a significant overhead. It does not tolerate any link failres. For better falt coverage, spare boards consisting of one spare processor and S crossbars are sggested. Each spare node then can replace any falty node directly. However, the degree of the spare node wold effectively be eqal to the size of the hypercbe. Frthermore, additional crossbars to interconnect the spares are needed. Several approaches are proposed by Banerjee et. al. [0, 7, ] to perform reconfigration sing spare nodes and spare links. In [] two spares per -cbe are assigned (Fig. ) and the spare processors form a (d ; )-cbe. Upon a node failre, the falty processor is replaced with the local spare. The link connecting the spare to the falty processor and the link connecting

23 6 PROCESSOR SUBCUBE (ONE OF EIGHT) FROM RCBs IN OTHER SUBCUBES x 4 TO CCBs IN OTHER SUBCUBES RCB x 4 6 x 4 7 x 4 CCB FROM PROCESSORS IN SUBCUBE SPARE COMPUTER Figre : Rennels scheme for assigning a spare to a clster Spare Node Reglar Node Figre : Agmented for-dimensional hypercbe topology proposed by Banerjee it to the processor diagonally opposite to the falty processor are disabled. All other links are enabled. The spare node (S) which replaces the falty one sends its address and the address of the falty node to all the spare nodes connected to S. The fnction of each of the other spares is now that of a simple switch. Conseqently each spare determines the node to which it needs to be connected by sing the bitwise XOR of its own address, the address of the spare S, and the address of the falty node. As an example let s consider Fig. when node 000 is falty. The reconfigration steps are as follows. Replace processor 000 with the spare processor 00. The links between the spare 00 and nodes 000 and 0 are disabled. All other links are enabled. The spare node 00 sends its address as well as the address of the falty node (000) to the its neighboring spare nodes (000 and 000). The spare nodes 000 and 000 calclate the address of the nodes they shold be connected to as: = 0000 and = The spare nodes 000 and 000 then act as switches. This system can only tolerate a single node failre. In [0] Banerjee proposes two schemes to tolerate falty processors. The first scheme ses two sets of nodes; P-nodes and S-nodes. P-nodes are the reglar nodes of the hypercbe. An S-node consists of a spare attached to a P-node. S-nodes are allocated sch that every reglar node is at a Hamming distance of one from a spare node. For example in a -cbe, spares can