Node-covering, Error-correcting Codes and Multiprocessors with Very High Average Fault Tolerance

Transcription

1 IEEE TRANSACTIONS ON COMPUTERS, VOL. 46, NO. 9, SEPTEMBER Node-covering, Error-correcting Codes nd Multiprocessors with Very High Averge Fult Tolernce Shntnu Dutt, Member, IEEE, nd Nihr R. Mhptr, Member, IEEE Abstrct Structurl fult tolernce (SFT) is the bility of multiprocessor to reconfigure round fulty processors or links in order to preserve its originl processor interconnection structure. In this pper, we focus on the design of SFT multiprocessors tht hve low switch nd link overheds, but cn tolerte very lrge number of processor fults on the verge. Most previous work hs concentrted on deterministic k-fult-tolernt (k-ft) designs in which exctly k spre processors nd some spre switches nd links re dded to construct multiprocessors tht cn tolerte ny k processor fults. However, fter k fults re reconfigured round, much of the extr links nd switches cn remin unutilized. It is possible within the bsic node-covering frmework, which ws introduced by Dutt nd Hyes s n efficient k-ft design method, to design FT multiprocessors tht hve the sme mount of switches nd links s, sy, two-ft deterministic design, but hve s spre processors, where so tht, on the verge, k = Q(s) (k s) processor filures cn be reconfigured round. Such designs utilize the spre link nd switch cpcity very efficiently, nd re clled probbilistic FT designs. An elegnt nd powerful method to construct covering grphs or CG s, which re key to obtining the probbilistic FT designs, is to use liner error-correcting codes (ECCs). We show how to construct probbilistic designs with very high verge fult tolernce but low wiring nd switch overhed using ECCs like the D-prity, full-two, 3D-prity, nd full-three codes. This design methodology is pplicble to ny multiprocessor interconnection topology nd the resulting FT designs hve the sme node degree s the non-ft trget topology. We lso nlyze the deterministic fult tolernce for these designs nd develop efficient lyout strtegies for them. Finlly, we compre the proposed probbilistic designs to some of the best deterministic nd probbilistic designs proposed in the pst, nd show tht our designs cn meet given men-time-to-filure (MTTF) specifiction t much lower hrdwre costs (switch complexity, redundnt wiring re, nd spre-processor overhed) thn previous designs. Further, for given number of spre processors, our designs hve close-to-optiml reconfigurbilities tht re much better thn those of previous probbilistic designs. Index Terms Averge fult tolernce, deterministic fult tolernce, fult-tolernt multiprocessors, liner error-correcting codes, mtching, network flow, node-covering, VLSI lyout, reconfigurtion. 1 INTRODUCTION AND MOTIVATION I S. Dutt is with the Deprtment of Electricl Engineering nd Computer Science, University of Illinois t Chicgo, 110 Science nd Engineering Offices, 851 S. Morgn St. Chicgo, IL E-mil: dutt@eecs.uic.edu. N.R. Mhptr is with the Deprtment of Electricl nd Computer Engineering, Stte University of New York t Bufflo, Bufflo, NY E-mil: mhptr@eng.bufflo.edu. Mnuscript received 5 My 1995; revised 8 Dec For informtion on obtining reprints of this rticle, plese send e-mil to: tc@computer.org, nd reference IEEECS Log Number T is importnt to design multiprocessors with fult tolernce nd reconfigurtion cpbilities so tht they continue to function correctly nd with very little degrdtion, in spite of few processor/link filures, nd without requiring the pplictions to be coded for fult tolernce. To chieve this, multiprocessors need to be designed with structurl fult tolernce (SFT), which is the bility to reconfigure round fulty components (processors or links) in order to preserve the originl processor interconnection structure. SFT is lso useful in mintining correct nd nondegrded opertion of specilized processor rrys like those used for signl nd imge-processing [1]. The SFT (or FT, for brevity) design methodologies tht we will develop cn thus be pplied to both generl-purpose multiprocessors nd specil-purpose VLSI processor rrys for signl/imge-processing pplictions. A number of reserchers hve proposed vrious design methods for structurl fult tolernce in multiprocessors, for exmple, [], [4], [5], [6], [7], [8], [10], [11], [1], [13], [14], [15], [19], [], [3], [4], [5], [6], [7], [8]. Most of this work hs delt with the design of deterministic k-fulttolernt (k-ft) multiprocessors in which there re k or more spre processors, nd the link nd switch complexity is high enough to tolerte ny k processor filures (but not more thn k processor filures) [], [4], [5], [6], [7], [8], [10], [11], [1], [13], [14], [15], [19], [7], [3], [4]. Unfortuntely, in such designs, round 50 percent of the redundnt link/switch overhed remins unutilized, on the verge, fter k fulty processors hve been reconfigured round. It intuitively seems possible to utilize this remining spre link nd switch cpcity of the system by putting in s > k spre processors so tht, on the verge, more thn k fults cn be reconfigured round. Looking t this from different ngle, we would like to design FT multiprocessors in which the link/switch complexity is kept quite low (for exmple, corresponding to tht of two-ft deterministic design), but mny more ) spre processors re dded so tht k processor fults cn be reconfigured round, on /97/$ IEEE

2 998 IEEE TRANSACTIONS ON COMPUTERS, VOL. 46, NO. 9, SEPTEMBER 1997 Fig. 1. () Possible settings of recon-switch. (b) A stndrd CG for two-ft 16-processor system of rbitrry topology rrnged on twodimensionl lyout the liner ordering / corresponds to snke-like row-mjor ordering of this D lyout. the verge (or with very high probbility), where k < s nd This will lso led to very high utiliztion of the spre link/switch cpcity. Since the hrdwre complexities in lmost ll multiprocessor systems re wire dominted, it is importnt to keep the link redundncy s low s possible in n FT design; the processor complexity is not tht criticl, nd, thus, dding some excess, sy, 5-10 percent, processor redundncy does not significntly ffect the hrdwre complexity of the system. Since such designs re gered towrd hving lrge fult tolernce on the verge (over ll fult ptterns), but the worst-cse fult tolernce my be lower, we cll them probbilistic FT designs. Such probbilistic designs hve lso been proposed previously, though minly for D mesh-connected multiprocessors [6], [], [6], [7], [8]. In contrst, in this pper, we develop probbilistic designs tht re pplicble to ny topology, nd, for the specil cse of the D mesh, we compre our resulting designs reconfigurbilities nd hrdwre overheds to those of the best previous probbilistic designs [6], [7], [8]. In erlier work, Dutt nd Hyes developed very efficient methodology clled node-covering for designing deterministic k-ft multiprocessors [11], [14], [15]. The bsic node-covering technique will be the underlying design process for probbilistic FT, with liner codes being used to determine the interconnection between the externl switches used for reconfigurtion in order to obtin very high verge FT. Before proceeding further, we first define verge fult tolernce s k vg = s  i=1 (Probbility of tolerting size - i fult tht is not contined in tolerble fult pttern of lrger size) i A tolerble j-fult pttern will contin i-fult ptterns, where j > i, tht should not be counted seprtely s tolerble i-fult ptterns. Assuming tht tolerble i-fult ptterns re uniformly distributed over the spce of tolerble nd intolerble fult ptterns of lrger sizes nd tht p i is the probbility of tolerting ny i-fult pttern (this, we might determine by, sy, simultion), we hve k = ( p ( 1 - p )) i vg s s  i i= 1 j= i+ 1 Note tht, for k-ft system, k vg = k. In Section, we briefly describe the node-covering method for k-ft designs nd set the stge for discussing the j probbilistic design method. In Section 3, we begin the description of the probbilistic method by discussing the concept of prtitioning multiprocessor into intersecting FT processor groups. In the next section, we give the rtionle for using error-correcting codes (ECCs) s the bsis for determining such processor groupings. Section 5 describes efficient lyouts of some ECC-bsed FT designs nd obtins the re overheds in ech cse. Next, Section 6 describes how we model nd solve the reconfigurtion process in our FT designs s network-flow problem. In Section 7, we determine the deterministic fult tolernce (DFT), which is the mximum number of fults tht re gurnteed to be tolerted, of ech ECC-bsed design. Section 8 presents comprisons of reconfigurbilities nd hrdwre overheds of ECC-bsed FT designs with previous designs. Finlly, we conclude in Section 9. Due to spce constrints, we give some of our results without proofs, which cn be found in [16]. THE DETERMINISTIC NODE-COVERING METHOD In this technique [11], [14], we designte smll subset cov(v) of nodes (i.e., processors) of the FT multiprocessor G to which v cn be mpped (i.e., by which v cn be replced) for reconfiguring round fults. The set cov(v) is clled the cover set of v, while ech node u in cov(v) is clled cover of v, denoted by u % v. The node v is sid to be dependent of ll its covers, nd the set of dependents of node u is denoted by dep(u). Note tht, for SFT, we need some mechnism for ech node u to be connected to the neighbors of dependent node v in the event it replces v; we will shortly describe this mechnism. A covering grph (CG) of n FT multiprocessor G is directed grph, whose node set consists of ll the nodes of G (including the spre nodes), nd whose rc set is {(u, v) : v Œ V(G), u Œ cov(v)}, where V(G) is the node set of the non-ft multiprocessor G; see Fig. 1b. A covering sequence for node v in the CG is n ordered subset (v, v 1, v, º, v h ) of the node set of the CG, such tht v 1 % v, v j % v j-1, for 1 < j h, nd v h is the only spre node in this ordered set. Bsiclly, covering sequence for v specifies mpping which cn be performed when v is fulty, so tht v is replced by v 1, which, in turn, is replced by v, nd so on, until v h-1 is replced by the spre node v h, thus configuring it into the system. In order to be ble to reconfigure round ny k fults, we need set of k nodedisjoint covering sequences, one for ech fult, in the CG. In order to provide ech node with the necessry switching cpbility so tht it cn connect to the neighbors

3 DUTT AND MAHAPATRA: NODE-COVERING, ERROR-CORRECTING CODES AND MULTIPROCESSORS WITH VERY HIGH AVERAGE FAULT TOLERANCE 999 Fig.. () Connecting u to its neightbors vi its recon-switches. (b) An interconnection between recon-switches of four consecutive nodes in / for two-ft design, nd the switch settings for llowing u to replce w nd v to replce x. of ny dependent node tht it replces, we use switch, clled recon-switch, in our designs, whose possible sttes re specified in Fig. 1. This switch hs four bidirectionl terminls (in spite of their lbels, like v-input nd h-output, these re ll bidirectionl) nd cn connect ny pir of them together s shown. These terminls cn hve ny width (number of wires), nd switch with d wires t ech terminl is termed d-link switch. In k-ft system, ech processor is connected to its d neighbors vi the v- inputs nd v-outputs of k d-link recon-switches, s shown in Fig. for k =. For n efficient k-ft design, we need stremlined wy of llowing node to replce ny of its k dependent nodes. This is done by determining liner ordering / of the nodes nd by linking the h-outputs of ech of the k reconswitches of node u to the h-inputs of the corresponding recon-switches of the node v to its immedite right in / by reconfigurtion d-links (recon d-links or, simply, recon-links); see Fig. b. The underlying CG of this interconnection structure is stndrd CG in which ech node is covered by the k nodes immeditely preceding it in /; see Fig. 1b. Now, for node u to replce dependent, sy, w (either when w becomes fulty or when w replces one of its dependents in the covering sequence of fulty node), the d links from u re reconfigured to connect to the h-inputs of w s recon-switch, fter pssing through the recon-switch of ny intermedite processor v, nd, thereby, to w s neighbors; the links from w will be blocked by its first reconswitch if it is fulty, or similr connection will be estblished from w to the neighbors of the node it replces (Fig. b). In order to minimize the link overhed, / should be determined from n optiml D or 3D lyout of the multiprocessor so tht djcent nodes in / re lso djcent in the lyout. Then, since recon-links re only required between sptilly djcent processors, the wiring overhed of the k-ft design is minimized. It cn be shown tht, ssuming uniformly distributed fult loctions, bout 50 percent of the spre links nd switches remin unutilized fter ll k fults hve been reconfigured round in node-covering k-ft design [11], [14] (see [16] for proof); this lso pplies to other k-ft designs, e.g., those in [5], [7], [8]. 3 INTERSECTING FT PROCESSOR GROUPS In order to obtin high fult tolernce t reduced reconswitch nd recon-link overheds, we need to prtition the processors into smller groups tht re individully mde, sy, k 1 -FT, using the technique described in the previous section. The rtionle for prtitioning into smller FT processor groups is s follows: Assuming the model of uniformly nd independently distributed processor fults, the relibility of k-ft system with N primry processors is k N k i N k i RkNf = F + I + - Â i p b1 - p H K g, i= 0 where p is the filure probbility of single processor. If we prtition the system into t disjoint groups, nd mke ech group k 1 -FT, then the system s relibility is ( R ( N t)). If k 1 = k, then it is esy to see tht the prtitioned system hs higher relibility t the sme recon-link nd switch cost (but more spre processor cost, which is not tht criticl). Even for k 1 < k, we cn get the sme or higher relibility in the prtitioned system t reduced link nd switch overheds. For exmple, when p = 0.0, the relibility R (16) of two- FT 16-processor system is 0.994, wheres comprble relibility of is obtined when the system is prtitioned into four disjoint groups, ech of which is mde only one- FT. This disjoint-group bsed FT design is lso termed locl spring, since ech spre cn replce processors in only its locl subsystem, nd is discussed in [14]. A crucil question is whether these processor groups should be disjoint or intersecting. Fig. 3 shows disjoint grouping of processors into two groups, ech of which is mde two-ft, while Fig. 3b shows nondisjoint grouping of processors into four groups, ech of which is mde one- FT. Ech group is shown s thick line spnning liner processor spce. The hrdwre overheds of the two processor-grouping schemes re similr ech requires four spre processors, nd two recon-switches nd recon d-links per processor (in Fig. 3b, ech processor belongs to two groups, nd one recon-switch is used to connect it to one of its groups nd the second one to the other group). It is esy to see tht the intersecting-groups scheme cn tolerte ll fult ptterns tht the disjoint-groups scheme cn (ssume, for the moment, tht ech group cn reconfigure round fults in it without ny conflict with reconfigurtions in intersecting groups we will shortly show tht this is true for our designs). Furthermore, the former scheme cn tolerte mny more fult ptterns thn the ltter. For exmple, the three-fult pttern shown in Fig. 3 cnnot be tolerted in the disjoint-grouping cse, but it cn be tolerted in the intersecting-groups scheme s follows: The leftmost fult cn be reconfigured in Group 4, the middle one in Group 1, nd the third one in Group. This simple exmple clerly shows tht, everything else being equl, n intersecting processor grouping yields more fult tolernce nd relibility thn disjoint processor grouping. This is essentilly becuse intersecting processor groups hve mny more pths between processors nd spres thn disjoint groups. The next crucil question is whether there is systemtic method to form different types of intersecting processor groups tht hve different fult tolernce nd hrdwre cost properties. The nswer to this is in the ffirmtive. In prticulr, we cn drw upon liner error-correcting codes (ECCs) in order to form processor groupings; different k1 t

4 1000 IEEE TRANSACTIONS ON COMPUTERS, VOL. 46, NO. 9, SEPTEMBER 1997 Fig. 3. () A disjoint processor grouping into two two-ft groups; it cnnot tolerte the three-fult pttern shown. (b) An intersecting processor grouping into four one-ft groups tht hs similr hrdwre overhed to tht of the disjoint grouping; it cn tolerte the three-fult pttern. ECCs will yield FT designs with different properties with respect to the bove prmeters. In Section 4, we show how ECCs cn be used to form (intersecting) processor groups, nd how the deterministic nd verge error detectbility of n ECC form lower bounds for the deterministic nd verge fult tolernce of the corresponding FT multiprocessor, nd, in generl, why they re good choice to determine these groupings. Some previous probbilistic methods [], [6], [8] hve lso used intersecting processor groups for the specil cse of the D mesh topology; their processor groups re the rows nd columns of the D mesh, nd, thus, they fortuitously hppen to be bsed on the D-prity code. In Section 8, we compre our designs to some of the best of such previous designs. 3.1 Switching Structure The switch implementtion of n FT multiprocessor with intersecting processor groups is n extension of the deterministic (k-ft) node-covering design described in Section. We ssume, for simplicity, tht ech processor group is mde one-ft, nd, thus, hs single spre processor. Suppose ech processor belongs to d g groups. Then, we connect d g recon-switches to ech processor in series (vi v-input nd -output terminls s in k-ft designs). We lbel the recon-switches from top (to which the processor is directly connected) to bottom s d g - 1, º, 0. For ech processor group P i, we order the processors in it in some mnner (essentilly by physicl proximity, s described in Section 5), nd connect, sy, the ith recon-switches of djcent processors in this ordering vi their h-inputs nd -outputs to form chin 1 ; the spre processor is t the end of the chin. Fig. 4 shows such recon-switch interconnection for 4 4 rry of processors (this rry cn correspond to multiprocessor of ny topology most homogeneous or lmosthomogeneous multiprocessors like k-ry n-cubes cn be optimlly lid out so tht processors pper in regulrly spced rows nd columns) in which ech row nd column corresponds to distinct processor group. Thus, ech processor belongs to exctly two processor groups, nd this grouping is clled degree-two grouping. Fig. 8 shows similr processor grouping nd recon-switch interconnection for n FT four-dimensionl hypercube. Note lso tht 1. It my not lwys be possible to connect ll processors in group by their ith recon-switches throughout, in which cse, fter connecting the first few processors by their ith recon-switches, we shift to connecting the rest vi, sy, their jth recon-switches; we lso connect the lst ith recon-switch of the first subset of processors to the first jth recon-switch of the second subset of processors. ech P i cn be mde two-ft by just ttching second spre t the hed end of this chin (the first spre processor is considered to be t the til end of the chin), without requiring ny extr recon-switches; see the second row of Fig. 4c, which hs two spres s h *,1 nd st *,1 t either end the lbeling conventions for spres nd groups will be specified in Section 5. When there re two fults in P i, ech cn be reconfigured by covering sequences tht go in opposite directions in P i, ech towrd the nerer spre processor. The underlying CG of Fig. 4 is shown in Fig. 4b. There re two slient points to note bout such CG for intersecting processor groups: 1) The covering rcs in the CG re bidirectionl, i.e., the covering reltion is symmetric (if u % v, then v % u). This is becuse the recon-switches re interconnected using bidirectionl recon-links so tht, in Fig. 4, processor (, 1) cn replce processor (1, 1) or vice vers. Also, note in Fig. 4c tht the covering sequences or reconfigurtion pths for fults (1, 0) nd (, 1) re in opposite directions. ) The covering reltion within ech processor group is lso trnsitive, i.e., if u % v nd v % w, where u, v, nd w belong to the sme processor group, then u % w. Agin, this is fcilitted by the switching structure. For exmple, in Fig. 4c, processor (0, 1) directly replces processor (, 1) (since (1, 1) is fulty) by pssing through processor (1, 1) s zeroth recon-switch. Note tht if (1, 1) ws nonfulty, then bypssing it in this mnner does not hinder its replcing cpbilities in its column group, i.e., it cn replce ny processor in column #1 or cn be replced by ny processor in it. This is becuse (1, 1) is bypssed by using only the h-input nd h-output links of its recon-switch corresponding to its row group, nd these links re not useful for (1, 1) to replce or be replced by processors in its column group; see Fig. 4c. It is in this sense tht the covering reltion within group is trnsitive, i.e., if u replces nondjcent processor in group P i, then this does not inhibit the replcement cpbilities of the bypssed intermedite processors in their other groups. We thus hve the following result. THEOREM 1. The covering reltion in CG for ny (disjoint or nondisjoint) processor grouping scheme in which ech group is mde k 1 -FT, for ny k 1 1, is symmetric nd, within ech group, it is trnsitive.

5 DUTT AND MAHAPATRA: NODE-COVERING, ERROR-CORRECTING CODES AND MULTIPROCESSORS WITH VERY HIGH AVERAGE FAULT TOLERANCE 1001 Fig. 4. () Recon-switch interconnections for intersecting groups long rows nd columns; this processor grouping is lso the one derived from D-prity ECC. (b) Its covering grph. (c) Illustrtion of vrious reconfigurtion pths. 3. Reconfigurtion We next chrcterize the fult ptterns tht cn be reconfigured round in design with intersecting groups. We first define the concept of mtching. A mtching system is set up s follows. There re two sets: $ of size n nd % of size m. With ech element i of $, subset B i of % is ssocited. A mtching is sid to exist between $ nd % with respect to the collection of subsets { B : Œ $ } if there is one-to-one function f :$ Æ % such tht f( ) Œ B, for ll i = 1, º, n. The following theorem, known s Hll s theorem, sttes the condition under which mtching exists [3]. THEOREM [3]. There exists mtching between sets $ = { 1, º, n } nd % = {b 1, º, b m } if nd only if, for ny k elements k { i, K, i } of $, B 1 k U hs t lest k elements (of %), for ll k Œ {1,, º, n}. j=1 The next theorem estblishes tht mtching between the set of fults nd the set of spres in the bove sense is sufficient for reconfigurtion. THEOREM 3. Let F be the set of processor fults. For ech primry fult u, let S u be the set of nonfulty spres in ech group tht u belongs to, nd, for ech spre fult v, let S v = {v}. Then, if there is mtching between F nd the set S of ll spres, with respect to {S u : u Œ F}, then the set of fults F is reconfigurble. PROOF. Let s u be the spre mtched to primry fult u by the mpping f : F Æ S (note tht we re not interested in reconfiguring fulty spre which is mpped to itself by f). Then, since, from Theorem 1, the covering reltion within ech group is trnsitive, nd since ech spre is ssocited with distinct group, (in the worst-cse) s u cn directly replce u without disturbing ny intermedite processor s replcement cpbilities in other groups.. Prcticlly speking, hving the spre directly replce distnt processor in its group is not such good ide, since this mkes the links from s u to its new neighbors (the previous neighbors of u) very long. Thus, ech fult should be directly replced by the nerest vilble nonfulty processor in i j i i i i While mtching is sufficient for reconfigurtion, it is not necessry. For exmple, in Fig. 4c, there is no mtching for the set of primry fults {(1, 0), (1, 1), (, 1), (, ), (3, 3)}, since both spres in (3, 3) s two groups re fulty. However, reconfigurtion is still possible s shown in the figure. In this cse, (3, 3) is replced by (3, ), which, in turn, is replced by spre s t *, resulting in bend in the reconfigurtion pth. This bend occurs becuse (3, ) s replcement of (3, 3) is long its column group nd then s t *, s replcement of (3, ) is long its row group. We re thus bending from group connected by higher numbered reconswitch (#1) to one connected vi lower numbered reconswitch (#0). The reverse type of bend, i.e., replcement long row followed by replcement long column is not possible becuse of the contention for the v-link connecting the first nd zeroth recon-switch of the processor t which the bend occurs. Finlly, note in the figure tht (1, 0), (1, 1), nd (, 1) re ll replced by mtched spres in their own groups, nd their reconfigurtion pths re thus stright. It is possible to exctly determine vlid set of reconfigurtion pths for set of fults in the bove-described switching structure using network mx-flow lgorithm, s described lter in Section 6. However, the mtching criterion is still importnt becuse 1) In most cses, we cn determine the reconfigurtion pths by using simple mtching lgorithm, ) It points us to the use of ECCs s systemtic method for determining FT processor groups, nd 3) It is lso useful in obtining the deterministic fult tolernce of our designs. 4 USING ECCS TO DETERMINE FT PROCESSOR GROUPS 4.1 Introduction A liner ECC hs N informtion bits nd g check-bits tht re relted to the informtion bits by liner or prity equtions, the group (to which s u belongs), which should lso be similrly replced, nd so on, until s u replces processor in this covering sequence for u. Only in the worst cse, for exmple, when ll intermedite processors re fulty, will s u need to directly replce u.

6 100 IEEE TRANSACTIONS ON COMPUTERS, VOL. 46, NO. 9, SEPTEMBER 1997 Fig. 5. () Informtion bit groups in the D-prity code. (b) A prity check mtrix H = [P I] for liner code with N informtion nd g check-bits. Multiplying H with vlid code word X results in vector of zeros. nd cn be used to determine if there re ny errors in the code word (the combintion of informtion nd check-bits), nd possibly correct them. The simplest liner ECC (herefter referred to only s ECC) is the single- or 1Dprity code in which the single check or prity bit c 0 N -1 i t t= 0 = Â, where the i t s re the informtion bits nd  nd + denote the modulo- sum or Exclusive-Or (XOR) opertion. In the rest of this section, ll rithmetic will be modulo ; note tht modulo- multipliction is the sme s the AND opertion. A single error in ny bit in the code word will result in violtion of the bove eqution nd thus led to detection of the error. Insted of tlking bout the prity of ll the informtion bits, we cn tlk bout prities of vrious groups of informtion bits. An exmple is the D-prity code, in which the N informtion bits re rrnged in two dimensions s n n m rry, where N = n m, nd ech row nd column of this mtrix is group with which prity check-bit is ssocited; see Fig. 5, in which two exmple prity equtions (for c 1 nd c 5 ) re lso given. The D-prity code cn detect ny two errors (its deterministic error detectbility) nd cn correct single error. Any (liner) ECC is essentilly collection of prity equtions, one for ech group of informtion bits. We cn construct processor groups from given ECC in the following mnner. Ech primry processor is uniquely ssocited with n informtion bit nd ech spre processor with check-bit. Ech group nd its check-bit in the ECC then gives rise to processor group nd its ssocited spre. 3 There re five metrics ssocited with ECCs tht re of interest to FT multiprocessor design: 1) Check-bit overhed, which is the rtio of check to informtion bits. This trnsltes to spre-processor overhed in n FT multiprocessor. If the D rrngement is n n n squre, i.e., N = n, then this overhed 3. Note tht the two-spre- per-group ssignment discussed in Section 3 does not exctly fit n ECC structure (since there is no point in hving two check-bits per informtion-bit group, both with the sme prity equtions). However, it is nturl ugmenttion of the one spre per processor group structure derived from the ECC tht leds to higher relibility nd higher utiliztion of the recon-switches nd links in the FT multiprocessor. is g N = n N = N. ) Group size, which is the size of the lrgest group. This is importnt, since, when single fult occurs tht cn be reconfigured using mtching (to spre in one of its groups), then only the processors in tht group need be involved in the replcement process, nd will not be vilble for the period it tkes to reconfigure. Thus, with lrge group size, the vilbility of the multiprocessor decreses. Group size of the squre D-code is N. 3) Group degree, which is the mximum number of groups n informtion bit lies in. In n FT multiprocessor, this trnsltes to the mximum number of recon-switches nd recon-links needed per processor. The D-prity code hs degree two. 4) Deterministic error detectbility (DED), which is the mximum number of errors tht the ECC will lwys detect. This provides lower bound on the DFT (mximum number of processor fults tht cn lwys be reconfigured round) of the FT multiprocessor, s we will shortly see. 5) Averge ersure correctbility, which is defined similrly to verge fult tolernce k vg (see Section 1). The concept of ersure correctbility, which ws introduced in [18], will be explined shortly. This metric provides lower bound on the verge fult tolernce of the multiprocessor, s shown lter. We will describe four useful ECCs, the D-prity, fulltwo, 3D-prity, nd full-three codes, tht provide different trde-offs of these metrics. The prtitioned 1D-prity code yields the disjoint-group or locl spring FT design discussed briefly in Section 3; however, s we will see in Section 8, its fetures re not s ttrctive s designs bsed on other ECCs tht result in intersecting FT processor groups. Before describing these other codes, we briefly revisit the theory of error-correcting codes [1] tht will be useful in formulting why ECCs re good bsis for determining FT processor groupings. 4. Relting Error Detection, Ersure Correction, nd Fult Tolernce A liner ECC cn lso be described in terms of g (N + g) prity check mtrix, H = [P I] shown in Fig. 5b. The N columns of P represent informtion bits, nd the g columns of

7 DUTT AND MAHAPATRA: NODE-COVERING, ERROR-CORRECTING CODES AND MULTIPROCESSORS WITH VERY HIGH AVERAGE FAULT TOLERANCE 1003 I the check-bits. Ech row corresponds to group; the columns tht hve 1s in row correspond to informtion bits in tht group. If vector X is vlid codeword, i.e., its checkbits re derived from the informtion bits using the prity equtions, then H X = 0 (recll tht we re using modulo- rithmetic). We next define the concept of liner independence nd then stte bsic theorem in error-correction coding theory. A set S of binry vectors is sid to be linerly dependent, if the modulo- sum of some subset of its vectors yields the 0 vector. If S is not linerly dependent, then it is sid to be linerly independent. THEOREM 4 [1]. The DED of n ECC is t if nd only if ny t columns of its prity mtrix re linerly independent. Similr to the mtching system between fulty processors nd spres described in Section 3, we cn set up mtching system between bits nd prity groups s follows. Let A be the set of informtion nd check-bits i 0, i 1, º, i N+g nd 3 the set of prity groups in code. With ech bit i j in A, we ssocite subset P ij of 3 of those prity groups tht contin i j. Then, given subset E of erroneous bits of A, there is mtching between E nd 3, with respect to { P ij : i j Œ E}, if there exists one-to-one function y : E Æ 3, such tht y ( i ) Œ P for ll i j Œ E. We next relte mtching nd j i j error detectbility of code; we first stte useful result from liner lgebr [17] given here in more relevnt form. THEOREM 5 [17]. Any set of m vectors, ech of length n, re linerly dependent if m > n. THEOREM 6. If t is the DED of code &, then there will lwys be mtching between ny set E of t most t erroneous bits nd the set 3 of prity groups of &. PROOF. Suppose there is subset E of t bits for which there is no mtching. Since & cn detect ny t errors, it mens tht the columns of the prity mtrix H corresponding to the bits in E re linerly independent (Theorem 4). For convenience, let these columns be H 0, º, H t-1. Note tht the set P ij is the set of rows (i.e., prity groups) in which the i j th column of H hs 1s. Since there is no mtching for bits i 0, º i t-1, from Theorem, there is nonempty subset E of E of size m t whose bits re ll contined within l m - 1 prity groups. In other words, the m columns in H, sy, H 0, º, H m-1, corresponding to the bits in E, together hve 1s in l m - 1 rows nd 0s in ll other rows. To determine if these columns re linerly dependent, we need only consider these l rows, since their sum will yield 0s in the other g - l rows. Thus, we cn only consider the m subcolumns of H 0, º, H m-1 of length l tht contin only the rows in which t lest one of them hs 1. Since l < m, it follows from Theorem 5 tht these subcolumns re linerly dependent, nd, thus, H 0, º, H m-1 re linerly dependent. Hence, H 0,º, H t-1 re lso linerly dependent nd we rech contrdiction. THEOREM 7. The DFT of multiprocessor whose FT processor groups re bsed on ECC & is lower bounded by & s DED when there is one spre per processor group, nd by twice its DED when there re two spres per group. PROOF. Let & be t error detecting. Then, from Theorem 6, there is mtching for ny t errors. We first consider the single spre-per-group design. Since ech bit corresponds to primry or spre processor in the multiprocessor, nd since the bit-to-prity eqution mtching system corresponds to tht between fulty processors nd the set of spres, there exists mtching in the ltter system for ny t fults. Hence, from Theorem 3, the t fults cn be reconfigured round. Now, consider the two-spre-per-group design, nd suppose the multiprocessor hs t fults; we rndomly prtition them into two size-t sets F 1 nd F. From the bove discussion, the fults in F 1 cn be mtched to the til spres in the system (ech group hs spre t the til of the group, s in the one-spre cse, nd second spre t its hed). Note tht the presence of fult set F does not hinder the reconfigurtion of the fults in F 1, since, if the former re in the mtching-bsed reconfigurtion pths of the ltter, then they re simply bypssed. Similrly, fult set F cn be mtched to the hed processors in the system. In this cse, however, the reconfigurtion pth from ny u Œ F to its mtched hed spre in, sy, group i, might conflict or overlp with tht of fult v Œ F 1 tht is mtched to the til spre in group i. This hppens when u is nerer to the til spre nd v nerer to the hed spre in group i. Thus, the mtched spres of u nd v cn be interchnged, nd u cn be reconfigured to the til spre nd v to the hed spre in group i without conflict. Note tht there re t most two reconfigurtion pths in group, the bove interchnge does not ffect ny reconfigurtion pths in ny other group, nd this type of interchnge cn be mde in every group in which there re conflicts between two reconfigurtion pths. In Section 7, we nlyze the exct DFT of multiprocessors bsed on vrious codes nd show tht these re quite higher thn the respective DEDs of these codes. In generl, if r > t errors occur, then they re detectble s long s the sum of the r columns corresponding to r error bits is nonzero [1]. Liner independence of the r columns is sufficient to yield nonzero sum; however, it is not necessry (for exmple, subset of the r columns cn be summed to yield 0 vector, but the sum of ll r columns my not be zero). A more relevnt concept for our purpose is ersure correctbility of code used in [18], where ECCs were pplied to design redundnt disk rrys (RAIDs) in which disk ersures (disk filures in which the dt on the disk is lost) cn be tolerted by reproducing the dt of the filed disks using prity equtions. In such RAIDs, ech primry (spre) disk corresponds to n informtion (check) bit of the

8 1004 IEEE TRANSACTIONS ON COMPUTERS, VOL. 46, NO. 9, SEPTEMBER 1997 code. An ersure is essentilly loss of bits in code word whose loctions re known but not vlues. If these unknown or lost bits cn be reproduced, then we sy tht the ersure is correctble. The deterministic ersure correctbility of code is the sme s its DED [18]. However, its verge ersure correctbility is upper-bounded by its verge error detectbility, since r > t disk ersures cn be corrected if nd only if the r columns corresponding to the filed disks re linerly independent [18]. The men-time-tofilure (MTTF) of RAID derived from n ECC ws determined in [18], nd this gives resonbly good mesure of the verge ersure correctbility of the code. The next theorem reltes the verge fult tolernce of n FT multiprocessor to the verge ersure correctbility of code. THEOREM 8. The verge number of ersures tht cn be corrected by code & is lower bound for the verge number of fults tht cn be tolerted by n FT multiprocessor whose processor groupings, with one spre per group, re bsed on &. PROOF. Suppose there re r > t errors (ersures in the corresponding RAID) in code word. These ersures cn be corrected if the corresponding r columns of the prity mtrix H re linerly independent. Proceeding in the sme mnner s in the proof of Theorem 7, we cn see tht, in such cse, the corresponding r processor fults in the FT multiprocessor cn be reconfigured to mtched spres. Thus, for ech correctble pttern of disk ersures, there is corresponding pttern of reconfigurble processor fults. The theorem follows from this observtion. We obtin the following useful corollry of this theorem. COROLLARY 1. The relibility (MTTF) of code & or of the corresponding RAID system, s determined by its ersure correctbility, is lower bound for the relibility (MTTF) of the corresponding FT multiprocessor, with one spre per group, if we ssume tht the filure probbility of disk in the RAID system is the sme s tht of processor. Also, the bove metrics for two-spre-per-group design re lower bounded by the respective metrics of the corresponding single-spre-per-group design. In [18], number of useful ECCs like the D-prity, 3Dprity, full-two, nd full-three codes were identified tht yield very high MTTF for RAID designs bsed on these codes, nd which hve ttrctive vlues of metrics, like check-bit overhed, group degree, nd group size. These metrics re lso relevnt to FT multiprocessor design s discussed erlier. In view of this, nd Theorems 7 nd 8 nd Corollry 1, we choose to use these codes to design FT multiprocessors. Lter, we show tht the DFT, lyout-re overhed, nd relibility of these FT multiprocessors re extremely fvorble. Next, we briefly describe the bove ECCs. 4.3 Some Useful ECCs The D-prity code ws described erlier; its prity mtrix is shown in Fig. 5c. The 3D-prity code cn be similrly formed by rrnging the informtion bits in threedimensionl rry nd ssociting prity eqution with ech dimensionl group. Its check-bit overhed is 3/N 1/3, its group size is N 1/3, it is degree-three code nd its DED is three. The full-two code is degree-two two-ded code. It cn be defined in terms of its prity mtrix H full = [P full I]. P full consists of ll possible distinct columns with exctly two 1s; see Fig. 5d. The number N of informtion bits in terms of g is given by N = g(g - 1)/, i.e., its check-bit overhed is pproximtely N, nd its group size is g - 1 ª N - 1. The full-three code is degree-three three-ded code, nd P full3 consists of ll possible distinct columns with exctly three 1s. For this code, N = g(g - 1)(g - )/6, thus, its checkbit overhed is pproximtely (6N) 1/3 /N = 1.8/N /3, nd its group size is 3N/g = (g - 1)(g - )/ < (6N) /3 /. The fulltwo nd full-three codes hve the lest possible check-bit overheds of ny degree-two nd degree-three codes, respectively. The D- nd 3D-prity codes hve better relibilities thn the full-two nd full-three codes, respectively, though, obviously, higher check-bit overheds. However, s we will see next, wiring overheds for FT multiprocessors bsed on full-two nd full-three codes re little higher thn those bsed on the D- nd 3D-prity codes, respectively. 5 WIRE-EFFICIENT LAYOUTS 5.1 Lyout Scheme In this section, given regulr D-squre or 3D-cubic lyout 4 of non-ft multiprocessor of n rbitrry topology, we will obtin similr lyout of n ECC-bsed FT multiprocessor by embedding processor groupings defined by the ECC in the given non-ft processor grid. Embeddings with minimum edge congestion (the number of recon d-links tht get mpped onto single grid line, i.e., the spce between djcent processor rows/columns/heights) will be used to minimize the redundnt wiring overhed, nd thus obtin efficient FT lyouts in topology independent mnner. Let 0, 1, º, g - 1 be the g processor groups in the FT multiprocessor nd let d g denote its group degree (the number of groups processor belongs to). Any processor u is ssocited with d g -digit lbel ( b, b, K, b -, b - ) which speci- 0 1 dg dg 1 fies its loction in d g -dimensionl grid for different codes, these lbels will be determined differently. Recll from Section 3 tht every processor uses d g recon-switches connected in series nd lbeled 0, 1, º, d g - 1. (see, e.g., Fig. 4). For the D- nd 3D-prity codes, the processor lbels re the subsets of grid points given by the squre {( b, b ) : 0 b, b < N } nd the cube {(b 0, b 1, b ) : 0 b 0, b 1, b < N 1/3 }, respectively. Processors in the sme row hve different b 0 but the sme b 1 nd b (for the 3D-prity code) vlues, nd belong to the sme group. These processors re linked from the smllest- (the hed processor) to the lrgest-lbeled processor (the til processor) vi their zeroth 4. Actully, s will become cler lter, our lyout schemes re eqully effective when the given non-ft lyout is rectngulr or cuboidl.

9 DUTT AND MAHAPATRA: NODE-COVERING, ERROR-CORRECTING CODES AND MULTIPROCESSORS WITH VERY HIGH AVERAGE FAULT TOLERANCE 1005 Fig. 6. Wire-efficient embeddings of processor groupings in n FT multiprocessor rry (of ny topology) derived from the D-prity code for N = 100 nd g = 0. recon-switch. In single-spre-per-group design, we connect spre s*, b 1, b t to the til processor in the group, nd, in the double-spre cse, we connect, in ddition, nother h spre s*, b 1, b to the hed processor vi their zeroth reconswitches. Similrly, column nd height groups re formed, nd hve spres connected to them. Note tht this lyout hs n edge congestion of one (see Fig. 6). Next, for the full-two nd full-three codes, the lbel ( b, b, K, b -, b - ) of processor u indictes, in ddition 0 1 dg dg 1 to its position, the groups to which u belongs, so tht 0 b i < g for ech b i. Moreover, now the ith recon-switch is used to connect u to group b i. Note tht ny two or three groups in the full-two nd full-three codes, respectively, contin exctly one common processor. Suitble subsets of grid points for these two codes tht stisfy this requirement nd tht give compct D nd 3D tringulr pyrmidl lyouts of processors, respectively, re {(b 0, b 1 ) : g > b 0 > b 1 0} (see Fig. 7) nd {(b 0, b 1, b ) : g > b 0 > b 1 > b 0} these lyouts will be modified to squre nd cubic lyouts, respectively, in Section 5.. We interconnect processors to form groups s follows. For ny group j, where 0 j < g, let j mx nd j min be the lrgest nd smllest digit positions, respectively, over ll processors in the group, t which the group number j ppers in processor s lbel. For instnce, in Fig. 7, for group j = 0, j mx = j min = 1, nd, for group j = 8, j mx = 1 nd j min = 0. The jth processor group is formed by first linking ll processors with b k = j, where j min k j mx, strting from the first to the lst processor in (d g - 1)-dimensionl rdix-g Gry-code ordering (obtined s n extension of reflected binry Gry codes) of their corresponding (d g - 1)-digit lbels ( b 0, K, b, *, b, K, b k- 1 k+ 1 d g -1, vi their kth recon-switches. For instnce, if g = 8, d g = 3, k = 1, j = 3, nd the set of processors with b 1 = 3 is {(4, 3, 0), (4, 3, 1), (4, 3, ), (5, 3, 0), (5, 3, 1), (5, 3, ), (6, 3, 0), (6, 3, 1), (6, 3, ), (7, 3, 0), (7, 3, 1), (7, 3, )}, then the two-dimensionl rdix-8 Gry-code ordering of the corresponding lbels (b, b 0 ) is (4, 0), (4, 1), (4, ), (5, ), (5, 1), (5, 0), (6, 0), (6, 1), (6, ), (7, ), (7, 1),(7, 0). In the cse of the full-three code, we ctully order the processors in n lmost-gry-code mnner, since not ll (d g - 1)-digit lbels bove pper in processor lbels. Consider group 0 (i.e., j = 0) nd k =. First, we order processors with lbels of the form (, *, 0) from the smllest to the lrgest lbeled processor processor (, 1, 0) is the only such processor. Then, we order ll processors with lbels of the form (3, *, 0) from the lrgest to the smllest lbeled processor nd dd them to the previous ordering of processors processors (3,, 0) nd (3, 1, 0) re the only such processors nd re rrnged in tht order. Next, processors with lbels of the form (4, *, 0) re ordered from the smllest to the lrgest lbeled nd ppended to the bove ordering thus, the sequence (4, 1, 0), (4,, 0), (4, 3, 0) is ppended to the bove ordering, nd so forth. A Gry-code (or lmost-gry-code) ordering, like the ones bove, is useful in minimizing reconfigurtion link length between djcent processors in group recll tht processor s lbel gives its loction in d g -dimensionl grid, nd, thus, djcent processors in this ordering will be sptilly djcent. Next, the lst processor (in the bove ordering) with b k = j is linked to the first processor with b k+1 = j, for j min k < j mx, vi the kth recon-switch of the former nd the (k + 1)th recon-switch of the ltter. This gives us totl ordering of ll the processors in group j with the first processor with bj = j t the hed of the group nd the lst processor min with bj = j t the til of the group. For instnce, in the mx

10 1006 IEEE TRANSACTIONS ON COMPUTERS, VOL. 46, NO. 9, SEPTEMBER 1997 Fig. 7. Wire-efficient embeddings of processor groupings in n FT multiprocessor rry (of ny topology) derived from the full- code in () tringulr D lyout nd (b) in squre D lyout obtined by modifying the tringulr lyout, for N = 105 nd g = 15. full-two code bsed FT multiprocessor in Fig. 7, group 8 is formed by first linking processors (8, 0), (8, 1), º, (8, 7) in tht order vi their zeroth recon-switches nd processors (9, 8), (10, 8), º, (14, 8) in tht order vi their first reconswitches, nd then linking processor (8, 7) to (9, 8) vi the former s zeroth nd the ltter s first recon-switches. The hed nd til processors in this group re processors (8, 0) nd (14, 8), respectively. We connect spres s j t nd s j h to the til nd hed processors, s in the D- nd 3D-prity cses, vi the j mx th nd j min th recon-switches, respectively. Note, gin, tht the lyouts obtined hve congestion of one. 5. Are/Volume Overhed We now give the frctionl re (volume) overhed f o of the D (3D) lyouts of FT designs bsed on the vrious codes. Denote the re (volume) of the non-ft multiprocessor by A nd let = A/N be the re (volume) per processor this re (volume) is ssumed squre (cubic) nd includes the re (volume) occupied by the ctul processor nd its shre of the interprocessor wiring. Recll tht the recon d- link for ech group corresponds ctully to d sets of wires, one for ech neighbor of processor, so tht its width is d units. THEOREM 9. The frctionl re overhed of n FT design bsed on the D-prity code compred to non-ft design is f o = d+ d. PROOF. The re of the FT design is e N d + dij = N e + d + d j N ( + d+ d )- N d+ d (see Fig. 6). Hence, f o = =. THEOREM 10. The frctionl re overhed of n FT design bsed on the full-two code compred to non-ft design is f o ª d+ d squre lyout. N d+ d for tringulr lyout nd f o ª for g - g g - N PROOF. Recll tht for full-two code ( 1 ) ª ( 1 ) =, so tht g ª ( g - 1) ª N. First, consider tringulr lyout, s in Fig. 7, tht hs bse nd height of g - 1 processors ech. The re of the FT design is 1 b gd i e g 1 d j N e d dj, - + ª + + d+ d. thus giving f o ª A squre lyout of the FT design cn be obtined by modifying the tringulr lyout of Fig. 7 s shown in Fig. 7b. The polygon of processors ((5, 0), (14, 0), (14, 10), (11, 10), (5, 4)) nd the ssocited spres on the right in Fig. 7 remin fixed. The tringle of processors ((1, 0), (4, 0), (4, 3)) is inverted, flipped, nd plced t the bottom left corner of the bove polygon. The tringle of processors ((1, 11), (14, 11), (14, 13)) is scnned in row-mjor order nd plced right to left t the bottom of the bove lyout, t t t s shown in Fig. 7b. Spre processors s, s, s, nd s t re plced ppropritely in the lst row. For ny N, the bove scheme cn be used to obtin squre lyout of N processors to side, with n edge congestion of ( g - N ) + 1 in the leftmost column nd

11 DUTT AND MAHAPATRA: NODE-COVERING, ERROR-CORRECTING CODES AND MULTIPROCESSORS WITH VERY HIGH AVERAGE FAULT TOLERANCE 1007 bottommost row (corresponding to recon links connected to ( g - N ) groups in the two displced tringle of processors bove) nd one in ll other rows nd columns (see Fig. 7b). Therefore, the re of the squre FT lyout is e Nd + di + dg - Nidj ª d N + N di e = N + d + d, d+ d so tht f o ª. THEOREM 11. The frctionl volume overhed of n FT design bsed on the 3D-prity code compred to non-ft design = is f o d d d. PROOF. As in the D-prity code bsed design, with ech processor group long row, column, or height, we hve wiring overhed of d, so tht the width of ech row, column, or height is ( 1/3 + d). Thus, the volume of the FT design is (N 1/3 ( 1/3 + d)) 3, which implies d+ 3 d + d f o =. THEOREM 1. The frctionl volume overhed of n FT design bsed on the full-three code, compred to non-ft design, is d+ 3 d + d f o ª d d d f o ª for tringulr pyrmidl lyout nd for cubic lyout. PROOF. Since the redundnt re overhed round ech processor in the squre D-prity code nd tringulr full-two code lyouts is the sme (see Figs. 6 nd 7), the corresponding overheds re lso the sme (see Theorems 9 nd 10). The cse is the sme with the cubic 3D-prity code nd tringulr pyrmidl fullthree code lyouts. Thus, from Theorem 11, in the ª ltter cse, f o d d d The tringulr pyrmidl lyout for the full-three code cn be modified to obtin cubic lyout in mnner similr to tht in the full-two code cse considered in the proof of Theorem 10. Agin, nlogous to the full-two code cse, the volume of cubic full-three code lyout is (N 1/3 ( 1/3 + d) + (g - N 1/3 ) d) 3. Since, for ( -1)( -) full-three code, g g g = N, we pproximte g by 6 (6N) 1/3, obtining f o d d d. ª j. The totl re or volume A of lyout for given network topology is proportionl to the squre or 3/ power, respectively, of its bisection width, which is defined s the minimum number of wires tht must be cut to seprte the network into two equl hlves. The bisection widths for mesh nd hypercube rchitectures re N nd N/, so tht their vlues re Q(1) nd Q(N/4), respectively. To get n ide of the re/volume overhed, let us consider n exmple. Recll tht d is the combined width of ll redundnt wires (i.e., recon d-link) entering processor when the wires re closely pcked (s in the lyouts of Figs. 6 nd 7b), nd or 1/3 is the width of side of squre re or cubic volume, respectively, contining processor, severl memory chips, the processor-memory bus, possibly seprte routing chip nd input/output lines, nd the processor s shre of nonredundnt interprocessor wiring. Furthermore, note tht d/4 nonredundnt wires enter ech side of the squre routing chip contined in this squre region, nd tht wire connected to pin on the chip will hve much smller width compred to the pin pitch. Therefore, resonble vlue for the rtio between the combined width of d/4 sets of closely pcked redundnt wires nd the side of n individul squre region would be d 4 log N 4 b g 4 = for mesh, nd d = for hypercube rchitecture, N 4 where note tht d = 4 for mesh nd d = log N for hypercube. The wire width d/4 is similrly relted to the width 1/3 of cubic region for 3D lyout. From the bove theorems, we obtin the following results for mesh rchitecture (of ny size): f o (squre Dprity or tringulr full-two code lyout) = 44 percent nd f o (squre full-two code lyout) = 65 percent; for 1,04- processor hypercube the corresponding overheds re six percent nd nine percent. The overheds for the 3D lyouts corresponding to the 3D-prity nd full-three codes re somewht higher. For the mesh rchitecture, the overheds corresponding to the D-prity nd full-two codes re resonble considering the improvement in relibility of the system (s will be seen in Section 8), while, for the hypercube rchitecture, the corresponding overheds re quite low. 6 RECONFIGURATION ALGORITHM The reconfigurtion problem in the node-covering method is equivlent to finding the mximum flow in flow grph G; such correspondence ws lso observed in [9] for the FT design presented therein. The flow grph corresponding to the 4 4 D-prity code bsed FT rry of Fig. 8 is shown in Fig. 8b nd, in the generl cse, is described s follows. For ech recon-switch, there is vertex in G, nd, between recon-switches in the sme or different processors tht re linked, there re two oppositely directed edges to model reconfigurtion in either direction between the corresponding switches. Furthermore, there is n edge from the topmost ((d g - 1)th) to the bottommost (zeroth) reconswitch vertex of processor. The zeroth recon-switch vertex of fulty processor is unit-surplus source denoted s. Finlly, there is n edge from ech vertex corresponding to recon-switches linked to spres, to sink vertex t. All edges hve unit cpcity. Note tht, since the edges between recon-switch vertices of processor hve unit cpcity, the subgrph corresponding to processor models the contention for v-links when reconfigurtion pths bend, s pointed out in Section 3. Any flow pth from source to sink in G obtined by mxflow lgorithm (e.g., in Fig. 8b) hs direct correspondence with reconfigurtion pth from fult to spre in the ctul hrdwre (e.g., in Fig. 8). It is cler tht fult set F will be reconfigurble if nd only if there is flow of vlue F in G. We first used the O( V 3 ) lift-to-front preflow-push lgorithm of [9] to determine if mtching exists between the set of fults (of size F ) nd the set of ssoci-

12 1008 IEEE TRANSACTIONS ON COMPUTERS, VOL. 46, NO. 9, SEPTEMBER 1997 Fig. 8. () A four-dimensionl FT hypercube bsed on the D-prity code showing settings of recon-switches determined by the reconfigurtion lgorithm of Section 6 to tolerte fult pttern. (b) The flow grph G corresponding to (), long with the reconfigurtion pths used for the different fults. (c) The simpler grph G derived from G, showing fult enclosures A nd B both of cpcity 14. (d) The flow grph G used in determining DFT cpbilities corresponding to the fult enclosure A in G. ted spres (of size O(g)) this tkes O(( F + g) 3 ) time. 5 Only if this filed did we solve the mxflow problem ssocited with G using the bove lgorithm in O(N 3 ) time. For moderte number of fults (e.g., for F <= 40 for 1,04- processor D-prity code bsed FT rry using g = 64 spres), mtching exists in substntil number of cses ( 75 percent of the cses in the bove exmple), so tht the reconfigurtion time is reduced pprecibly. 7 DETERMINISTIC FAULT TOLERANCE (DFT) CAPABILITIES 7.1 Preliminries In this section, we derive the DFT cpbility, defined s the mximum number of fults gurnteed reconfigurble, for FT multiprocessors, bsed on the vrious codes, by determining tolerble worst-cse fult pttern of mximum size we cll these DFT fult ptterns. For given fult size, the worst-cse fult pttern will be clustered rther thn dispersed, since, in the former cse, the cpcity of minimum source-sink cut 6 in G is no more thn in the ltter cse, nd, from the well-known mxflow-mincut theorem, this cpcity gives the mount of flow possible, i.e., it gives the number of tolerble fults. We will be computing DFT cpbilities t the corner nd interior of the vrious FT lyouts s follows: For the single-spre-per-group design, the corner DFT cpbility is computed t the corner of the ly- 5. One could lso use the O( V E ) -time biprtite mtching lgorithm of [0], which, since there re O(g) edges in the biprtite grph, would run in O(( F +g) 1.5 ) time. 6. A source-sink cut (S, T) in G = (V, E) is prtition of V into S nd T = V - S, such tht s ΠS nd t ΠT. The cpcity of cut (S, T) is the totl cpcity of ll edges from vertex in S to vertex in T. out tht is not djcent to spres (e.g., loction of processor (0, 0) in Fig. 6 nd (1, 0) in Fig. 7), which is the worst cse; in the double-spre cse, the prticulr corner is immteril. Also, the interior DFT cpbility in the single-spre cse is computed by considering the DFT fult pttern to be sufficiently wy from nonspre boundries of the lyout so tht reconfigurtion pths coming out of the DFT fult pttern on sides fcing these boundries cn bend to ccess spres on the other boundries (s in the cse of the reconfigurtion pth for processor (1, ) in Fig. 8); in the doublespre cse, there is no such considertion, since there re spres on ll sides. Note tht the number of corner DFT fult ptterns will be constnt, since there re constnt number of corners, while the number of interior DFT fult ptterns for both single- nd double-spre cses will increse with N, since the region over which such fult ptterns re considered grows with N. Therefore, the likelihood of n interior reltive to corner DFT fult pttern will increse likewise with N. Next, to determine the DFT fult pttern, we consider the mximum flow in simpler flow grph G derived in two steps from the flow grph G of Section 6 s follows. First, flow grph G = (V, E ) is obtined from G by replcing the processor subgrphs in G by single vertex s illustrted in Fig. 8c. We lso replce the two oppositely directed edges between recon-switch vertices of djcent processors in G by single undirected edge in G. Define fult enclosure in G s miniml cut-set 7 whose removl disconnects vertex sets V 1 nd V, where V 1 F nd V = V - V 1, nd its cpcity s the number of edges, clled enclosure edges, it includes (see Fig. 8c). The grph G = (V, E ) corresponding to fult enclosure is tht induced by the vertex set V 1 in G. In G, every vertex on the fult enclosure 7. A miniml cut-set is miniml set of edges whose removl destroys ll pths between two mutully exclusive sets of vertices in some grph.

13 DUTT AND MAHAPATRA: NODE-COVERING, ERROR-CORRECTING CODES AND MULTIPROCESSORS WITH VERY HIGH AVERAGE FAULT TOLERANCE 1009 is connected to sink (see Fig. 8d). We will find reconfigurtion pths in this simpler grph G to determine DFT cpbilities using mxflow nd mtching concepts. From the bove discussion, it is cler tht, if F is reconfigurble in G (i.e., if F is tolerble), then it will be so in G lso. Generlly, however, the converse is not necessrily true when there re bent reconfigurtion pths in G, since processor subgrphs in G, which modeled v-link contention during bending (s pointed out in Section 3) hve been replced by single vertices in G. As we will see shortly, ll reconfigurtion pths in G in our DFT proofs re stright, nd, thus, they re vlid reconfigurtion pths in G tht re hrdwre supportble. Finlly, since boundry processor vertices connected to sinks in G my correspond to vertices of interior processors not djcent to the sink (spres) in G (e.g., processor (1, 1) in Figs. 8b nd 8d), we need to ensure tht reconfigurtion pth formed until such n interior processor in G cn be extended from it to the sink (spre) in G. Below, we identify the DFT fult pttern for the D-prity code by finding flows for the fult pttern in G tht meet this requirement, nd, thereby, determine its DFT cpbility. For the other codes, we only stte their DFT cpbilities; derivtions re similr to tht for the D-prity code nd cn be found in [16]. 7. DFT Cpbilities Associted with Vrious ECC Codes THEOREM 13. The corner DFT cpbility for the FT design bsed on the D-prity code is min(s, 5) when there is only one spre per processor group nd min(s, 10) when there re two spres per processor group, nd the interior DFT cpbility is min(s, 18), where s is the number of spres. PROOF. For the D-prity code, the lest cpcity of ny enclosure for fult pttern is equl to the perimeter of the circumscribing rectngle (see, e.g., Fig. 8c). Since of ll rectngles of the sme size, the squre hs the minimum perimeter, the worst-cse fult pttern F for ny fult size will be rectngle (, b, c, g) s close to squre s possible plus, possibly, prtil column (e, f) djcent to it s in Fig. 9, becuse, then, the fult pttern will be most difficult to tolerte. Specificlly, F will be either of the form F 1 = r + r, where 0 < r r, or of the form F = r(r + 1) + r, where 0 < r (r + 1), nd, therefore, will correspond to n r -length column djcent to either n r r squre or n r (r + 1) rectngle, respectively. To determine the corner DFT cpbility, first ssume one spre per processor group nd consider the worst-cse fult pttern of Fig. 9 to be in the left-top corner of the lyout so tht point is t the loction of processor (0, 0) in Fig. 6. Since there re no enclosure edges on sides (, b) nd (, f) (becuse there re no edges leding to spres on those sides), the minimum cpcity of ny fult enclosure will be either C 1 = r + 1 or C = r +, corresponding to F 1 nd F fults, respectively. Note tht both F i nd C i depend on just vribles r nd r. Stepping through vlues 0, 1,, º of r, nd, for ech such vlue, stepping through vlues 1 through r (for F 1 fults) or (r + 1) (for F fults) of r, we find tht the lrgest fult size F i, which does not exceed cpcity C i, i = 1,, occurs for i = 1 for F 1 = C 1 = 5 or for r = nd r = 1; denote this fult pttern by F 1 mx. Note tht, since F mx 1 is the limiting fult pttern beyond which F will exceed C, we hve lso estblished tht, for ny F Õ F mx 1, the cpcity (or perimeter) of its circumscribing rectngulr enclosure will be t lest F. If we now mtch fults to perimeter points of the circumscribing rectngle of F mx 1, then, from the bove result nd Theorem, we hve mtching, nd, hence, from Theorem 3, there exist mx stright reconfigurtion pths from ech fult in F 1 to unique perimeter point on the circumscribing rectngle. These reconfigurtion pths cn be extended further stright till spre is reched to obtin complete reconfigurtion pths for ll fults in F mx 1. Thus, the corner DFT cpbility in the single-spre cse is five or s, whichever is less. In the double-spre cse, it is esy to see tht the DFT fult pttern will be the sme s in the single-spre cse, but, with the five spres djcent to sides (, b) nd (, f) in Fig. 9 lso fulty, thus, giving DFT cpbility of min(s, 10). Finlly, consider the fult pttern of Fig. 9, locted in the interior of the lyout. In this cse, the minimum cpcity of ny enclosure is either C 1 = (r + 1) or C = (r + ), corresponding to F 1 nd F fults, respectively. Agin, stepping through vlues of r nd r s in the corner fult pttern cse bove, we find tht the limiting fult pttern corresponds to F mx 1 = 18 or to r = 4 nd r =. Similr to the corner DFT fult pttern, there exists mtching-bsed reconfigurtion pth from every fult in F 1 mx to unique perimeter point on the circumscribing rectngle. In the doublespre cse, these reconfigurtion pths cn be extended stright up to spre s before. However, in the single-spre cse, two sides of the circumscribing rectngle (sides (, b) nd (, f) in Fig. 9) will not hve spres on their side. In this cse, if the fult pttern is sufficiently in the interior, reconfigurtion pths exiting out of such sides cn bend to ccess spres on sides tht hve spres (s, for exmple, in the cse of the reconfigurtion pth for fulty processor (1, ) in Fig. 8b). Thus, s before, we cn extend reconfigurtion pths from perimeter points to spres even on sides not directly ssocited with spres. Hence, the interior DFT cpbility is min(s, 18). THEOREM 14. The corner DFT cpbility for the FT design bsed on the full-two code is min(s, 4) when there is only one spre per processor group nd min(s, 8) when there re two spres per processor group, nd the interior DFT cpbility is min(s - l, 18) in the single-spre cse nd min(s, 18) in the double-spre cse, where s is the number of spres nd l the distnce of the fult pttern from the right primryprocessor boundry.

14 1010 IEEE TRANSACTIONS ON COMPUTERS, VOL. 46, NO. 9, SEPTEMBER 1997 bsed FT designs nd compre them to previous probbilistic designs employing the sme number of spres. Then, in Section 8., we compre the hrdwre overheds of ECCbsed FT designs to previous deterministic nd probbilistic designs with similr MTTFs. Fig. 9. Shpe of worst-cse fult pttern for the D-prity code. THEOREM 15. The corner DFT cpbility for the FT design bsed on the 3D-prity code is min(s, 3) when there is only one spre per processor group nd min(s, 64) when there re two spres per processor group, nd the interior DFT cpbility is min(s, 34), where s is the number of spres. THEOREM 16. The corner DFT cpbility for the FT design bsed on the full-three code is min(s, 7) when there is only one spre per processor group nd min(s, 11) when there re two spres per processor group, nd the interior DFT cpbility is min es,, 34 ( s-l)( s- l+1) j in the single-spre cse nd min(s, 34) in the double-spre cse, where s is the number of spres nd l the horizontl distnce of the fult pttern from the right primry-processor boundry. From the bove results, it is evident tht our probbilistic FT designs hve good DFT cpbilities, in ddition to their high verge-cse fult tolernce which we will see in Section 8. Although the full-two nd full-three codes bsed FT designs hve smller corner DFT cpbilities thn those bsed on the D- nd 3D-prity codes, respectively, their interior DFT cpbilities re equl (when s is lrge enough, i.e., when N is lrge enough, or when l = 0). This is noteworthy, since the full-two nd full-three codes bsed FT designs use much smller number of spres (in fct, they hve the minimum number of spres of ny degree-two nd degree-three code bsed designs, respectively [18]) thn the D- nd 3D-prity code bsed designs, respectively. 8 RECONFIGURABILITIES AND HARDWARE OVERHEADS OF ECC-BASED AND PREVIOUS FT DESIGNS In this section, we compre our ECC-bsed probbilistic FT designs to some of the best deterministic [4], [6], [7], [8], [14], [7] nd probbilistic [6], [7], [8] FT designs proposed previously, in terms of their reconfigurbilities nd hrdwre overheds for two populr topologies, the D mesh, nd the hypercube. In [6], [7], both deterministic nd probblistic FT designs re presented; [6] gives severl probbilistic designs, but we consider here only the design, denoted M 4, with the highest reconfigurbility. Moreover, [14] gives two different deterministic designs, globlspring nd locl-spring design, tht we will shortly discuss. The FT designs of [4], [7], [8], [14], like the ECC-bsed designs, re pplicble to ny topology, while the designs of [6], [7], [8] re specific to D meshes. 8 First, in Section 8.1, we present reconfigurbility results for the vrious ECC- 8.1 Reconfigurbility Comprisons We first summrize results on the reconfigurbility of 1,04- processor FT systems derived from vrious ECCs. The reconfigurbility results were obtined using the mxflow-bsed reconfigurtion lgorithm of Section 6 nd Monte Crlo simultions verged over 1,000 smples for ech fult size from s (the number of spres) down to the lrgest fult size for which we get 100 percent reconfigurbility smller fult sizes obviously hve 100 percent reconfigurbility. A D-prity code bsed FT system using 64 spres (single-spre cse) ws found to tolerte 9.8 percent of ll 64-processor fults (the lrgest fult size tolerble), 98.8 percent of ll 63-processor fults, nd 100 percent of ll smller-sized fults; thus, k vg = In the double-spre cse, the sme FT multiprocessor ws ble to tolerte 98.3 percent of ll 18-processor fults nd 100 percent of ll smller-sized fults (k vg = 17.98). A full-two code bsed FT system using 46 spres (single-spre cse) ws ble to tolerte 94. percent of ll 46- processor fults nd 100 percent of ll fults of smller sizes (k vg = 45.94). In the double-spre cse, this FT system ws ble to tolerte 100 percent of fults of sizes 9 (mximum possible) or smller (k vg = 9). An FT system derived from the 3D-prity code hd 100 percent reconfigurbility for ll fult sizes 363 (the totl number of groups) nd less (k vg = 363); using two spres per group in this system is not prcticl even one spre per group is quite expensive hence, tht cse ws not simulted. A full-three code bsed FT system hd 100 percent reconfigurbility in both the single nd double spre cses (using 0 nd 40 spres, respectively) for ll fult sizes less thn or equl to the number of spres (k vg = 0 nd 40, respectively). Note tht the verge fult tolernce k vg for the different ECC-bsed FT multiprocessors re pproximtely equl to the number of spres s, s desired. As we sw in Section 5., FT systems derived from both the D-prity nd full-two codes hve resonbly low re overheds nd re, therefore, cost-effective to build. Moreover, their DFT cpbilities, derived in Section 7, nd verge-cse fult tolernce, noted bove, re quite high. The full-two code bsed system hs the dvntge tht it hs lower spre processor overhed nd lso, s seen bove, is ble to more effectively utilize its spres. The Dprity code bsed system, on the other hnd, hs smller wiring re overhed (see Section 5.). Therefore, the vrious codes considered here offer different trde-offs between wiring re/volume overheds, reconfigurtion cpbility (or relibility/mttf), nd spre-processor overhed. Next, in Fig. 10, we give reconfigurbility results for the D-prity code bsed design, the probbilistic design M 4 of [6], nd the three-trck probbilistic design of [8], obtined using Monte Crlo simultions verged over 1,000 smples. 9 The vrious prmeters of the probbilistic design of 8. Actully, the design of [7] is pplicble to constnt dimensionl toruses, but we consider only D meshes here. 9. The progrms for ECC-bsed designs nd tht of [8] were coded by us, while tht for [6] ws provided by one of its uthors, Dr. R. Cypher.

15 DUTT AND MAHAPATRA: NODE-COVERING, ERROR-CORRECTING CODES AND MULTIPROCESSORS WITH VERY HIGH AVERAGE FAULT TOLERANCE 1011 () (b) Fig. 10. Reconfigurbility of three-trck [8], M 4 [6], nd D-prity code bsed probbilistic designs for rrys of N = 100, 400, nd 1,04 processors for () s = N (single-spre cse) nd (b) s = 4 N (double-spre cse) spres. [7] re given only in order terms (e.g., the number of spres is Q(N)) insted of in rel terms, so tht it could not be simulted to determine its reconfigurbility; lso, becuse of the constrints tht different design prmeters need to stisfy, the design of [7] ppers to be pplicble to only very lrge systems with t lest hundreds of thousnds of processors. Note lso tht the D-prity code bsed design (nd, in generl, ny ECC-bsed design) is pplicble to ny topology, wheres the designs of [6], [8] re pplicble to only D meshes. The figure shows plots of the percentge of successful reconfigurtions verged over 1,000 runs for fult sizes one through s (the number of spres) for number of processors N = 100, 400, nd 1,04, when the number of spres is kept the sme either N (single-spre cse Fig. 10) or 4 N (double-spre cse Fig. 10b) in ll three designs. Note tht the reconfigurbility of our D-prity code bsed probbilistic design is 100 percent for ll fult sizes except s - to s. Even for these lrge-sized fults, the reconfigurbility is t lest 90 percent nd, in most cses, close to 100 percent. This is much better thn the reconfigurbility of the three-trck design of [8], which, in turn, hs much higher reconfigurbility thn the probbilistic design M 4 of [6] s seen from Fig. 10. These results lso demonstrte tht the spre-processor utiliztion in our probbilistic design is close-to-optiml, or, in other words, its verge fult tolernce is close to the number s of spres for different rry sizes. 8. Hrdwre Overhed Comprisons In Tble 1, we compre the hrdwre overheds of the Dprity nd full-two code bsed probbilistic FT designs to those of previous deterministic nd probbilistic FT designs [7], [8], [4], [14], [13] with similr MTTFs. We ssume tht individul processor systems (including processor nd ssocited routing, memory, nd other chips) hve independent filure rtes of l = filures/yer nd tht their relibility functions follow the exponentil filure lw (i.e., R = 1 - e -lt, where R is the relibility nd t is the elpsed time). 10 This corresponds to highly relible single-processor subsystems, ech with n MTTF of yers. As noted in the introduction, we re concerned here with multiprocessor systems in which it is importnt to preserve the originl structure, in spite of ny filures, for correct nd nondegrded opertion. For such requirement, single processor filure in non-ft multiprocessor results in the filure of the entire system, i.e., the filure rte of n N-processor non-ft system is Nl. The MTTF of non-ft 1,04-processor system, using processors with the bove filure rte, is esily computed to be just one month. This drmticlly underscores the need for fult tolernce in multiprocessor systems. The hrdwre overheds (specificlly, the redundnt switch complexity nd the redundnt wire re) of n FT design depend upon the number of spres s. The hrdwre overheds given in Tble 1 re those of designs with number s k of spres such tht their MTTFs re the sme s those of k-ft deterministic design. Before we compute other hrdwre overheds, we determine s k (lst column in Tble 1) for the different designs. First, consider the deterministic FT designs of [4], [6], [7], [8], [14], [7]. The designs of [4], [7], [8], [14] re k-ft deterministic globl-spring designs, i.e., they require k spres to tolerte ny k or fewer fults, hence for them s k = k. The designs of [6] nd [7] re deterministic FT designs tht require s k = k 3 + 4k nd sk = 4 3 N k + 8 k 3 spres, respectively, to be ble to toler- 10. This filure rte for processor nd its ssocited hrdwre is resonble considering tht the filure rte of just single Motorol 80C386 microprocessor chip is filures/yer [9].

16 101 IEEE TRANSACTIONS ON COMPUTERS, VOL. 46, NO. 9, SEPTEMBER 1997 TABLE 1 COMPARISON OF THE HARDWARE OVERHEADS OF N-PROCESSOR FT D MESH AND HYPERCUBE SYSTEMS BASED ON THE D-PARITY AND FULL-TWO CODE PROBABILISTIC NODE-COVERING METHODS TO THOSE BASED ON PREVIOUS DETERMINISTIC AND PROBABILISTIC FT METHODS c 1 nd c re unknown constnts. All FT systems hve the sme MTTF s n N-processor k-ft deterministic system. te ny k or fewer fults. Finlly, [14] lso gives deterministic locl-spring FT design of the type discussed in Section 3, in which ech group of N processors is k 1 -FT with exctly k 1 spres per group, or N k 1 spres in ll. A 1,04- processor deterministic locl-spring FT design with k 1 = 5 (or = 160 spres) is found to hve n MTTF of 5.71 yers, which lies between the MTTFs of deterministic globl-spring 64-FT (5.55 yers) nd 65-FT (5.333 yers) designs. By linerly interpolting between the ltter two MTTFs, we find tht 1,04-processor globl-spring FT design with =. spres will hve, t most, the sme MTTF s the bove locl-spring design with 160 spres. Assuming tht the bove reltionship between the number of spres in globl-spring design nd tht in locl-spring design with similr MTTF holds for ll N-processor systems, we conclude tht locl-spring design with sk = 160 k =. 4837k spres will hve t lest the sme MTTF s k-ft deterministic design. We found the s k vlues derived from the bove ssumption to be resonbly ccurte for different N-processor systems. For exmple, for N = 56, the MTTF of 3-FT globlspring design is pproximtely 10.3 yers compred to n MTTF of pproximtely 11.5 yers for locl-spring design with È = 80 spres, nd, for N =,500, the MTTF of 100-FT globl-spring design is pproximtely 3.4 yers compred to n MTTF of pproximtely 3.1 yers for locl-spring design with È < 50 spres. Next, we consider the probbilistic designs of [6], [8]

17 DUTT AND MAHAPATRA: NODE-COVERING, ERROR-CORRECTING CODES AND MULTIPROCESSORS WITH VERY HIGH AVERAGE FAULT TOLERANCE 1013 nd our ECC-bsed designs; the probbilistic design of [7], s pointed out in Section 8.1, could not be simulted to determine its reconfigurbility/mttf. To determine the s k s for these designs, we proceed in mnner similr to the cse of locl-spring designs bove. Thus, we find tht the MTTF M for 1,04-processor systems with s spres of the probbilistic designs of [6], [8] nd those bsed on the Dprity nd full-two codes lies between the MTTFs M nd M of k -FT nd (k + 1)-FT deterministic designs, respectively, s follows. 1) For n M 4 design of [6] with s = N = 64 spres, M = yers, k = 17, M = yers, nd M = yers; ) For three-trck design of [8] with s = N = 64 spres, M = 4.38 yers, k = 5, M = yers, nd M = yers; 3) For D-prity code bsed design with s = N = 64 spres, M = 5.48 yers, k = 63, M = yers, nd M = 5.55 yers; 4) For full-two code bsed design with s = N = 46 spres, M = 3.87 yers, k = 45, M = 3.75 yers, nd M = 3.83 yers. From the bove observtions, we conclude tht probbilistic design with s k L M s M - M k + M - M = k O P spres will hve t lest the sme MTTF s k-ft deterministic design. Thus, we obtin: 1) For n M 4 design, s k = È3.5580k ; ) For three-trck design, s k = È1.093k ; 3) For D-prity code bsed design, s k = È k ; nd 4) For full- code bsed design, s k = È k. Note tht D-prity code bsed nd other ECC-bsed designs with È k spres (or ny number of spres between one nd g for single-spre designs nd one nd g for double-spre designs), cn be obtined by combining the regulr processor groups into lrger ones nd ssociting one (or two for double-spre designs) spres with these groups. Hving obtined the spre-processor overheds s k for the different FT designs with similr MTTFs, we now turn to their wiring nd switch overheds. The hrdwre overheds of ll previous designs given in Tble 1, except those of the deterministic nd probbilistic FT D mesh designs of [6], [7], cn be found in the respective ppers (or is cler from the discussions therein) or in [14]. Discussion of lyouts nd overhed computtion of the designs of [6], [7] cn be found in [16]. Hrdwre overheds of ll designs re summrized in Tble 1. It is pprent from Tble 1 tht, for resonble-mttf systems, the probbilistic designs, except tht of [7], will incur much lower hrdwre overheds thn deterministic designs, since the ltter will require sufficiently lrge k vlues to ensure the required relibility. Note tht, mong competing deterministic designs [4], [7], [8], [14], the globlnd locl-spring node-covering designs [14] hve the lest overheds for given MTTF, with the former incurring less spre overhed nd the ltter less wiring nd switch overheds. Among probbilistic designs, the D-prity nd fulltwo code bsed designs hve much less overhed (including spre overhed) compred to previous designs with similr MTTFs, s shown in Tble 1. The wiring nd switch overheds for the D-prity code bsed probbilistic design re the sme s tht for the corresponding two-ft deterministic globl-spring node-covering design. Also, the D-prity code bsed design is better thn the locl-spring nodecovering design when the disjoint groups in the ltter need to be mde more thn two-ft to obtin resonble MTTF vlues. Finlly, the full-two code bsed design hs somewht lrger wiring overhed compred to the D-prity code bsed design, s noted in Section 5., but hs smller spre overhed. Another ttrctive feture of our probbilistic designs is tht they cn provide incresing relibility just by incresing the number of spres from one to g (or g in double-spre design), while their other hrdwre overheds remin constnt. To conclude this section, we now give specific exmple compring the hrdwre overheds of the two best previous designs tht re pplicble to rbitrry topologies, viz., the deterministic node-covering designs with globl nd locl spring [14], nd our D-prity code bsed probbilistic design. For 1,04-processor mesh nd hypercube systems with MTTFs of pproximtely five yers, the wiringre overheds f o s (computed using the sme d 4 vlues s in Section 5.) nd spre-processor overheds s o s for the bove designs re: 1) 64-FT deterministic node-covering with globl spring [14]: f o (mesh) = 384%, f o (hypercube) = 37.5%, nd s o = 6.5%; ) Deterministic node-covering with locl spring [14] in which ech disjoint processor group is five-ft: f o (mesh) = 100%, f o (hypercube) = 15.63%, nd s o = 15.63%; nd 3) D-prity code bsed probbilistic node-covering (single-spre design): f o (mesh) = 44%, f o (hypercube) = 6%, nd s o = 6.5% (see Section 5.). From the bove discussions, it is cler tht our probbilistic FT designs provide significnt reductions in hrdwre cost compred to previous designs for comprble relibilities/mttfs. 9 CONCLUSIONS Our im ws to design FT multiprocessors of ny topology with low spre-link nd switch overheds, but with very high verge fult tolernce. We developed methodology to effectively use liner ECCs to design such FT multiprocessors by prtitioning them into intersecting FT processor groups bsed on the prity groups of n ECC. We showed tht the deterministic error detectbility nd the verge ersure correctbility of liner codes re lower bounds for the deterministic nd verge fult tolernce, respectively, of the FT multiprocessor derived from them in this mnner. These lower-bound results provide us with the rtionle for using ECCs with high verge ersure correctbility, but low group degree, group size, nd check-bit overhed to

18 1014 IEEE TRANSACTIONS ON COMPUTERS, VOL. 46, NO. 9, SEPTEMBER 1997 design FT multiprocessors the ltter metrics ffect its hrdwre overhed. Such ECCs were identified in [18] to be the D-prity, 3D-prity, full-two, nd full-three codes, nd were used to design redundnt disk rrys or RAIDs. We used these ECCs to design FT multiprocessors nd developed efficient lyout strtegies for them. Our results show tht, indeed, the resulting designs hve very high verge fult tolernce (k vg < s) or relibility with very smll (six to nine percent) to resonble (44-65 percent) re/volume overheds. Another ttrctive feture of our designs is tht the processor degree in the FT system is the sme s tht in the non-ft system. We lso showed tht the deterministic fult tolernces of these designs re quite high. The bove codes lso offer vrious trde-offs between spre processor overhed, lyout re or wiring overhed, nd verge fult tolernce, s specified erlier. The code whose trdeoffs mtch the cost constrints nd relibility gols of the trget system cn be used to design the FT multiprocessor. Finlly, we compred our probbilistic FT designs to some of the best deterministic nd probbilistic designs proposed in the pst, nd showed tht our designs cn meet given men-time-to-filure (MTTF) specifiction t much lower hrdwre costs (switch complexity, redundnt wiring re, nd spre-processor overhed) thn previous designs. Further, for given number of spre processors, our designs were shown to hve close-to-optiml reconfigurbilities tht re much better thn those of previous probbilistic designs. In conclusion, we believe tht the methodology presented here offers very cost-effective technique to build highly-vilble nd highly-relible multiprocessor systems of ny topology tht cn be used in environments rnging from misson- or life-criticl systems to business trnsction processing. ACKNOWLEDGMENTS We thnk Dr. R. Cypher for providing us the code to simulte the probbilistic FT mesh design M 4 of [6]. We lso thnk the referees for their helpful comments. This reserch ws supported by U.S. Ntionl Science Foundtion grnt MIP REFERENCES [1] B. Arzi, A Commonsense Approch to the Theory of Error-correcting Codes. MIT Press, [] P. Bnerjee, S.Y. Kuo, nd W.K. Fuchs, Reconfigurble Cube- Connected Cycles Architecture, Proc. 16th Fult Tolernt Computing Symp., pp , June [3] R.A. Bruldi, Introductory Combintorics, pp New York: North Hollnd, [4] J. Bruck, R. Cypher, nd C.-T. Ho, Wildcrd Dimensions, Coding Theory nd Fult-Tolernt Meshes nd Hypercubes, Proc. 3rd Fult Tolernt Computing Symp., pp , June [5] J. Bruck, R. Cypher, nd C.-T. Ho, Fult-Tolernt Meshes nd Hypercubes with Miniml Number of Spres, IEEE Trns. Computers, vol. 4, no. 9, pp. 1,089-1,104, Sept [6] J. Bruck, R. Cypher, nd C.-T. Ho, Fult-Tolernt Meshes with Smll Degree, Proc. ACM Symp. Prllel Algorithms nd Architectures, pp. 1-10, [7] S.-C. Chu nd A.L. Liestmn, A Proposl for Fult-Tolernt Binry Hypercube, Proc. 19th Fult Tolernt Computing Symp., pp , Chicgo, June [8] F.R.K. Chung, F.T. Leighton, nd A.L. Rosenberg, Diogenes: A Methodology for Designing Fult-Tolernt VLSI Processor Arrys, Proc. 13th Fult Tolernt Computing Symp., pp. 6-31, June [9] T.H. Cormen, C.E. Leiserson, nd R.L. Rivest, Introduction to Algorithms. McGrw-Hill, [10] S. Dutt nd J.P. Hyes, An Automorphic Approch to the Design of Fult-Tolernt Multiprocessors, Proc. 19th Fult Tolernt Computing Symp., pp , Chicgo, June [11] S. Dutt nd J.P. Hyes, On Designing nd Reconfiguring k-fult- Tolernt Tree Architectures, IEEE Trns. Computers, vol. 39, no. 4, pp , Apr [1] S. Dutt nd J.P. Hyes, Designing Fult-Tolernt Systems Using Automorphisms, J. Prllel nd Distributed Computing, pp , July [13] S. Dutt nd J.P. Hyes, A Locl-Spring Design Methodology for Fult-Tolernt Multiprocessors, to pper in the Specil Issue on Grph Theory in Computer Science, Chemistry, nd Other Fields of Computers nd Mthemtics with Applictions, Elsevier Science. [14] S. Dutt nd J.P. Hyes, Some Prcticl Issues in the Design of Fult-Tolernt Multiprocessors, IEEE Trns. Computers, vol. 41, no. 5, pp , My 199. [15] S. Dutt, Fst Polylog-Time Reconfigurtion of Structurlly Fult- Tolernt Multiprocessors, Proc. Fifth IEEE Symp. Prllel nd Distributed Processing, pp , Dec [16] S. Dutt nd N.R. Mhptr, Node-covering, Error-correcting Codes nd Multiprocessors with Very High Averge Fult Tolernce, technicl report, Univ. of Minnesot, Minnepolis, 1996 ccessible t ftp site: ftp-mount.ee.umn.edu/pub/fculty/dutt/ftcomput/multiproc-ft/ieee-tc-rep96.ps.z. [17] S.H. Friedberg, A.J. Insel, nd L.E. Spence, Liner Algebr. Englewood Cliffs, N.J.: Prentice Hll, [18] G.A. Gibson, L. Hellerstein, R.M. Krp, R.H. Ktz, nd D.A. Ptterson, Filure Correction Techniques for Lrge Disk Arrys, Proc. ASPLOS 89, pp , [19] J.P. Hyes, A Grph Model for Fult Tolernt Computing Systems, IEEE Trns. Computers, vol. 5, no. 9, pp , Sept [0] J.E. Hopcroft nd R.M. Krp, An n 5/ Algorithm for Mximum Mtching in Biprtite Grphs, SIAM J. Computing, vol., pp. 5-31, [1] S.Y. Kung, VLSI Arry Processors. Prentice Hll, [] F. Lombrdi, M.G. Smi, nd R. Stefnelli, Reconfigurtion of VLSI Arrys by Covering, IEEE Trns. Computers, vol. 8, no. 9, pp , Sept [3] C.S. Rghvendr, A. Avizienis, nd M.D. Ercegovc, Fult Tolernce in Binry Tree Architectures, IEEE Trns. Computers, vol. 33, no. 6, pp , June [4] D.A. Rennels, On Implementing Fult-Tolernce in Binry Hypercubes, Proc. 16th Fult Tolernt Computing Symp., pp , June [5] V.P. Roychowdhury, J. Bruck, nd T. Kilth, Efficient Algorithms for Reconfigurtion in VLSI/WSI Arrys, IEEE Trns. Computers, vol. 39, no. 4, pp , Apr [6] M. Smi nd R. Stefnelli, Reconfigurble Architectures for VLSI Processing Arrys, Proc. IEEE, vol. 74, no. 5, pp. 71-7, My [7] H. Tmki, Construction of the Mesh nd the Torus Tolerting Lrge Number of Fults, Proc. ACM Symp. Prllel Algorithms nd Architectures, pp , [8] T.A. Vrvrigou, V.P. Roychowdhury, nd T. Kilth, Reconfiguring Processor Arrys Using Multiple-Trck Models: The 3-Trck-1- Spre-Approch, IEEE Trns. Computers, vol. 4, no. 11, pp. 1,81-1,93, Nov [9] L. Weil, M. Pecht, nd E. Hkim, Relibility Evlution of Plstic Encpsulted Prts, IEEE Trns. Relibility, vol. 4, no. 4, pp , Dec

19 DUTT AND MAHAPATRA: NODE-COVERING, ERROR-CORRECTING CODES AND MULTIPROCESSORS WITH VERY HIGH AVERAGE FAULT TOLERANCE 1015 Shntnu Dutt (S 87-M 90) received the BE degree in electronics nd communiction engineering from the M.S. University of Brod, Indi, in 1983, the MTech degree in computer engineering from the Indin Institute of Technology, Khrgpur, Indi, in 1984, nd the PhD degree in computer science nd engineering from the University of Michign, Ann Arbor, in In , he ws reserch nd development engineer t CMC Ltd., Secunderbd, Indi. He is currently n ssocite professor in the Deprtment of Electricl Engineering nd Computer Science t the University of Illinois t Chicgo. He ws previously t the Deprtment of Electricll Engineering, University of Minnesot, Twin Cities. He ws wrded Ntionl Merit Scholrship by the Government of Indi, University Fellowship by the M.S. University of Brod, Rckhm Predoctorl Fellowship by the University of Michign, nd Reserch Initition Awrd by the U.S. Ntionl Science Foundtion. His current technicl interests include CAD for VLSI circuits, prllel nd distributed computing, fult-tolernt computing, nd computer rchitecture. He hs published more thn 30 ppers journl nd refereed conference ppers in ll these res. He received Best-Pper Awrd t the Design Automtion Conference, On the occsion of the 5th nniversry of the Fult-Tolernt Computing Symposium (FTCS) in 1995, one of his ppers (published in FTCS 88) in the re of fult-tolernt multicomputers ws selected to be mong the most influentil ppers published over the lst 5 yers in FTCS. He hs been session chir t the Interntionl Conference on Supercomputing, 1996, nd t the sclbility workshop t IEEE Prllel Processing Symposium, He is on the progrm committee of the Fult-Tolernt Computing Symposium, 1997/1998, nd ws on n U.S. Ntionl Science Foundtion pnel for selecting CAREER wrds. He is member of the IEEE, the IEEE Computer Society, nd the ACM Specil Interest Groups on Computer Architecture nd Design Automtion. Nihr R. Mhptr (S 91-M 96) received the BTech degree in electricl engineering from the Indin Institute of Technology, Delhi, Indi, in 1990, nd the MS nd PhD degrees in electricl engineering, both with minor in computer nd informtion science, from the University of Minnesot, Twin Cities, in 1993 nd 1996, respectively. He is currently n ssistnt professor in the Deprtment of Electricl nd Computer Engineering t the Stte University of New York t Bufflo. His reserch interests include prllel nd distributed processing, computer rchitecture nd networking, VLSI, nd fult-tolernce. He is on the progrm committee of the Interntionl Conference on Computer Design, He is member of the IEEE Computer Society, nd the ACM Specil Interest Group on Computer Architecture.