Deterministic Voting in Distributed Systems Using Error-Correcting Codes

Transcription

1 IEEE TRASACTIOS O PARALLEL AD DISTRIBUTED SYSTEMS, VOL. 9, O. 8, AUGUST Deterinistic Voting in Distributed Systes Using Error-Correcting Codes Lihao Xu and Jehoshua Bruck, Senior Meber, IEEE Abstract Distributed voting is an iportant proble in reliable coputing. In an Modular Redundant (MR) syste, the coputational odules execute identical tasks and they need to periodically vote on their current states. In this paper, we propose a deterinistic ajority voting algorith for MR systes. Our voting algorith uses error-correcting codes to drastically reduce the average case counication coplexity. In particular, we show that the efficiency of our voting algorith can be iproved by choosing the paraeters of the error-correcting code to atch the probability of the coputational faults. For exaple, consider an MR syste with 31 odules, each with a state of bits, where each odule has an independent coputational error probability of In this MR syste, our algorith can reduce the average case counication coplexity to approxiately copared with the counication coplexity of 31 of the naive algorith in which every odule broadcasts its local result to all other odules. We have also ipleented the voting algorith over a network of workstations. The experiental perforance results atch well the theoretical predictions. Index Ters MR syste, counication coplexity, ajority voting, error-correcting codes, MDS code. F 1 ITRODUCTIO D ISTRIBUTED voting is an iportant proble in the creation of fault-tolerant coputing systes, e.g., it can be used to keep distributed data consistent, to provide utual exclusion in distributed systes. In an Modular Redundant (MR) syste, when the coputational odules execute identical tasks, they need to be synchronized periodically by voting on the current coputation state (or result, and they will be used interchangeably hereafter), and then all odules set their current coputation state to the ajority one. If there is no ajority result, then other coputations are needed, e.g., all odules recopute fro the previous result. This technique is also an essential tool for task-duplication-based checkpointing [1]. In distributed storage systes, voting can also be used to keep replicated data consistent. Many aspects of voting algoriths have been studied, e.g., data approxiation, reconfigurable voting, and dynaic odification of vote weights, etadata-based dynaic voting [3], [5], [9]. In this paper, we focus on the counication coplexity of the voting proble. Several voting algoriths have been proposed to reduce the counication coplexity [4], [7]. These algoriths are nondeterinistic because they perfor voting on signatures of local coputation results. Recently, oubir and ussbauer [8] proposed a ajority voting schee based on errorcontrol codes: Each odule first encodes its local result into a codeword of a designed error-detecting code and sends part of the codeword. By using the error-detecting code, discrepancies of the local results can be detected with soe ²²²²²²²²²²²²²²²² The authors are with the Electrical Engineering Departent, California Institute of Technology, Mail Code , Pasadena, CA E-ail: {lihao, bruck}@paradise.caltech.edu. Manuscript received 1 Jan. 1998; revised 30 June For inforation on obtaining reprints of this article, please send e-ail to: tpds@coputer.org, and reference IEEECS Log uber probability and, then, by a retransission of full local results, a ajority voting decision can be ade. Though the schee drastically reduces the average case counication coplexity, it can still fail to detect soe discrepancies of the local results and ight reach a false voting result, i.e., the algorith is still a probabilistic one. In addition, this schee only uses the error-detecting capabilities of codes. As this paper will show, in general, using only error-detecting codes (EDC) does not help to reduce counication coplexity of a deterinistic voting algorith. Though they have been used in any applications such as reliable distributed data replication [1], error-correcting codes (ECC) have not been applied to the voting proble. For any applications [1], deterinistic voting schees are needed to provide ore accurate voting results. In this paper, we propose a novel deterinistic voting schee that uses error-correcting/detecting codes. The voting schee generalizes known siple deterinistic voting algoriths. Our ain contributions related to the voting schee include: 1) using the correcting in addition to the detecting capability of codes (only the detection was used in known schees) to drastically reduce the chances of retransission of the whole local result of each node, thus the counication coplexity of the voting, ) a proof that our schee provably reaches the sae voting result as the naive voting algorith in which every odule broadcasts its local result to all other odules, and 3) the tuning of the schee for optial average case counication coplexity by choosing the paraeters of the error-correcting/detecting code, thus aking the voting schee adaptive to various application environents with different error rates. The paper is organized as follows: In Section, we describe the ajority voting proble in MR systes. Our /98/$ IEEE

2 814 IEEE TRASACTIOS O PARALLEL AD DISTRIBUTED SYSTEMS, VOL. 9, O. 8, AUGUST 1998 voting algorith, together with its correctness proof, are described in Section 3. Section 4 analyzes both the worst case and the average case counication coplexity of the algorith. Section 5 presents experiental results of perforances of the proposed voting algorith, as well as two other siple voting algoriths for coparison. Section 6 concludes the paper. THE PROBLEM DEFIITIO In this section, we define the odel of the MR syste and its counication coplexity. Then, we address the voting proble in ters of the counication coplexity..1 MR Syste Model An MR syste consists of coputational odules which are connected via a counication ediu. For a given coputational task, each odule executes a sae set of instructions with independent coputational error probability p. The counication ediu could be a bus, a shared eory, a point-to-point network, or a broadcast network. Here, we consider the counication ediu as a reliable broadcast network, i.e., each odule can send its coputation result to all other odules with only one error-free counication operation. The syste evolution is considered to be synchronous, i.e., the voting process is round-based.. Counication Coplexity The counication coplexity of a task in an MR syste is defined as the total nuber of bits that are sent through the counication ediu in the whole execution procedure of the task. In a broadcast network, let ij be the nuber of the bits that the ith odule sends at the jth round of the execution of a task, then the counication coplexity of K the task is i= 1 j = 1 ij, where is the nuber of the odules in the syste and K is the nuber of rounds needed to coplete the task..3 The Voting Proble ow, consider the voting function in an MR syste. In an MR syste, in order to get a final result for a given task, after each odule copletes its own coputation separately, it needs to be synchronized with other odules by voting on the result. Denote X i as the local coputational result of the ith odule, the ajority function is defined as follows: % &K 'K Majority X1, K, X : X if i, K, i : X = L = X φ i i1 i + 1 otherwise, 7 = where in general, is an odd natural nuber and f is any predefined value different fro all possible coputing results. EXAMPLE 1. If X 1 = 0000, X = 0001, X 3 = 0100, X 4 = 0000, X 5 = 0000, then Majority(X 1, X, X 3, X 4, X 5 ) = 0000; if X 5 changes to 0010 and other X i s reain unchanged, then Majority(X 1, X, X 3, X 4, X 5 ) = f. The result of voting in an MR syste is that each odule gets Majority(X 1, L, X ) as its final result, where X i (i = 1, L, ) is the local coputation result of the ith odule. One obvious algorith for the voting proble is that, after each odule coputes the task, it broadcasts its own result to all the other odules. When a odule receives all other odules results, it siply perfors the ajority voting locally to get the result. The algorith can be described as follows: Algorith 1 Send-All Voting For Module P i, i [1 : ]: Broadcast(X i ); Wait Until Receive all X j, j [1 : ]\i; X := Majority(X 1, L, X ); Return(X); This algorith is siple: Each odule only needs one counication (i.e., broadcast) operation but, apparently, its counication coplexity is too high. If the result for the task has bits, then the counication coplexity of the algorith is bits. In ost cases, the probability of a odule having a coputational error is rather low, naely, at ost ties all odules shall have the sae result, thus, each odule only needs to broadcast part of its result. If all the results are identical, then each odule shall agree with that result. If not, then odules can use Algorith 1. aely, we can get another siple iproved voting algorith as follows: Algorith Siple Send-Part Voting For Module P i, i [1 : ]: Partition the local result X i into sybols: X i [1 : ]; Broadcast(X i [i]); Wait Until Receive all X j [j], j [1 : ]\i; X := X 1 [1] * L * X []; F i := (X = X i ); Broadcast(F i ); If Majority(F 1, L, F ) = TRUE Return(X); Else Broadcast(X i [j]), j [1 : ]\i; Wait Until Receive all X j, j [1 : ]\i; Return(Majority(X 1, L, X )); In the above algorith, * is a concatenation operation of strings, e.g., 000*100 = ,and = is an equality evaluation: 0X Y5 : TRUE = = % & ' FALSE if X equals to Y otherwise. Soe padding ay be needed if the local result is not an exact ultiple of. The following exaple deonstrates a rough coparison of the two algoriths. EXAMPLE. X 1 = X = X 3 = X 4 = 00000, X 5 = 10000, with Algorith 1, one round of counication is needed and, in total, 5 bits are transitted. On the other hand, with Algorith, P i s (i = 1, L, 5) all broadcast 0, and X = 00000, thus (F 1, L, F 5 ) = 11110, so Majority(F 1, L, F 5 ) = 1, and X is the ajority voting result. In this case, two rounds of counication are done, and 10 bits (5 bits for X and 5 bits for F) are transitted.

3 XU AD BRUCK: DETERMIISTIC VOTIG I DISTRIBUTED SYSTEMS USIG ERROR-CORRECTIG CODES 815 If X 5 = 00001, and all other X i s reain the sae, then, with Algorith, X = 00001, which results in (F 1, L, F 5 ) = 00001, thus Majority(F 1, L, F 5 ) = 0, and the Else part of the algorith is executed; finally, the ajority voting result is obtained by voting on all the X i s, i.e., X = Majority(X 1, L, X 5 ) = ow three rounds of counication are needed and, in total, 30 bits (5 bits for X i s and 5 bits for F) are transitted. Fro the above exaple it can be observed: 1) Algorith 1 always requires only one round of counication and Algorith requires two or three rounds of counication; ) The Else part of Algorith is actually Algorith 1; 3) The counication coplexity of Algorith 1 is always, but the counication coplexity of Algorith ay be + or +, depending on the X i s; 4) In Algorith, by broadcasting and voting on the voting flags, i.e., F i s, the chance for getting a false voting result is eliinated. If the Else part of Algorith, i.e., Algorith 1, is not executed too often, then the counication coplexity can be reduced to + fro and, in ost cases,, thus +. So, the key idea used to reduce the counication coplexity is to reduce the chance to execute Algorith 1. In ost coputing environents, each odule has low coputational error probability p, thus, ost probably all odules either 1) get the sae result or ) only few of the get different results fro others. In Case 1, Algorith has low counication coplexity, but, in Case, Algorith 1 is actually used and the counication coplexity is high (i.e., + ), but if we can detect and correct these discrepancies of the inor odules results, then the Else part of the Algorith does not need to be executed, the counication coplexity can still be low. This detecting and correcting capability can be achieved by using error-correcting codes. 3 A SOLUTIO BASED O ERROR-CORRECTIG CODES Error-correcting codes (ECC) can be used in the voting proble to reduce the counication coplexity. The basic idea is that instead of broadcasting its own coputation result X i directly, P i, the ith odule, first encodes its result X i into a codeword Y i of soe code and, then, broadcasts one sybol of the codeword to all other odules. After receiving all other sybols of the codeword, it reassebles the into a vector again. If all odules have the sae result, i.e., all X i s are equal, then the received vector is the codeword of the result, thus it can be decoded to the result again. If the ajority result exists, i.e., Majority(X 1, L, X ) f, and there are t (t ) odules whose results are different fro the ajority result X, then the sybols fro all these odules can be regarded as error sybols with respect to the ajority result. As long as the code is designed to correct up to t errors, these error sybols can be corrected to get the codeword corresponding to the ajority result, thus Algorith 1 does not need to be executed. When the code length is less than, the counication coplexity is reduced copared to Algorith 1. On the other hand, if only error-detecting codes are used, once error results are detected, Algorith 1 still needs to be executed and, thus, increases the whole counication coplexity of the voting. Thus, error-correcting codes are preferable to errordetecting codes for voting. By properly choosing the errorcorrecting codes, the counication coplexity can always be lowered than that of Algorith 1. But, it is possible that the ajority result does not exist, i.e., Majority(X 1, L, X ) = f, yet the vector that each odule gets can still be decoded to a result. As observed fro the above exaple, introduction of the voting flags can avoid this false result. 3.1 A Voting Algorith with ECC With a properly designed error-correcting code which can detect up to d and correct up to t error sybols (0 t d), a coplete voting algorith using this code is as follows: Algorith 3 ECC Voting For Module P i, i [1 : ]: Y i := Encode(X i ), partition Y i into sybols: Y i [1 : ]; Broadcast(Y i [i]); Wait Until Receive all Y j [j], j [1 : ]\i; Y := Y 1 [1] * L * Y []; If Y is undecodable Execute Algorith 1 ; Else X := Decode(Y); F i := (X = X i ); Broadcast(F i ); If Majority(F 1, L, F ) = TRUE(1) Return(X); Else Execute Algorith 1; otice that to execute Algorith 1, each odule P i does not need to send its whole result X i. It only needs to send additional - (d + t) - 1 sybols of its codeword Y i. Since the code is designed to detect up to d and correct up to t sybols, it can correct up to d + t erasures, thus the unsent d + t sybols of Y i can be regarded as erasures and recovered, hence, the original X i can be decoded fro Y i. To see the algorith ore clearly, the flow chart of the algorith is given in Fig. 1, and the following exaple shows how the algorith works: EXAMPLE 3. There are five odules in an MR syste, and the task result has 6 bits, i.e., = 5 and = 6. Here, the EVEODD code [] is used, which divides 6-bit inforation into three sybols and encodes inforation sybols into a five-sybol codeword. This code can correct one error sybol, i.e., d = t = 1. ow, if X i = , i = 1,, 3, 4, and X 5 = , then, with the EVEODD code, Y i = , i = 1,, 3, 4, and Y 5 = ; after each odule broadcasts one sybol (i.e., bits) of the codewords, the reassebled vector is Y = Since Y has only

4 816 IEEE TRASACTIOS O PARALLEL AD DISTRIBUTED SYSTEMS, VOL. 9, O. 8, AUGUST 1998 Fig. 1. Flow chart of Algorith 3. one error sybol, it can be decoded into X = Fro the flow chart of the algorith, we can see that F i = 1, i = 1,, 3, 4, and F 5 = 0, thus, Majority(F 1, L, F 5 ) = 1, so X = is the ajority voting result. In this case, there are two rounds of counication, and the counication coplexity is 15 bits. As a coparison, Algorith 1 needs one round of counication and its counication coplexity is 30 bits; on the other hand, Algorith needs three rounds of counication and the counication coplexity in this case is 35 bits. In this exaple, the EVEODD code is used but, actually, the code itself does not affect the counication coplexity as long as it has sae properties as the EVEODD code, naely, an MDS code with d = t = 1. Fro the flow chart of the algorith, the introduction of voting on F i s ensures not reaching a false voting result, and going to the Send-All Voting in worst case guarantees not failing to reach the ajority result if it exists. Thus, the algorith can give a correct ajority voting result. A rigorous correctness proof of the algorith is as follows. 3. Correctness of the Algorith THEOREM 1. Algorith 3 gives Majority(X 1, L, X ) for a given set of local coputational results X i s (i = 1, L, ). PROOF. Fro the flow chart of the algorith, it is easy to see that the algorith terinates in the following two cases: 1) Executing the Send-All Voting algorith: The correct ajority voting result is certainly reached; ) Returning an X: In this case, since Majority(F 1, L, F ) = TRUE, i.e., ajority of X i s are equal to X, X is the ajority result. o To see how the algorith works with various cases of the local results X i s (i = 1, L, ), we give two stronger observations about the algorith which also help to analyze the counication coplexity of the algorith. OBSERVATIO 1. If Majority(X 1, L, X ) = f, then Algorith 3 outputs f, i.e., Algorith 3 never gives a false voting result. PROOF. It is easy to see fro the flow chart that, after the first round of counication, each odule gets a sae vote vector Y. According to the decodability of Y, there are two cases: 1) If Y is undecodable, then the Send-All Voting algorith is executed, and the output will be f; ) If Y is decodable, the decoded result X now can be used as a reference result. But, since there does not exist a ajority voting result, the ajority of the X i s are not equal to the X, i.e., Majority(F 1, L, F ) =

5 XU AD BRUCK: DETERMIISTIC VOTIG I DISTRIBUTED SYSTEMS USIG ERROR-CORRECTIG CODES 817 FALSE(0), so the Send-All Voting algorith is executed, and the output again will be f. o OBSERVATIO. If Majority(X 1, L, X ) = X ( f), then Algorith 3 s output is exactly the X, i.e., Algorith 3 will not iss the ajority voting result. PROOF. Suppose there are e odules whose local results are 1 different fro the ajority result X, then e. 1) If e t, then there are e error sybols in the vote vector Y with respect to the corresponding codeword of the ajority result X, so Y can be correctly decoded into X, and the ajority of X i s is equal to X, i.e., the ajority of F i s is TRUE(1), hence, the correct ajority result X is output. ) If e > t, then Y is either undecodable or incorrectly decoded into another XŠ, where XŠ X. In either case, the Send-All voting algorith is executed and the correct ajority result X is reached. o 3.3 Proper Code Design In order to reduce the counication coplexity, we need an error-correcting code which can be used in practice for Algorith 3. Consider a block code with length M. Because of the syetry aong the odules, M needs to be a ultiple of, i.e., each codeword consists of sybols and each sybol has k bits, thus M = k. If the iniu distance of the code is d in, then d in (d + t)k + 1, where 0 t d, since the code should be able to detect up to d error sybols and correct up to t error sybols [6]. Recall that the final voting result has bits, the code to design is an (k,, (d + t)k + 1) block code. To get the sallest value for k, by the Singleton Bound in coding theory [6], k (d + t)k + 1, (1) we get k 1d + t6. () Equality holds for all MDS Codes [6]. So, given a designed (d, t), the sallest value for k is d+ t. If is an integer, all MDS Codes can achieve ( ) ( d+ t) this lower bound of k. One class of coonly used MDS codes for arbitrary distances is the Reed-Soloon code [6]. If is not an integer, then any (k,, (d + t)k + 1) block ( d+ t) code can be used, where k = d + ; one of the exaples is ( t) the BCH code, which can also have arbitrary distances [6]. The exact paraeters of (k, d, t) can be obtained by shortening (i.e., setting soe inforation sybols to zeros) or puncturing (deleting soe parity sybols) proper codes [6]. 1 otice that 0 t d, thus k. In ost applications,, thus the bits of F i s can be neglected, and k is approxiately the nuber of the bits that each odule needs to send to get final voting result, so the counication coplexity of Algorith 3 is always lower than that of Algorith 1. In this paper, only the counication coplexity of voting is considered since, in any systes, coputations for encoding and decoding on individual nodes are uch faster than reliable counications aong these nodes, which need rather coplicated data anageent in different counication stacks, retransission of packets between distributed nodes when packet loss happens. However, in real applications, design of proper codes should also ake the encoding and decoding of the codes as coputationally efficient as possible. When the distances of codes are relatively sall, which is the case for ost applications when the error probability p is relatively low, ore coputation-efficient MDS codes exist, such as codes in [], [10], and [11], all of which require only bitwise exclusive OR operations. 4 COMMUICATIO COMPLEXITY AALYSIS 4.1 Main Results Fro the flow chart of Algorith 3, we can see if the algorith terinates in branch 1, i.e., the algorith gets a ajority result, then the counication coplexity is (k + 1); if it terinates in branch, then the counication coplexity is ( + 1); finally, if the algorith terinates in branch 3, the counication coplexity is, thus the worst case counication coplexity C w is ( + 1). When 1, C w. Denote C a as the average case counication coplexity of Algorith 3, and define the average reduction factor a as the ratio of C a over the counication coplexity of the Send-All Voting algorith (i.e., ), naely α = C a, then, the following theore gives the relation between a and the paraeters of an MR syste and the corresponding code: THEOREM. For an MR syste with odules, each of which executes an identical task of -bit result and has coputational error with probability p independent of other odules activities, if Algorith 3 uses an ECC which can detect up to d and correct up to t error sybols, and > 1, then the following relation holds for the average reduction factor of Algorith 3: where P1 α < d + t P P , (3) t i p i p 11 6 i. (4) = i= 0 PROOF. To get the average case counication coplexity C a of Algorith 3, we need to analyze the probability P i of the algorith terinating in the branch i, i = 1,, 3. First, assue that if a odule has an erroneous result X i, then it contributes an error sybol to the voting vector Y. Fro the proof of Observation, if the algorith terinates in the branch 1, then at ost t odules have coputational errors, thus, the probability of this event is exactly P 1. The event that the algorith reaches the branch corresponds to the decoder error event of a code with iniu distance of d + t + 1, thus [6]

6 818 IEEE TRASACTIOS O PARALLEL AD DISTRIBUTED SYSTEMS, VOL. 9, O. 8, AUGUST 1998 d+ t P = A P, (5) i i= d+ t+ 1 k= 0 where {A i } is the weight distribution of the code being used and P ik is the probability that a received vector Y is exactly Haing distance k fro a weight-i (binary) codeword of the code. More precisely, P ik = k j= 0 i k j i j p ik + 11 p6. (6) i k+ j i k j If the weight distribution of the code is unknown, P can be approxiately bounded by P d+ t i i 1 6, (7) i= 0 1 i p 1 p since the second ter in the right side of the inequality above is just the probability of event that correctable errors happen. Finally, P 3 is the probability of the decoder failure event, P 3 = 1 - P 1 - P. (8) ow, notice the fact that a odule has an erroneous result can also contribute a correct sybol to the voting vector, the average case counication coplexity is C a P 1 (k + 1) + P ( + 1) + P 3 (9) and the average reduction factor is k + α + + P P P1 P (10) otice that k = d + and P ( t) 1 + P < 1, we get the result of a as in (3). o REMARKS O THE THEOREM. Fro (3), we can see the relation between the average reduction factor a and each branch of Algorith 3. The first ter relates to the first branch whose reduction factor is k or 1 ( d+ t) when is large enough relative to, the round-off error of partition can be neglected, and P 1 is the probability of that branch. One would expect this ter to be the doinant one for the a, since, with a properly designed code tuned to the syste, the algorith is supposed to terinate at Branch 1 in ost cases. The second ter is siply the probability that the algorith terinates at either Branch or Branch 3, where the reduction factor is 1 (i.e., there is no counication reduction since all the local results are transitted), without considering the 1 bit for F i s in Branch. The last ter is due to the 1 bit for voting on F i s. When the local result size is large enough, i.e., 1, this 1 bit can be neglected in our odel. Thus, in ost applications, the result in the theore can be siplified as P1 α + 1 P1 d + t since the assuption that 1 is quite reasonable , (11) Fro the above theore and its proof, it can be seen that for a given MR syste (i.e., and p), P 1 is only a function of t, so, if t is chosen, fro (3) or (11), it is easy to see that a onotonically decreases as d decreases. Recall that 0 t d, thus, for a chosen t, setting d = t can ake a iniu when 1. Even though it is not straightforward to get a closed for of t which iniizes a, it is alost trivial to get such an optial t by nuerical calculation. Fig. shows relations between a and (t, p, ). Fig. a and Fig. b show how a (using (11)) changes with t for soe setup of (, p) when d = t. It is easy to see that, for sall p and reasonable, a sall t (e.g., t, for 51 with p = 0.01) can achieve inial a. These results show that, for a quite good MR syste (e.g., p 0.01), only by putting a sall aount of redundancy of the local results and applying error-correcting codes on the, the counication coplexity of the ajority voting can be drastically reduced. Since the ajority result is of bits and each odule shall get an identical result after the voting, the counication coplexity of the voting proble is at least bits, thus α 1, i.e., 1 is the lower bound of a. Fig. c shows the closeness of the theoretical lower bound of a and the iniu a that Algorith 3 can achieve for soe setup of MR systes. 4. More Observations Fro the above results, we can see that the counication coplexity of the Algorith 3 is deterined by the code design paraeters (d, t). In an MR syste with odules, we only need to consider the case where at ost odules have different local results with the ajority result, thus the only constraint of (d, t) is 0 t d. For soe specific values of (d, t), the algorith reduces to the following cases: 1 1) When d = t =, i.e., a repetition code is used, the algorith becoes Algorith 1. Since a repetition code is always the worst code in ters of redundancy, it should always be avoided for reducing the counication coplexity of voting. On the other hand, when d = t = 0, the algorith becoes Algorith and, fro Fig., we can see that, for a sall enough p and reasonable, e.g., p = 10-5 with = 31, Algorith actually is a best solution of the ajority voting proble in ters of the counication coplexity. Besides, Algorith has low coputational coplexity since it does not need any encoding and decoding operations. Thus, the ECC voting algorith is a generalized voting algorith and its counication coplexity is deterined by the code chosen. ) t = 0, then the code only has detecting capability but, if, then fro the analysis above, increasing d actually akes a increasing. Thus, it is not good to put soe redundancy to the local results only for detecting capability when, i.e., using only EDC (error-detecting code) does not help to reduce the counication coplexity of voting. The schee proposed in [8] is in this class with d =.

7 XU AD BRUCK: DETERMIISTIC VOTIG I DISTRIBUTED SYSTEMS USIG ERROR-CORRECTIG CODES 819 (a) (b) (c) Fig.. Relations between a and (t, p, ). (a) a vs. t for different p with fixed = 31. (b) a vs. t for different with fixed p = (c) Miniu a vs.. 3) d = : As analyzed above, in general it is not good to have d > t in ters of a since increase of d will increase a when t is fixed. But, in this case, Algorith 3 has a special property: Branch of the algorith can directly coe to declare there is no ajority result without executing the Send-All Voting algorith, siply because the code now can detect up to errors, so, if there was a ajority result, then Y (refer to Fig. 1) can have at ost erroneous odules and, since Y is decodable, the ajority of the local results should agree with the decoded result X, i.e., Majority(F 1, L, F ) = TRUE; this contradicts with the actual Majority(F 1, L, F ), so there is no ajority result. By setting d to, Algorith 3 always has two rounds of counication and the worst case counication coplexity is, thus,, instead of ( + 1), for the general case, and this achieves the lower bound of the worst case counication coplexity of the distributed ajority voting proble [8]. 5 EXPERIMETAL RESULTS In this section, we show soe experiental results of the three voting algoriths discussed above. The experients are perfored over a cluster of Intel Pentiu/Linux-.0.7 nodes connected via a 100 Mbps Ethernet. Reliable counication is ipleented by a siple iproved UDP schee: Whenever there is a packet loss, the voting operation is considered as a failure and redone fro beginning. By choosing suitable packet size, there is virtually no packet loss using UDP. To exaine real perforances of the above three voting algoriths, nodes vote on a result of length using all

8 80 IEEE TRASACTIOS O PARALLEL AD DISTRIBUTED SYSTEMS, VOL. 9, O. 8, AUGUST 1998 (a) Fig. 3. Average reduction factors (C(i ) is the experiental average reduction factor of counication tie for voting using Algorith i, and a(i) is the theoretical bound of the average counication reduction factor using Algorith i, i =, 3). (a) Error probability p = (b) Error probability p = 0.1. (b) the three voting algoriths. For the ECC Voting algorith, an EVEODD Code is used, which corrects 1 error sybol, i.e., d = t = 1 for the ECC Voting algorith. Rando errors are added to local coputing results with a preassigned error probability p, independent of results at other nodes in the MR syste. Perforances are evaluated by two paraeters for each algorith: the total tie to coplete the voting operation T and the counication tie for the voting operation C. Aong all the local Ts and Cs, the axiu T and C are chosen to be the T and C of the whole MR syste, since, if the voting operation is considered as a collective operation, the syste s perforance is deterined by the worst local perforance in the syste. For each set of the MR syste paraeters ( nodes and error probability p), each voting operation is done 00 ties and rando coputation errors in each run are independent of those in other runs, and the arithetic average of Cs and Ts are regarded as the perforance paraeters for the tested MR syste. Experiental results are shown in Figs. 3, 4, and 5. Fig. 3 copares the experiental average reduction factors of the voting algoriths with the theoretical results as analyzed in the previous section, when they are applied in an MR syste of five nodes. Fig. 4 shows the perforances (T and C) of the voting algoriths. Detailed counication patterns of the voting algoriths are shown in Fig. 5 to provide soe deeper insight into the voting algoriths. Fig. 3a and Fig. 3b show the experiental average reduction factors of the voting counication tie (C) for the Siple Send-Part Voting algorith and the ECC Voting algorith. Fig. 3a and Fig. 3b also show the theoretical average reduction factors of Algoriths and 3 as coputed fro (11). otice that the average counication tie reduction factors a of both Algorith and Algorith 3 are below 1 and, as the coputing result size increases, the reduction factor approaches the theoretical bound, with the exception of the sallest coputing result size of 1 Kbyte. Fig. 4 shows the perforances of each voting algorith applied in an MR syste of five nodes. Figs. 4a and 4b show the total voting tie T and Figs. 4c and 4d show the counication tie C for voting. The only different paraeter of the MR systes related to Figs. 4a and 4b (syetrically, Figs. 4c and 4d) is the error probability p: p = 0.1 in Figs. 4a and 4c, while p = 0.01 in Figs. 4b and 4d. It is easy to see fro the figures that, for the voting Algorith 1 (Send-All Voting), T and C are the sae since, besides counication, there is no additional local coputation. Figs. 4a and 4b show that Algoriths (Siple Send-Part Voting) and 3 (ECC Voting) perfor better than Algorith 1 (Send- All Voting) in ters of the total voting tie T. On the other hand, Figs. 4c and 4d show, in ters of C, i.e., the counication coplexity, the ECC Voting algorith is better than the Siple Send-Part Voting algorith when the error probability is relatively large (Fig. 4c) and worse than the Siple Send-Part Voting algorith when the error probability is relatively sall (Fig. 4d), which is consistent with the analysis results in the previous section.

9 XU AD BRUCK: DETERMIISTIC VOTIG I DISTRIBUTED SYSTEMS USIG ERROR-CORRECTIG CODES 81 (a) (b) (c) (d) Fig. 4. Experiental voting perforances of five-node MR syste (T(i ) and C(i ) are the total and counication tie for voting using algorith i, respectively, i = 1,, 3). (a) error probability p = 0.1., (b) error probability p = 0.01, (c) error probability p = 0.1, (d) error probability p = 0.01.

10 8 IEEE TRASACTIOS O PARALLEL AD DISTRIBUTED SYSTEMS, VOL. 9, O. 8, AUGUST 1998 (a) (b) (c) (d) Fig. 5. Detailed counication tie pattern of voting (Ri(k) is the counication tie in round i using the voting algorith k, i = 1,, 3, and k =, 3). a) error probability p = 0.01., (b) error probability p = 0.1, (c) error probability p = 0.01, (d) error probability p = 0.1.

11 XU AD BRUCK: DETERMIISTIC VOTIG I DISTRIBUTED SYSTEMS USIG ERROR-CORRECTIG CODES 83 In the analysis in the previous section, the size of local coputing result does not show up as a variable in the average reduction factor function a, since the counication coplexity is only considered as proportional to the size of the essages that need to be broadcast. But, practically, counication tie is not proportional to the essage size, since startup tie of counication also needs to be included. More especially, in the Ethernet environent, since the axiu packet size of each physical send (broadcast) operation is also liited by the physical ethernet, counication copletion tie becoes a ore coplicated function of the essage size. Thus, fro the experiental results, it can be seen that, for a coputing result of sall size, e.g., 1 Kbyte, the Send-All Voting algorith actually perfors best in ters of both C and T, since the startup tie doinates the perforance of counication. Also, the counication tie for broadcasting the 1-bit voting flags cannot be neglected, as analyzed in the previous section, particularly for a sall size coputing result. This can also be seen fro the detailed voting counication tie pattern in Figs. 5a and 5b: Round of the counication is for the 1-bit voting flag, even though it finishes in a uch ore shorter tie than round 1, but is still not negligibly sall. This explains the fact that, for sall size coputing results, the average counication tie reduction factors of Algorith and Algorith 3 are quite apart fro their theoretical bound. Further exaination of the detailed counication tie pattern of voting provides a deeper insight into Algorith 3. Fro Figs. 5c and 5d, it is easy to see that, in the first round of counication, Algorith needs less tie than Algorith 3 since the size of the essage to be broadcast is saller for Algorith. Besides, the first round of counication tie does not vary as the error probability p varies for both algoriths. The real difference between the two algoriths lies in the third round of counication. Fro Fig. 5c, this tie is sall for the both algoriths since the error probability p is sall (0.01). But, as the error probability p increases to 0.1, as shown in Fig. 5d, for Algorith, this tie also increases to be bigger than the first round tie, since it has no error-correcting capability and, once full essage needs to be broadcast, its size is uch bigger than in the first round. On the other hand, for Algorith 3, though it also increases, the counication tie for the third round is still uch saller than in the first round; this coes fro the error-correcting codes that Algorith 3 uses, since the code can correct errors at one coputing node, which is the ost frequent error pattern that happens. Thus, even though the error probability is high, in ost cases, the ost expensive third round of counication can still be avoided, and Algorith 3 perfors better (in ters of counication coplexity or tie) than Algorith in high error probability systes, just as the predicted analysis in the previous section. 6 COCLUSIOS We have proposed a deterinistic distributed voting algorith using error-correcting codes to reduce the counication coplexity of the voting proble in MR systes. We also have given a detailed theoretical analysis of the algorith. By choosing the design paraeters of the errorcorrecting code, i.e., (d, t), the algorith can achieve a low counication coplexity which is quite close to its theoretical lower bound. We have also ipleented the voting algorith over a network of workstations, and the experiental perforance results atch the theoretical analysis well. The algorith proposed here needs two or three rounds of counication. It is left as an open proble whether there is an algorith for the distributed ajority voting proble with its average case counication coplexity less than using only 1 round of counication. ACKOWLEDGMETS This work was supported in part by U.S. ational Science Foundation Young Investigator Award CCR , by the Sloan Research Fellowship, and by DARPA through an agreeent with ASA/OSAT. REFERECES [1] K.A.S. Abdel-Ghaffar and A. El Abbadi, An Optial Strategy for Coputing File Copies, IEEE Trans. Parallel and Distributed Systes, vol. 5, no. 1, Jan [] M. Blau, J. Brady, J. Bruck, and J. Menon, EVEODD: An Efficient Schee for Tolerating Double Disk Failures in RAID Architectures, IEEE Trans. Coputers, vol. 44, no., pp. 19-0, Feb [3] D.M. Blough and G.F. Sullivan, Voting Using Predispositions, IEEE Trans. on Reliability, vol. 43, no. 4, pp , [4] K. Echtle, Fault-Masking with Reduced Redundant Counication, Proc. 16th Ann. Int l Syp. Fault-Tolerant Coputing Systes, vol.16, pp , [5] D.D.E. Long and J.-F. Pâris, Voting Without Version ubers, Proc. Int l Conf. Perforance, Coputing, and Co., pp , Feb [6] F.J. MacWillias and.j.a. Sloane, The Theory of Error Correcting Codes. Asterda: orth-holland, 1977 [7] J.F. ebus, Parallel Data Copression for Fault Tolerance, Coputer Design, pp , Apr [8] G. oubir and H.J. ussbauer, Using Error Control Codes to Reduce the Counication Coplexity of Voting in MR Systes, technical report, Dept. of Coputer Science, Swiss Federal Inst. of Technology in Lausanne (EPFL), [9] B. Parhai, Voting Algoriths, IEEE Trans. Reliability, vol. 43, no. 4, pp , [10] L. Xu and J. Bruck, X-Code: MDS Array Codes with Optial Encoding, IEEE Trans. Inforation Theory, to appear, [11] L. Xu, V. Bohossian, J. Bruck, and D. Wagner, Low Density MDS Codes and Factors of Coplete Graphs, Proc IEEE Syp. Inforation Theory, Aug [1] A. Ziv and J. Bruck, Checkpointing in Parallel and Distributed Systes, Parallel and Distributed Coputing Handbook, pp McGraw-Hill, 1996.

12 84 IEEE TRASACTIOS O PARALLEL AD DISTRIBUTED SYSTEMS, VOL. 9, O. 8, AUGUST 1998 Lihao Xu received the BSc and MSc degrees in electrical engineering fro the Shanghai Jiao Tong University, China, in 1988 and 1991, respectively. Fro 1991 to 1994, he was a lecturer in the Electrical Engineering Departent of the Shanghai Jiao Tong University. He is currently a PhD candidate in the Electrical Engineering Departent of the California Institute of Technology. His current research interests include parallel and distributed coputing, fault-tolerant coputing, error-correcting codes, and server systes. He holds one pending patent. Jehoshua Bruck received the BSc and MSc degrees in electrical engineering fro the Technion, Israel Institute of Technology, in 198 and 1985, respectively, and the PhD degree in electrical engineering fro Stanford University in He is a professor of coputation and neural systes and electrical engineering at the California Institute of Technology. His research interests include parallel and distributed coputing, fault-tolerant coputing, error-correcting codes, coputation theory, and biological systes. Dr. Bruck has extensive industrial experience, including serving as anager of the Foundations of Massively Parallel Coputing Group at the IBM Aladen Research Center fro 1990 to 1994, a research staff eber at the IBM Aladen Research Center fro 1989 to 1990, and as a researcher at the IBM Haifa Science center fro 198 to Dr. Bruck is the recipient of a 1997 IBM Partnership Award, a 1995 Sloan Research Fellowship, a 1994 U.S. ational Science Foundation Young Investigator Award, a 199 IBM Outstanding Innovation Award for his work on Haronic Analysis of eural etworks, and a 1994 IBM Outstanding Technical Achieveent Award for his contributions to the design and ipleentation of the SP-1, the first IBM scalable parallel coputer. He has published ore than 130 journal and conference papers in his areas of interests and he holds 1 patents. Dr. Bruck is a senior eber of the IEEE and a eber of the editorial board of the IEEE Transactions on Parallel and Distributed Systes.