On the Relationships of Faults for Boolean Specification Based Testing

Transcription

1 On the Relationships of Faults for Boolean Specification Based Testing M. E LAU* School of Information Technology Swinbume University of Technology John Street, Hawthom 3122, AUSTRALIA edmonds@it.swin.edu.au Y. T. YU Department of Computer Science City University of Hong Kong Tat Chee Avenue, Kowloon Tong, HONG KONG csytyu@cityu.edu.hk Abstract Various methods of generating test cases based on Boolean specijications have previously been proposed. These methods are fault-based in the sense that test cases are aimed at detecting particular types of faults. Empirical results suggest that these methods are good at detecting particular types of faults. However, there is no information on the ability of these test cases in detecting other types of faults. This paper summarizes the relationships of faults in a Boolean expression in the form of a hieramhy. A test case that detects the faults at the lower level of the hierarchy will always detect the faults at the upper level of the hierarchy. The hierarchy helps us to better understand the relationships of faults ina Boolean expression, and hence to select fault-detecting test cases in a more systematic and eflcient mannel: Keywords: Boolean expression, fault class analysis, specification based testing, verification 1 Introduction Software testing aims at detecting software faults that are the result of human errors during software development. Specification based testing derives test cases from the specification rather than the actual implementation. A fault-based approach of generating test cases selects test cases that aim at detecting certain types of faults. Although a similar approach has also been used in the testing of hardware, the main focus in software testing is on human errors whereas in hardware testing the focus is mainly on manufacturing flaws [7, 121. In the past, many fault-based approaches on selecting test cases from the specifications have been proposed [2, 3, 6, 8, 9, 11, 121. Various types of faults are considered, such as expression negation fault (an * Corresponding author. edmondsgit. swin. edu. au expression being negated), variable negation fault (a variable being negated), variable reference fault (a variable being replaced by another variable or a constant), operator reference fault (an operator being replaced by another operator), and literal insertion fault (a literal being inserted into a term of the expression). Detailed discussions on the types of faults will be presented in later sections. Empirical results indicate that the proposed test case generation methodologies are quite effective in detecting many types of faults [l, 8, 10, 121. Recently, Kuhn has discovered certain relationships among several classes of faults [7]. The theoretical results show that any test that reveals a variable reference fault can also reveal the corresponding variable negation fault, and any test that reveals a variable negation fault can also reveal the corresponding expression negation fault. With this knowledge, test cases should be generated first that aim at detecting variable reference faults. However, it should be noted that, in Kuhn's study, the variable reference (negation) fault results in every occurrence of a variable being replaced (negated) by another variable. This leads to the following open questions: 1. Will the results hold if only one occurrence of a variable is being replaced or negated? In general, it is more difficult to reveal the software faults if only one occurrence of a variable is wrongly implemented. What are the relationships among other types of faults? With the knowledge of the relationships of more types of faults, fault-detecting test cases can be selected in a more systematic and efficient manner. In this paper, we attempt to provide some answers to these two questions. We study the relationship among seven types of faults that are related to the operands in a Boolean expression. Our results can be summarized by a hierarchy showing the relationships of these faults (see Section 4) /01 $ IEEE 21

2 A test case that detects the faults at the lower level of the hierarchy will always detect the faults at the upper level of the hierarchy. We also demonstrate how the theoretical results can be applied in the domain of mutation testing. Section 2 of this paper introduces the notation and terminology used. It also describes the seven types of faults that are related to the operands in a Boolean expression. In Section 3, we discuss previous work related to this research. Section 4 studies the theoretical relationships among the types of faults discussed in this paper. Section 5 discusses the application of the results to mutation testing, and some related issues. Section 6 summarizes and concludes the paper. 2 Preliminaries 1 Notation and Terminology We first introduce the notation and terminology used in this paper. We use -, + and - to denote the Boolean operators AND, OR and NOT, respectively. Moreover, we will omit the - symbol whenever it is clear from the context. The truth values TRUE and FALSE are denoted by 1 and 0, respectively. We use IB to denote the set of all truth values, that is B = (0, I}. The n-dimensional Boolean space is denoted by Bn. In a Boolean expression, a variable may occur as a positive literal or a negative literal. For example, for the same variable a in the Boolean expression a6 + &, a and H are positive literal and negative literal, respectively. Let S be a Boolean expression having n variables in disjunctive normal form. In general, S = p pm (1) where m is the number of terms in S and pi (i = 1,..,.. m) denotes the i-th term of S. Moreover, let pi = Y1...Yki where ki is the number of literals in pi, and x: (j = 1,..., ki) denotes the j-th literal in pi. A test case for testing the Boolean expression S is an element in the n-dimensional Boolean space F. 2 Types of Faults We now describe the types of faults related to the operands in the Boolean expression that may occur during implementation due to human errors. The Boolean expressions are assumed to be in disjunctive normal form. This paper considers only single operand faults, which may be further classified into the following seven types of faults. 1. Expression Negation Fault (ENF):- The Boolean expression is implemented as its negation. In this case, the implementation denoted by IENF will be equivalent to S. Term Negation Fault (TNF):- A term in the Boolean expression is replaced by its negation. For example, the specification abc + de is implemented as abc + - de. If the i-th term of S is negated, the implementation denoted by will be equivalent to p pi pm. 3. Literal Negation Fault (LNF):- A literal in a particular term in the Boolean expression is replaced by its negation. For example, the specification abc + de is implemented as ab?+ de. If the j-th literal of the i-th term of S is negated, the implementation denoted.by ZhF will be equivalent to pl x;...2; pm. 4. Term Omission Fault (T0F):- A particular term in the Boolean expression is omitted. For example, the specification ab + cd + ef is implemented as ab + cd. If the term pi of S is omitted, the implementation denoted by will be equivalent to p Pi-1 +pi+l+.+pm. 5. Literal Omission Fault (L0F):- A literal in a particular term of the Boolean expression is omitted. For example, the specification ubcd + def is implemented as abc + def. If the literal xf in pi of S is omitted, the implementation denoted by Z& will be equivalent to p $-l.$+l ---xki +...+p me 6. Literal Insertion Fault &IF):- A literal is inserted into a term of the Boolean expression. For example, the specification &I?+ de is implemented as abed + de or dd+ de. For LE, we do not consider the insertion of the following literals in the expression: (a) a literal that is already present in the term, because the resulting implementation is equivalent to the original expression; and (b) the negation of a literal that is already present in the term, because the result is effectively the same as that due to TOF with that particular term omitted. 22

3 For example, inserting E, 6, or c in the first term of abf+ de will produce an expression equivalent to de, which is the same as that due to TOF with the first term omitted. If a literal x is inserted in pj of S in which neither x nor its negation occur in pi, the implementation denoted by will be equivalent to pl pi. x pm. It should be noted that inserting a literal in an expression does not necessarily cause an error. For example, when the literal b is inserted in the first term of a + b, the resulting implementation ab + 6 is equivalent to the original specification. 7. Literal Reference Fault (LRF):- A literal in a particular term of the Boolean expression is replaced by another literal. For example, the specification abc+ de is implemented as abd + de (in which C has been replaced by d) or ubd+ de (in which E has been replaced by 2). For LRF, we do not consider the replacement of a literal by (a) its negation, because the result is effectively the same as that due to LNF, (b) another literal that is already present in the original expression, because the result is effectively the same as that due to LOF; or (c) the negation of another literal that is already present in that particular term, because the result is effectively the same as that due to TOE If the literal xf of pl in S is replaced by the literal x such that neither x nor X is present in pi, the 24 implementation denoted by will be equivalent to p x;-l i. k, ex Pm. It should be noted that the replacement of a literal by another literal does not necessarily cause an error. For example, when the literal b in the first term of bc+eb+ &+Ed is replaced by the literal a, the resulting implementation ac + r ~b + ab + ~d is equivalent to the original specification. 3 Related Work In the broader sense, testing of Boolean expressions in software bears some similarity to testing of digital circuits in computer hardware. However, as explained in Kuhn [7] and Weyuker et al. [12], there are some fundamental differences between the two. For example, from the circuit testing point of view, various hardware manufacturing flaws have been identified and hypothesized, and test cases are then generated to detect these flaws. On the other hand, from the software point of view, the number of possible software faults due to human errors during the development of the software is larger and the faults are more varied. 3.1 Test Case Generation from Boolean Specifications Tai and Su [lo] propose two test case generation algorithms which guarantee the detection of operator errors. However, their work is restricted to the testing of singular Boolean expressions in which each operand is a simple Boolean variable that cannot occur more than once in the Boolean expression. Tai [S, 91 extends this work to include the detection of Boolean operator faults, relational operator faults and a type of faults involving arithmetic expression. The Boolean operator faults considered by Tai involve the incorrect use of an AND operator, the incorrect use of an OR operator, the omission of a NOT operator, and the use of an extra NOT operator. Empirical studies reported in [S, 101 show that at least 98.6% and 96.8% of the mutants can be detected for singular Boolean expressions with 3 and 4 variables, respectively. By making simple syntactic changes to a Boolean expression, a mutant is created. For example, ae+ bc and a + bc are two possible mutants of the Boolean expression a + bc. However, Chilenski and Miller [3] have pointed out that the Boolean functions that can be expressed in singular Boolean expressions form only a small subset of all possible Boolean functions. They further investigate the applicability of the modified conditioddecision coverage criterion as an enhancement of the conditioddecision coverage criterion for testing of singular Boolean expressions. Weyuker et al. [12] propose and analyse a family of strategies for automatically generating test cases from a Boolean specification. Their MAX-B meaningful impact strategy, or simply the MAX-B strategy, is the most powerful since it subsumes all other strategies they propose. Their empirical results show that the basic meaningful impact strategy and the MAX-B strategy can distinguish at least 94.9% and 95.4% of the mutants, respectively. Their study has considered expression negation fault, variable negation fault, operator reference fault, variable reference fault, and associative shift fault. Chen and Lau [2] propose three other strategies for 23

4 generating test cases from Boolean specifications, namely, MUTP, CUTPNF" and MNFP. Test cases are derived to detect various types of faults that are related to operand faults as well as operator faults. The faults considered in [2] are expression negation fault, literal negation fault, literal omission fault, term omission fault, operator reference fault, literal insertion fault, and literal reference fault. Their results show that if the Boolean expression S is in irredundant disjunctive normal form, test cases that satisfy the three strategies can distinguish S from incorrect implementations that result from any one of the above faults. Empirical results [l] show that the number of test cases selected by the MUMCUT strategy, which is a combination of the MUTF', CUTPNFP, and MNFP strategies, ranges from 1.5% to 41.7% of the exhaustive test set, with 15.3% being the average. Moreover, this number ranges from 3.6% to 91.3% of all test cases selected by the MAX-B strategy, with an average of 45.1%. In other words, on average the three strategies saves 54.9% of the test cases generated by the MAX-B Strategy. Our approach in this paper is Werent from previous work in two aspects. First, all these studies basically attempt to propose methodologies for generating test cases from a Boolean expression, whereas this paper analyses the relationships among different types of faults. Second, most of the previous work perform empirical studies to analyse the effectiveness of their test case generation methodologies, whereas this paper performs a theoretical analysis of the characteristics of the faults. 3.2 Fault Class Analysis Kuhn [7] studies several classes of faults that may be made during the implementation of a Boolean expression S in disjunctive normal form. The faults he has analysed are expression negation fault, variable negation fault, and variable reference fault. Although Kuhn uses the same terms as those appearing in Weyuker et al. [12], he actually uses them with different meanings. For example, for variable negation fault, Kuhn refers to the fault which negates every occurrence of a variable in the Boolean expression, whereas Weyuker et al. refer to the fault which negates only one occurrence of a particular variable. Similarly, for variable reference fault, Kuhn refers to the fault which replaces every occurrence of a variable by another variable, whereas Weyuker et al. refer to the fault which replaces only one occurrence of the variable by another variable or a constant. Figure 1: Relationships among fault classes by Kuhn Kuhn denotes by SENF, SVNF and SVRF the expressions that are incorrectly implemented due to an expression negation fault, a variable negation fault and a variable reference fault, respectively. Two main results are proved in [7]. 1. If the variable being replaced in SVRF is the same variable as the one being negated in SVNF, any test that detects VRF can also detect VNF. If the terms containing the variable being negated in SVNF are negated in SENF, any test that detects VNF can also detect ENF. In other words, if there is a test case that detects VRF, the same test case can also detect VNF, provided that the variable being replaced in SVRF is the same as the variable negated in SVNF. Moreover, the same test case can also detect ENF, provided that all the terms containing the same variable are negated in SENF. Figure 1 shows the relationship among SVRF, SVNF and SENF [7l. Kuhn demonstrates how these results can be applied to generate a modified conditioddecision coverage test set for testing a Boolean expression. Moreover, he illustrates how to compare the effectiveness of different test strategies by using the theoretical results proved in 171. Our work in this paper complements the work of Kuhn [7] in two aspects. First, we analyse more types of faults than those in Kuhn's paper. We analyse faults due to negation, insertion, omission or replacement, whereas Kuhn has analysed only faults due to negation or replacement. Second, while Kuhn has analysed only faults that affect every occurrence of a variable, the faults we consider change only one occurrence of a literal. The former faults have much greater impact than the latter, while the latter faults model human errors that are of even 24

5 ENF t TNF LNF TOF LRF LOF LIF Figure 2: The Hierarchical Relationship more minimal in nature. As such, the faults considered here are more interesting, since they are in general easier to escape the programmer s attention, and harder to detect than their counterparts in [7]. 4 A Hierarchy of Relationships of Faults In this section, we discuss the theoretical foundation of the relationships among various types of faults given in Section The results of this section can be neatly summarized as a hierarchy shown in Figure The hierarchical relationships are interpreted in the following way. Let us consider the relationship between literal insertion fault (LIF) and term omission fault (TOF). Recall that a literal insertion fault is made if a literal is inserted into a term in a Boolean expression whereas a term omission fault is made if a term is omitted in the Boolean expression. An arrow from LIF to TOF means that if the term to be omitted due to TOF is the same term in which a literal is inserted due to LIF, any test case that detects the LIF will be able to detect the TOF as well. Other arrows in the figure are to be interpreted in a similar manner. In the rest of this section, we shall formally state and prove the relations between pairs of faults in the hierarchy one by one. Theorem 4.1 Let S be a Boolean expression in disjunctive.. can distinguish S and I,!$ for some i (15 i 5 m), the test case can also distinguish S and I&. Proof : Let 2 be a test case that can distinguish S and Ifig for some i where 15 i 5 m. In other words, S and IE; evaluate to different truth values. Recall that, S = p pm andi,!$=pl +...+pi- X+...+pm. This implies that 2 must satisfy of the following conditions: 1. every term other than pi evaluates to o (otherwise, so =If$(?) = l), the term pi evaluates to 1 (otherwise, S(?) = Zfigo = O), ancl 3. the literal x evaluates to o (otherwise, I,$:@ = so = 1). Under such circumstances, S(z) = 1 and I&.@ = 0. Hence, S(t) and 0 are different. This completes the proof. 0 Theorem 4.2 Let S be a Boolean expression in disjunctive nom1 form with m terms. If there is a test case that can distinguish S and I:; for some i (15 i 5 m), the test $ case can also distinguish S and I& for every j = I,..., ki, where ki is the number of literals in pi. Proof : Let 2 be a test case that can distinguish S and IEF for some i where 1 5 i 5 m. In other words, S and If$ evaluate to different truth values. Recall that, S = p pm andi,!$ +...+pi X+...+pm. This implies that 2 must satisfy the following conditions: 1. every term other than pi evaluates to 0, the term pi evaluates to 1, and 3. the literal x evaluates to 0. jr Under such circumstances, S(i) = 1 and = PI &.. a xi-,.~+,...x~i.x+...+pm = o for every j J = 1,..., ki. Hence, S(?) and I&@ are different. This completes the proof. 0 In other words, if the literal x being inserted into a term pi due to LIF is the same as the literal replacing any literal $., 1 5 j 5 ki, in the same term pi due to LW, any test case that detects the LIF can also detect the LRF. Theorem 4.3 Let S be a Boolean expression in disjunctive can distinguish S and I& for some i (1 5 i 5 m), the same test case can also distinguish S and I&F for every j = I,..., ki, where ki is the number of literals in pi. 25

6 Proof : Let? be a test case that can distinguish S and ZFiF for some i where 15 i 5 m. In other words, S and ZFiF evaluate to different truth values. Recall that, S = PI pm andz&f 'p pi-1 +pi+l.. +Pm. This implies that i must satisfy the following conditions: 1. every term other than pi evaluates to 0, and the term pi evaluates to 1 (that is, every literal 4,...,xij evaluates to 1). Under such circumstances, S(7) = $ are different. This completes the proof. 0 In other words, if a term pi being omitted due to TOF is the same term that contains the literal xi being negated due to LNF, any test case that detects the TOF can also detect the LNF. (7) = p pm = 0. Hence, S(7) and&f(?') Theorem 4.4 Let S be a Boolean expression in disjunctive q can distinguish S and I& for some i (15 i 5 m) and some j (15 j 5 ki), where ki is the number of literals in pi, the same test case can also distinguish S and ZAF. Proof : Let i be a test case that can distinguish S and I$ for some i and some j where 1 5 i 5 m and 1 5 q j 5 ki. In other words, S and Z& evaluate to different q truth values. Recall that, S = p pm and I& = pi xil -...x$-~-~+~..-yii.x+... +pm. his implies that?must satisfy the following conditions: 1. every term other than pi evaluates to 0, every literal in pi other thanx) evaluates to 1, and 3. the literals x) and x evaluate to different truth values. Under such circumstances, S(7) = $@ and ZAF@ = $@). Hence, SO and I&F@) are different. This completes the proof. 0 In other words, if the literal x) in a term pi being replaced by another literal due to LRF is the same literal being negated in the same term due to LNF, any test case that detects the LRF can also detect the LNF. Theorem 4.5 Let S be a Boolean expression in disjunctive can distinguish S and ZAF for some i (1 5 i 5 m) and for some j (1 5 j 5 ki), where ki is the number of literals in pi, the same test case can also distinguish S and IAF. Proof : Let i be a test case that can distinguish S and kf for some i and some j where 15 i 5 m and 1 5 j 5 ki. In other words, S and ZLoF evaluate to different truth values. Recall that, S = pi Pm and I& = p l...xji, pm. his implies that?must satisfy the following conditions: 1. every term other than pi evaluates to 0, every literal in pi other than xi evaluates to 1, and 3. the literal x) evaluates to 0. Under such circumstances, S(7) = 0 and = 1. Hence, S(T) are different. This completes the proof. 0 In other words, if the literal x$ in a term pi being omitted due to LOF is the same literal being negated in the same term due to LNF, any test case that detects the LOF can also detect the LNF. Theorem 4.6 Let S be a Boolean expression in disjunctive can distinguish S and for some i (15 i 5 m) and for some j (15 j 5 ki), where ki is the number of literals in pi, the same test case can also distinguish S and Proof : Let? be a test case that can distinguish S and I&F for some i and some j where 1 5 i 5 m and 1 5 j 5 ki. In other words, S and ZAF evaluate to different truth values. Recall that, S = PI pm and I,& = p1+...+xi,...$... Xi,+...+p,. hi^ implies that Imust satisfy the following conditions: 1. every term other than pi evaluates to 0, and every literal other than x) evaluates to 1. Under such circumstances, = pi(?'), which is different from S(T) = pi(?'). This completes the proof. 0 In other words, if the term pi that contains a literal x) being negated due to LNF is the same term being negated due to TNF, any test case that detects the LNF can also detect the TNF. Finally, the implementation ZENF is equivalent to s, which always evaluates differently from S. Hence, Theorem 4.7 should be obvious. Theorem 4.7 Let S be a Boolean expression in disjunctive normal form with m terms. I f there is a test case that can distinguish S and ZPiF for some i (1 5 i 5 m), the same test case can also distinguish S and ZENF. 26

7 5 An Application of the Fault Hierarchy As explained in [7], the theoretical results on the hierarchy of the types of faults may help to explain the experimental results in fault-based testing. We now show how the theoretical results in Section 4 can be applied to mutation testing, and discuss the related issues. The idea of mutation testing is to select test cases that can distinguish the program from its mutants [4,5]. A mutant of a program is created by making simple syntactical changes to the original program. The process of making the changes to a program is considered as a mutation operation. A test case is said to kill a mutant if, when executing with the test case as input, the results of the program and the mutant are different. A set of test cases is said to be mutation-adequate if the test cases can collectively kill all those non-equivalent mutants under consideration. Below are two examples showing how the results can be applied. The first example is a direct application of the results. The second example discusses some issues related to the application of the results. Example 1. Let us consider the Boolean expression S = ab + cd. For S, there are eight non-equivalent LIF mutants, namely abc + cd, abf+ cd, abd + cd, abd+ cd, ab + acd, ab +ad, ab+ bcd, and ab+ kd. For the LIF mutant abc + cd (in which the literal c has been inserted in the first term), a test case 1101 (a = b = 1, c = 0, d = 1) is then derived to kill this mutant. According to Theorems 4.1 and 4.2, this test case also kills the TOF mutant cd (in which the first term has been omitted) and the LRF mutants cb + cd (in which the literal a has been replaced by c) and ac + cd (in which the literal b has been replaced by c). According to Theorems 4.4,4.6 and 4.7, this test case also kills the two LNF mutants Zb + cd and & + cd, the TNF mutant &+ cd and the ENF mutant ab + cd. Furthermore, this same test case (1101) also kills the LIF mutant abd+ cd. According to Theorem 4.2, it also kills the LFW mutants db + cd (in which the literal a has been replaced by d) and ad+ cd (in which the literal b has been replaced by 2). For the rest of the LIF mutants, the test cases 1110, 0111, and 1011 can collectively kill all of the remaining LIF mutants. The four test cases thus form a test set that kills all the LIF, TOF, LRF, LNF, TNF and ENF mutants. On the other hand, altogether there are 4 LOF mutants forab+cd,namely b+cd,a+cd,ab+d,andab+c. The test cases 1001 and 0110 can kill all these LOF mutants. In summary, we have derived six test cases (namely, 1101,1110, 0111,1011,1001 and 0110) which together kill all the LIF, TOF, LRF, TOF, LNF, TNF, and ENF mutants. 0 In the previous example, test cases always exist to detect the non-equivalent mutants. However, for some Boolean expressions, this may not be true. When no test case can kill a particular mutant, the mutant is equivalent to the original Boolean expression. In such a case, the corresponding mutants at the next level up in the hierarchy will have to be considered. This is illustrated in the following example. Example Let us consider the Boolean expression S = abc + abd + 6d+ E. In this example, we shall consider only the first term of S. For other terms of S, the analysis is similar. For the first term, there are 4 LIF mutants, namely, abcd + abd + 6d+ e, abcd+ abd + bd+ E, abce + abd + id+ E, and abce+ abd + bz+ E. A test case can be used to kill the LIF mutants abcd + abd + bd+ E and abce + abd + &d+ E. According to Theorems 4.1 and 4.2, the same test case kills the TOF mutant in which the first term is omitted, as well as the LRF mutants in which any literal in the first term is replaced by either d or Since S is equivalent to the remaining two LIF mutants of the first term (in which the literal d or e has been inserted in the first term), no test cases can be selected to kill them. The fault type immediately above LIF in the hierarchy in Figure 2 is LRF. It is therefore necessary to derive test cases to kill the corresponding LRF mutants in which a literal in the first term has been replaced by d or e. Among these six LRF mutants, the test cases 01101, and can kill five of them. The remaining LRF mutant, a& + ubd + 5J+ E (in which the literal b in the first term has been replaced by z), is equivalent to S. Since test cases can be derived to kill LRF mutants in which the literal a, b or c has been replaced by one of the other literals, all LNF mutants related to the first term (in which any of the literal a, b or c in the first term has been negated) can be killed, according to Theorem 4.4. As a result, the corresponding TNF mutant on the first term and the ENF mutant can be killed. In addition, it happens that the test cases 01101,10111 and can kill all the three LOF mutants related to the first term. In summary, the four test cases , 01101, and can collectively kill all the non-equivalent LIF, TOF, LRF, LOF, LNF, TNF, and ENF mutants related to the first term of S. 0 27

8 6 Summary and Conclusion In this paper, a hierarchy summarizing the relationships among different types of faults for Boolean expressions is established. A test case that detects literal insertion fault can also detect literal reference fault and term omission fault. A test case that detects literal reference fault can also detect literal negation fault. A test case that detects term omission fault can also detect literal negation fault. A test case that detects literal omission fault can also detect literal negation fault. Finally, a test case that detects literal negation fault can also detect term negation fault and expression negation fault. These theoretical results suggest that test cases detecting literal insertion fault can also detect other types of faults, and should therefore be generated as early as possible. Moreover, we have demonstrated how these theoretical results can be applied to generate test cases to kill all related mutants. Our results complement those of Kuhn because Kuhn considers faults that affect (by negating or replacing) all occurrences of a variable, whereas we only consider faults that wrongly implement one occurrence of a literal. Moreover, we study a larger set of possible faults than those of Kuhn. With the identification of more types of faults and the study of how these types of faults are related in the form of a hierarchy, more effective test sets, in the sense of detecting more types of faults, can be derived. Moreover, the hierarchy provides guidelines to the generation of test cases. Specifically, test cases should be generated to detect the types of faults at the lower level of the hierarchy first. In general, this helps to detect all such faults earlier than before. A mutation system is currently being developed to support experimental investigations of the effectiveness and efficiency of the guidelines proposed in this paper. Acknowledgement The work described in this paper is partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (RGC ref. no. CityU 1083/00E). References T. Y. Chen, M. F. Lau, and Y. T. Yu. An empirical evaluation of the MUMCUT strategy for testing Boolean specifications. In Proceedings of the 3rd Annual IASTED International Conference on Software Engineering and Applications (SEA 99), pp , October T. Y. Chen and M. F. Lau. Test case selection strategies based on boolean specifications. To appear in Software Testing, Verijcation and Reliability, J. J. Chilenski and S. P. Miller. Applicability of modified condition/decision coverage to software testing. Software Engineering Jouml, 9(5): , September R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 11(4):34-41, April R. A. DeMillo and A. J. Offutt. Constraint-based automatic test data generation. IEEE Transactions on Software Engineering, 17(9):9W910, September K. A. Foster. Sensitive test data for logic expressions. ACM SIGSOFT Software Engineering Notes, 9(2): , April D. R. Kuhn. Fault classes and emr detection capability of specification-based testing. ACM Transactions on Software Engineering and Methodology, 8(4):411424, K. C. Tai. Condition-based software testing strategies. In Proceedings of COMPSAC 90: the 14-th Annual Intemtional Computer Software and Application Conference, pp , October K. C. Tai. Theory of fault-based predicate testing for computer programs. IEEE Transactions on Software Engineering, 22(8): , August K. C. Tai and H. K. Su. Test generation for boolean expressions. In Proceedings of COMPSAC 87: the 11-th Annual Intemational Computer Software and Application Conference, pp , M. A. Vouk, K. C. Tai, and A. Paradkar. Empirical studies of predicate-based software testing. In Proceedings of IEEE Symposium on Sofrware Reliability Engineering, pp , November E. J. Weyuker, T. Goradia, and A. Singh. Automatically generating test data from a boolean specification. IEEE Transactions on Software Engineering, 20(5): , May