[22] Glenford J. Myers, \The art of Software Testing," John Wiley & Sons, Inc., New York,

Transcription

1 [22] Glenford J. Myers, \The art of Software Testing," John Wiley & Sons, Inc., New York, [23] A. J. Outt, \Investigations of the software testing coupling eect," ACM Trans. on Software Engineering Methodology, 1(1):3{18, January [24] A. J. Outt, G. Rothermel, and C. Zapf, \An experimental evaluation of selective mutation," in Proceedings of the 15th International Conference on Software Engineering, pages 100{107, Baltimore, MD, May [25] E. J. Weyuker, S. N. Weiss, and R. G. Hamlet, \Comparison of program testing strategies," in Proceedings of the Fourth Symposium on Software Testing, Analysis and Verication, pages 154{164, Victoria, British Columbia, Canada, October [26] W. E. Wong, \On Mutation and Data Flow," PhD thesis, Department of Computer Science, Purdue University, West Lafayette, IN, December (available as SERC-TR- 149-P) 11

2 [11] P. G. Frankl and S. N. Weiss, \An experimental comparison of the eectiveness of branch testing and data ow testing," IEEE Trans. on Software Engineering, 19(8):774{787, October [12] P. G. Frankl and E. J. Weyuker, \A formal analysis of the fault-detecting ability of testing methods," IEEE Trans. on Software Engineering, 19(3):202{213, March [13] P. G. Frankl and E. J. Weyuker, \Provable improvements on branch testing," IEEE Trans. on Software Engineering, 19(10):962{975, October [14] J. R. Horgan and S. A. London, \Data ow coverage and the C language," in Proceedings of the Fourth Symposium on Software Testing, Analysis, and Verication, pages 87{97, Victoria, British Columbia, Canada, October [15] J. C. Maldonado, M. E. Delamaro, M. Jino, and M. Chaim, \PROTEUM: A testing tool based on mutant analysis". in Proceedings of the Seventh Brazilian Symposium on Software Engineering, Rio de Janiero, Brazil, October [16] A. P. Mathur, \Performance, eectiveness, and reliability issues in software testing," in Proceedings of the Fifteenth Annual International Computer Software and Applications Conference, pages 604{605, Tokyo, Japan, September [17] A. P. Mathur, \Mutation testing," Encyclopedia of Software Engineering, pages 707{713, [18] A. P. Mathur and W. E. Wong, \Comparing the fault detection eectiveness of mutation and data ow testing: An empirical study," Technical Report SERC-TR-146-P, Software Engineering Research Center, Purdue University, September [19] A. P. Mathur and W. E. Wong, \Evaluation of the cost of alternate mutation testing strategies," in Proceedings of the Seventh Brazilian Symposium on Software Engineering, Rio de Janiero, Brazil, October [20] A. P. Mathur and W. E. Wong, \A theoretical comparison between mutation and data ow based test adequacy criteria," in Proceedings of 1994 ACM Computer Science Conference, Phoenix, Arizona, March [21] A. P. Mathur and W. E. Wong, \An empirical comparison of data ow and mutation-based test adequacy criteria," The Journal of Software Testing, Verication, and Reliability, 4(1), June (To appear) 10

3 References [1] A. T. Acree, \On mutation," PhD thesis, School of Information and Computer Science, Georgia Institute of Technology, Atlanta, GA, (available as GIT-ICS-80/12) [2] H. Agrawal, R. A. DeMillo, R. Hathaway, Wm. Hsu, Wynne Hsu, E. W. Krauser, R. J. Martin, A. P. Mathur, and E. H. Spaord, \Design of mutant operators for the C programming language," Technical Report SERC-TR-41-P, Software Engineering Research Center, Purdue University, March [3] T. A. Budd, \Mutation Analysis of Program Test Data," PhD thesis, Yale University, New Haven, CT, [4] T. A. Budd, \Mutation analysis: Ideas, examples, problems and prospect," in Computer Program Testing, B. Chandrasekaran and S. Radicchi, Eds. Amsterdam, North Holland, July [5] T. A Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward, \The design of a prototype mutation system for program testing," in Proceedings 1978 National Computer Conference, Alexandria, VA, [6] T. A Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward, \Theoretical and empirical studies on using program mutation to test the functional correctness of programs," in Proceedings of the Seventh ACM Symposium on Principles of Programming Languages, Las Vegas, [7] B. J. Choi, R. A. DeMillo, E. W. Krauser, A. P. Mathur, R. J. Martin, A. J. Outt, H. Pan, and E. H. Spaord, \The Mothra toolset," in Proceedings of the Twenty-Second Annual Hawaii International Conference on System Sciences, HI, January [8] R. A. DeMillo, D. S. Guindi, K. N. King, W. M. McCracken, and A. J. Outt, \An extended overview of the Mothra software testing environment," in Proceedings of the Second Workshop on Software Testing, Verication and Analysis, pages 142{151, Ban, Alberta, Canada, July IEEE Computer Society Press. [9] R. A. DeMillo, R. J. Lipton, and F. G. Sayward, \Hints on test data selection: Help for the practicing programmer," IEEE Computer, 11(4):34{41, April [10] R. A. DeMillo and A. J. Outt, \Constraint based automatic test data generation," IEEE Trans. on Software Engineering, 17(9):900{910, September

4 When mutation operators are excluded from MO such that the resulting mutation covers some criterion R, then there exists some program-specication pair P and S such that M(mutation, P, S) < M(R, P, S) and E(mutation, P, S) < E(R, P, S) However, in attempting to nd a more positive interpretation of Frankl-Weyuker's result, we turned to the description of the test case selection model dened by the authors at the beginning of their paper. In this regard Frankl-Weyuker are correct only in the following combinatorial and thus not very interesting sense. Mutation analysis only requires polynomially many test cases as a function of program size. Therefore, any test criterion that can in the worst case require exponentially many subdomains may also include subdomains that cannot be covered by polynomially bounded methods. This in itself does not automatically imply the dominance of such methods in either the M or E hierarchies since there does not appear to be any way to guarantee a priori that exponentially many products of relative frequencies has a rate of convergence that is more favorable than M(mutation; P; S). Even if such a dominance were to be established mathematically, the intractability of criteria that require exponentially many test cases should be obvious. This observation works in both directions, however. It seems rather farfetched for Frankl- Weyuker to claim their test selection model is not biased in favor of some criterion because any criterion that is insensitive to test case redundancy can gain unexpected fault detection ability when compared to a method like mutation. Redundant test cases will by sheer chance eventually detect some faults. In this respect, weaker adequacy criteria can enjoy a larger marginal benet than a stronger one. Without relating the eectiveness of a criterion to a measurement of test eort, all such theoretical studies are bound to be awed. What conclusions can be drawn from this counterpoint to the analysis in [?] beyond the obvious conclusion that there is little to be gained in setting up straw-man criteria to bolster a preconceived notion of relative fault detecting ability? We suspect that the Frankl-Weyuker hierarchy is of no practical signicance at least insofar as it attempts to nd analytical comparisons between such widely disparate methods as mutation-based and structurally-based test criteria. It is our belief that simple and elegant hierarchies of test criteria are interesting academic artifacts that ultimately have little to do with what engineers do in practice. In practical situations, the important factors have less to do with orderings over grossly simplied models of reality and more to to with the immeasurably harder tasks of analyzing the cost and eectiveness of specic test methods for specic environments, tasks and risk factors. In such a world it is unlikely that Figure 4 of the [?] even if it were correct 00 would lead to increased or better usage of existing test criteria. 8

5 of mutation systems have processed increasingly structured programs, the need for (3') has decreased. In the Mothra system for example which processes Fortran 77, (3') is represented only by an operator called ONETRIP that ensures loop traversal. The application of mutation to the programming language C increases the possibility of unstructured branches into conditionals, but the mutation operators set for C does not include an operator (3') [?,?]. This is because in C every boolean expression is also an arithmetic expression. There is already an operator ZPUSH (NZPUSH) that when applied to an expression returns TRAP only if the expression evaluates to zero (nonzero). This operator applied to expressions controlling loops and if statements is obviously sucient to guarantee branch coverage. Based on suggestions made by Budd and Acree [?,?]; Mathur, Wong [?], as well as Outt and his co-workers [?] have discovered some complex relationships between selected subsets of standard MO's through experimentation. This short historical sketch serves to remind us of the danger in the Frankl-Weyuker assertion that one can by inspection determine \the only mutation operators that are relevant..." for some comparison. 3 Mutation and DataFlow There is a growing body of evidence that the fault detecting ability of mutation testing methods are at least as strong as any of the other methods mentioned in [?] provided the analysis exploits one of the crucial features of mutation testing: the inclusion of robust sets of mutation operators accounts for the ability of mutation analysis to detect errors through operator, domain and operand mutations that are not related in an obvious way to structurally based test case selection criteria [?,?]. For example, one of us [?] has shown that for the most general class of programs with at least one variable of multiple denitions and multiple uses, mutation is incomparable with c-use, p-use and all-uses criteria with respect to the weaker subsumption relation [?]. This is true in particular for the class of mutation operators dened in the Mothra environment. On the other hand, in the same work, Wong has also demonstrated experimentally that for an interesting range of seeded faults, mutation dominates the fault detecting ability of dataow testing criteria [?], a result that appears to be dicult to reconcile with Figure 4 of [?] 4 Concluding Remarks It is not a far stretch to paraphrase Theorem 4 from [?] as follows: 7

6 Now let P 000 be the mutant of P in which y := x div 2 is replaced by TRAP and let P 0000 be the mutant in which y := x/2 is replaced by TRAP. Then D C=T = fx j C(x)g = D(P 000 ) and D C=F = fx jnot C(x)g = D(P 0000 ). In other words any set of limited mutation operators that includes (3) properly covers branch testing for this program. In particular, if S is any specication for P then M(limited mutation; P; S) M(decision; P; S) and E(limited mutation; P; S) E(decision; P; S). Therefore, Frankl-Weyuker's proof is incorrect, since there are sets of mutation operators that obviously universally cover branch testing. As Frankl-Weyuker hint, mutation testing limited to operators (1)-(3) is sucient to properly cover branch testing for any structured program P. This, of course, does not help if the program contains control transfers like BREAK or GOTO that allow conditional backward control transfers into THEN or ELSE branches of decisions or into the body of a loop. Historically, this problem was especially severe in early mutation testing tools [?,?,?] which processed Fortran II, Fortran IV and Cobol. It was solved by adding a variant of (3) to the list of mutation operators: (3') replace each conditional control transfer point by TRAP. When (3') is applied in Fortran DO loops, for example, it insures that each loop is traversed exactly once. When applied to the THEN clause of an IF statement it produces a mutant containing THEN TRAP which can be distinguished only if the controlling boolean expression evaluates to true. By the same token when applied to the ELSE clause of an IF statement, it produces a mutant is distinguished only when the controlling expression evaluates to false. Thus, for any pair of subdomains D C=T and D C=F where C controls a conditional statement of the form if C then... else... there are mutants resulting from the application of (3') which have D C=T and D C=F as subdomains. We therefore have the following theorem: Theorem 1 Let MO be any set of mutation operators containing (3') and let mutation represent the mutation-adequacy criterion relative to MO, then (i) mutation universally covers branch testing, and (ii) for any program-specication pair (P, S), we have M(mutation,P,S) M(decision,P,S), and E(mutation,P,S) E(decision,P,S). The somewhat inelegant formulation of (3') given above was rened in the EXPER system [?] as a pair of operators that take a boolean expression as an argument and are distinguished exactly when the expression evaluates to true and false, respectively. As successive generations 6

7 refer to mutation testing with only these two operators." As we see below this assumption is incorrect. When coupled with a corresponding misunderstanding of mutation testing itself, it leads the authors to both the following claim (Theorem 4 in [?]): \Limited mutation testing does not universally cover decision-coverage." and its Corollary (Example 3 in [?]): \There is a program-specication pair P and S such that 0 = M(mutation; P; S) < M (decision; P; S) and 0 = E(mutation; P; S) < E (decision; P; S)." Let us rst describe the specic error that invalidates the proofs of these results: it is incorrect that only the operators (1) and (2) above are directly related to branch testing. As Budd explicitly points out in [?] and has been armed in several studies since then [?,?,?,?,?] it is the combination of operators that is important. In [?] for example, Budd's intent was to describe a mutation analysis that included (1) and (2), not one that was limited to just these two. To see why this is important, consider the following mutation operator that has been a member of every MO of which we are aware: (3) replace a statement by TRAP where TRAP is a special statement that upon execution causes the mutant to be distinguished immediately. As described in [?], and several other places, the TRAP mutations are used to insure statement coverage. Since it is not possible to insure branch coverage and exclude statement coverage, it seems to us that (3) is directly related to branch coverage. So let us assume that limited mutation includes (3) (at the very least including (3) seems no more arbitrary than excluding it). The proof of Frankl-Weyuker's claim hinges on constructing the following program P containing a decision C such that the even elements of D C=T and D C=F are excluded from the mutation subdomains: var x: integer; y: real; begin read(x); if C(x) then y := x div 2 else y := x/2; write(y); end; 5

8 Theorems (1) and (2) in [?] state that if C1 properly covers C2 with respect to program P and specication S then M(C1; P; S) M(C2; P; S) and E(C1; P; S) E(C2; P; S). This notion is contrasted with the somewhat weaker concept of subsumption of C2 by C1 which occurs when satisfaction of C2 implies satisfaction of C1. As Frankl-Weyuker point out, the fact that a criterion C1 subsumes C2 does not necessarily imply that C1 has greater fault detecting ability. 2 Mutation Testing and Decision Coverage Frankl-Weyuker use more or less standard denitions of mutation adequate test coverage and decision coverage. A mutant of a program P is a variant of P dened by applying exactly one mutation operator to P [?] at exactly one point. Each mutation operator is a rule for making a simple syntactic transformation of P. A test set satises a mutation criterion relative to a set of mutation operators MO if it distinguishes P from all non-equivalent mutants of P that can be formed using MO. In practice, a \standard" (but language dependent) set of mutation operators is supplied by a mutation testing tool [?,?,?,?]. In these cases explicit reference to MO is omitted. Mathur, Wong [?], as well as Outt and his colleagues [?] have recently begun to explore the eects of various schemes for selecting subsets of MO. Mutation testing can be easily seen to be a subdomain-based criterion by dening for each mutant program m the subdomain fx j P (x) 6= m(x)g. Decision coverage corresponds to simple decision-to-decision branch coverage, a common structural test coverage criterion [?,?]. The subdomains for decision coverage consist for each program decision controlled by a predicate p(x), the set of all input values t that cause p(x) to evaluate to true at some point during program execution (denoted D p=t ) and the set of all input values t that cause p(x) to evaluate to false at some point during program execution (denoted D p=f ). As a single input t can cause p to evaluate to true and at some later point during execution to evaluate to false, D p=t and D p=f are not necessarily disjoint. We follow [?] and use the terms decision coverage and branch testing interchangeably. Frankl-Weyuker in [?] recall an observation due to Budd [?], if MO includes: (1) replace a decision by true (2) replace a decision by false then mutation testing subsumes decision coverage. From this, Frankl-Weyuker conclude that limiting mutation to just these operators would result in an interesting comparison under M and E and justify the denition of limited mutation as follows: \Since these are the only mutation operators that are directly relevant to branch testing, we will use the term limited mutation to 4

9 derstanding or other inadvertent misuse of one of the mutation operators, the eect is clear: Frankl-Weyuker have set up a straw-man testing criterion specically to support a desired result. The remainder of this note will be devoted to establishing our claims. We will conclude with some observations on the feasibility of the Frankl-Weyuker hierarchy. 1 The Frankl-Weyuker Model We assume as in [?] a suitable denition of test criteria C, programs P and specications S. In particular, we assume standard notions for determining whether or not P is correct with respect to S for a given input called a test case, whether or not a set of test cases satisfy the criterion C and whether or not a fault that for a given input causes P to be incorrect with respect to S. Inputs that satisfy this latter condition are said to be failure-causing. Finally, we assume as in [?] that each candidate criterion C denes a multiset, say SD C (P,S ), of not necessarily disjoint subsets of P 's domain called subdomains. Subdomains have the property that C can only be satised by test sets whose intersection with each of the subdomains is nonempty. Now, let C, P, and S be xed and for each subdomain D i induced on the domain of P, d i =j D i j and m i =j fx j x in D i and x causes a failure in P with respect to Sg j, then dene: and ny M(C; P; S) = 1 0 (1 m i 0 ) (1) i=1 d i nx E(C; P; S) = i=1 m i d i (2) M(C,P,S ) is the probability that a test set satisfying C induces at least one failure when test cases are chosen randomly from each subdomain induced by C. Frankl-Weyuker call E(C,P,S ) the expected number of failures detected by C with respect to P and S. 2 The authors use the notion of proper coverage of criterion C2 by criterion C1 as the basis for their hierarchy cased in M and E. Roughly speaking C1 properly covers C2 with respect to P and S just in case each subdomain of SD C2 (P; S ) can be written as a union in SD C1(P; S) so that the multiplicity of no subdomain of C1 is increased. If C1 properly covers C2 for every program-specication pair (P; S) then C1 is said to universally properly cover C2. 2 It is not entirely clear why the authors view Equation (??), which is clearly a sum of ratios, as the expected number of failures detected. This confusion with terminology, however, does not aect their results. 3

10 to establish that mutation testing is less eective than branch testing in terms of its ability to detect faults. Using arguments presented below we claim that any reasonable interpretation of the Frankl- Weyuker's result can be invalidated by simple counter-examples. The collapse of one segment of their hierarchy does not invalidate the remainder of the results. It does, however, cast considerable doubt on its signicance. Frankl-Weyuker initially frame their comparison narrowly in terms of a mutation criterion which appears to have been invented especially for their work. They refer to this criterion as limited mutation. We are not aware of any literature on limited mutation prior to the Frankl- Weyuker's work. After establishing their comparison for limited mutation, the authors go on to conjecture that their hierarchy results obtain for all extensions of limited mutation, including those criteria studied in the past [?,?,?,?,?,?,?,?,?,?,?]. In particular, Frankl-Weyuker claim that \...most mutation operators proposed in the literature do not necessarily lead to proper coverage of branch testing." They go on to make the following conjecture: \We therefore believe that, even with the addition of other mutation operators, it will be possible to construct programs for which mutation is less likely to detect faults than branch testing...". The claim and conjecture are both false. Furthermore since they essentially identify the eectiveness of a set of mutation operators with the least powerful operator in the set, they, when taken together, indicate a misunderstanding of the way in which mutation analysis is used to test programs. At various points in their work (such as in the abstract) the appellation limited is dropped from the exposition. Casual readers are thus left with the impression that a provable hierarchy of fault detecting ability has been established in which mutation testing is less eective than branch testing. As will be shown below, this impression is based on a false assumption and leads to conclusions that are contradicted by experimental and analytical results. Not only is their specic result concerning limited mutation ill-formulated, it can be shown that no reasonable interpretation of Frankl-Weyuker's broad and completely unsupported speculation can be true since mutation testing has, since its rst introduction in 1976 [?], incorporated mutation operators that include branch coverage as a special case. Even more importantly, recent results by one of us [?,?] demonstrate: (1) a provable incomparability between mutation testing and the all-uses criterion, and (2) the superiority of mutation testing, as demonstrated experimentally, in the fault detecting arena studied by Frankl and Weyuker. While it is unclear from [?] whether or not this particular error resulted from a misun- 2

11 Some Critical Remarks on a Hierarchy of the Fault-Detecting Ability of Test Methods 3 Richard A. DeMillo, Aditya P. Mathur, and W. Eric Wong September 28, 1994 Abstract In a recent article Frankl and Weyuker report results that appear to establish a hierarchy of abilities of software testing methods to detect faults. A key result of this hierarchy is a relationship between decision coverage and mutation testing, a fault-based testing criterion, that appears to establish that mutation testing is less eective than branch testing in terms of its ability to detect faults. In this note we argue that any reasonable interpretation of this result can be invalidated by simple counter-examples. In a recent article [?], Frankl and Weyuker report results that appear to establish a hierarchy of abilities of software testing methods to detect faults. 1 The methods used by Frankl-Weyuker to obtain this hierarchy constitute a new and important addition to their arsenal of tools aimed at establishing simple, useful comparisons of test data generation methods. This is the latest step in an ambitious classication programme undertaken by the authors of [?] collaborators (see e.g., [?,?,?]). and their A signicant aspect of the Frankl-Weyuker method is the utilization of a probabilistic measure of test eectiveness intended to support a model of test selection that is \well-dened, not obviously biased in some criterion's favor and not too far from reality." [?]. Such a hierarchy, if valid, would be of great theoretical interest since it would (as the title of [?] suggests) establish a provable ordering of test criteria based on the inherent and universal properties of the criteria and not some artifact related to how the criteria might be applied in practice. A key result of Frankl-Weyuker's hierarchy is a relationship between decision coverage and mutation testing, a fault-based testing criterion, (see e.g., [?] for a recent survey) that appears 3 Richard A. DeMillo is the Director of Software Engineering Research Center, Department of Computer Sciences, Purdue University. He is aliated with the Dipartimento di Elettronica e Informatica, University of Padova, Padova, Italy on his sabbatical leave until July Aditya P. Mathur is an associate professor at the Department of Computer Sciences, Purdue University. W. Eric Wong is a member of technical sta at the Department of Software Technology, Hughes Network Systems, Germantown, MD Unless specied otherwise all our references to Frankl-Weyuker are references to their work reported in [?]. 1