Measuring the Effectiveness of Method Test Sequences Derived from Sequencing Constraints

Transcription

1 Measuring the Effectiveness of Method Test Sequences Derived from Sequencing Constraints F.J. Daniels Bell Laboratories K.C. Tai North Carolina State University Abstract Intra-class testing refers to the testing of the interaction among methods and data structures encapsulated within a single class. Our approach to intra-class testing is to execute sequences of instance methods that are derived from sequencing constraints and evaluate their results for correctness. These constraints impose restrictions on method behaviors and can be derived from a formal or informal specification of a class. We present an empirical evaluation of different method sequence generation approaches, and analyze their effectiveness in detecting software faults. In addition, we define a number of coverage criteria based on method sequencing constraints for a class and examine the differences between them. 1. Introduction Most of the work in intra-class testing has focused on specification-based testing techniques [7,10]. These techniques involve selecting sequences of methods that document the correct order in which the methods of classes can be invoked. In order to detect faults among the interacting methods, one can check whether a correct method sequence puts an object of a class into an inconsistent state. Our idea is to execute sequences of class methods that are derived from sequencing constraints, and to evaluate their results for correctness. These constraints impose restrictions on method behaviors and can be derived from a formal or informal specification of a class. This paper presents new method sequence generation approaches, and uses simulations to evaluate their effectiveness in detecting software faults. Focus is on issues involved in generating a set of method sequences for a class that are error sensitive yet practical. 2. Related work on intra-class testing Kirani and Tsai [10] introduced a technique called message sequence specification that describes the correct order in which the methods of a class can be invoked by the methods in other client classes. The method sequence specification associated with an object specifies all sequences of messages that the object can receive while still providing correct behavior. Their strategy uses regular expressions to model the causal relationships between classes, where method names are used as the alphabet of the grammar. The grammar is then used to statically verify the program s implementation for improper method sequences. A runtime verification system identifies incorrect method invocations by checking for sequence consistency with respect to the sequence specification. Doong and Frankl [7] developed a scheme for object-oriented (OO) testing using algebraic specifications. The idea is that the natural units to test are classes, and that in testing classes, one should focus on whether a sequence of messages puts an object of the class under test into the correct state. Each test case consists of a pair of sequences of messages, along with a tag indicating whether these sequences should result in objects that are in the same "abstract state". The authors developed A Set of Tools for Object-Oriented Testing (ASTOOT) that include: an

2 interactive specification-based test case generation tool and a tool that automatically generates test drivers. 3. Method sequencing constraints Current work uses method pre- and post-conditions to model a class specification [12]. The pre-condition of a method is a set of requirements that must be met before the method is used, and the post-condition specifies the expected property resulting from the method. In our approach to testing, pre- and post-conditions are used to derive method sequencing constraints. These method sequencing constraint impose restrictions on method behaviors and are then used to guide the selection of method sequences during testing. The following notation follows the CSPE (Constraints on Succeeding and Proceeding Events) [14, 15], but it now has been adapted to method sequence selection. Each method sequence constraint involves a constraint operator and two methods. These sequencing constraints allow us to derive always (a), possibly true (pt), possibly false (pf), and never (~) constraint operators. Let C represent a class and let M1 and M2 represent methods within class C. Then: - a[m1; M2] denotes that for an instance of class C, immediately after completion of method M1, invoking method M2 is always valid. Thus the post-condition of M1 implies the pre-condition of M2. - p[m1; M2] denotes that for an instance of class C, immediately after completion of method M1, invoking method M2 is possibly valid. Whether M2 can be invoked immediately after M1 depends on the input and the method sequence that precedes M1. Let K be the necessary and sufficient condition under which M1 can be immediately followed by M2. Thus, p[m1; M2]K if and only if the conjunction of K and the post-condition of M1 implies the pre-condition of M2. To simplify our discussion, constraint p[m1; M2]K is split into two constraints: possibly true constraint pt[m1 ; M2]K and possibly false constraint pf[m1; M2](not K). - ~[M1; M2] denotes that for an instance of class C, immediately after completion of method M1, invoking method M2 is never valid. Thus post-condition M1 implies the negation of the pre-condition of M Example Class Queue To illustrate the method sequencing constraint notation, we use the unbound queue abstract data type [1] to model the class Queue. The queue object can be in either of three relevant states: 1) Empty, 2) Not Empty, and 3) Exception. Items can be enqueued in either the Empty or Non Empty state. However, if the state of the queue is Empty, items cannot be dequeued, the top item of the queue cannot be returned, and an exception is raised from the Exception State. The class Queue implements five methods. Method Init creates the queue and initializes all data members. Method Empty reports the state (empty/not empty) of the queue. Method Eque adds items to the queue while method Dque removes items form the queue. Method Top returns the top item of the queue. Listed below are the class attributes, restrictions and pre and post-conditions for the queue class. Method Empty does not affect the attributes of the queue (non-attribute effecting method), and it can be invoked at any place in a method sequence once the queue has been initialized. Hence, pre- and post-conditions for method Empty are omitted. Class Queue Attributes: Restrictions: 1. Count = number of elements in Queue (q). 1. Once a queue is created it cannot be recreated in the same instance. 2. Queue is unbounded and there are no other resource limitations.

3 3. All operations must be performed on a queue that has been initialized. 4. Count cannot be negative. Pre- and Post Conditions Method Init (q:queue) 1. Queue (q) does not exist 1. Queue (q) exists 2. Queue (q) is empty Method Empty (q:queue) 1. Queue (q) exist 1. Return 1 if Queue (q) is empty (count = 0) 2. Returns 0 if Queue (q) is not empty (count > 0) Method Eque (q:queue, e:element) 1. Queue (q) exist 1. Element e added to the tail of Queue (q) 2. Queue is not empty (count = old count + 1) Method Dque (q:queue, e:element) 1. Queue (q) exists 1. Element (e) removed from the front of Queue (q) 2. Queue is not empty (count > 0) 2. Count = count 1 3. Queue (q) exists Method Top (q: Queue, e: element) 1. Queue (q) exist 1. First element (e) returned 2. Queue (q) is not empty (count > 0) 2. Queue (q) exists 3. Queue is not empty (count > 0) 3.2 Method Sequencing Constraints for Class Queue Method sequencing constraints are derived manually by evaluating the pre- and postconditions of each method. Several examples are shown below that illustrate how a few of the constraints (a, pt, and pf) are derived. The remaining constraints for this example are listed in [4]. The post-condition of method Init and the pre-condition of method Eque are as follows: Method Init (q: Queue) Method Eque (q: Queue, e: element) Pre-Condition: 1. Queue (q) exist 1. Queue (q) exist 2. Queue (q) is empty Since, the post-condition of method Init satisfies the pre-condition of method Eque, invoking method Init followed by method Eque is always valid. Thus, the always constraint a[init; eque] is derived. To illustrate derivation of possibly true and possibly false constraints, consider the postcondition of method Dque and the pre-condition of method Top. Method Dque (q: Queue, e: element) Method Top (q: Queue, e: element) 1. Element (e) removed from the front 1. Queue (q) exists of Queue (q) 2. Queue (q) is not empty (count>0) 2. count = count 1 3. Queue (q) exists The post-condition of method Dque implies the pre-condition of method Top if and only if the queue is not empty. Hence the possible true constraint: pt[dque; top] with the condition (queue (q) is not empty) and the possible false constraint: pf[dque; top] with the condition (queue (q) is empty) are derived.

4 Method sequencing constraints for several of the methods in the queue example are shown below. When illustrating the method sequences, # denotes the start of a method sequence. Refer to [4] for a complete listing of all method sequences. Method Init (q: Queue) S1. a[#; init] S4. ~[init; init] S2. a[init; eque] S5. ~[init; top] S3. ~[init; dque] A queue must be initialized first (S1). Once the queue is created it is always valid to invoke method Eque (S2). It is invalid to invoke method Dque (S3) or Top (S5) immediately after method Init (queue is Empty). It is invalid to invoke method Init once a queue has previously been initialized (S4). Method Eque (q: Queue, e: element) S6. ~[#, eque] S9. a[eque; eque] S7. a[eque; dque] S10. ~[eque; init] S8. a[eque; top] It is always valid to invoke methods Dque (S7), Top (S8) and Eque (S9) immediately after method Eque. It is invalid to invoke method Init once a queue has previously been created (S10) and it is invalid to start a method sequence with Eque (S6). Method Dque (q: Queue, e: element) S11. ~[#; dque] S15. pf[dque; dque] Queue (q) is empty S12. a[dque; eque] S16. pt[dque; top] Queue (q) is not empty S13. ~[dque; init] S17. pf[dque; top] Queue (q) is empty S14. pt[dque; dque] Queue (q) is not empty It is always valid to invoke method Eque (S12) immediately after method Dque. If the queue is not empty, it is always valid to invoke method Dque (S14) and Top (S16) immediately after invoking method Dque. If the queue is empty, invoking methods Dque (S15) and Top (S17) are invalid. It is invalid to invoke method Dque once a queue has previously been initialized (S13) and it is invalid to start a method sequence with Dque (S11). 4. Method Sequencing Test Coverage Criteria This section discusses several test coverage criteria used in generating method sequences. These test coverage criteria (a, a-s, a/pt, a/pt-s, a/pt-s/pf/~) and their definitions are shown in Table 1, and they follow the CSPE test coverage criterion defined in [14]. To demonstrate how method sequences are derived from the test criteria, we use two types of method sequences: 1) valid method sequences and 2) invalid method sequences. A valid method sequence is a sequence of method names such that every two consecutive method names satisfy the "always" or "possibly true" constraint for these two methods. Thus, a valid method sequence specifies a correct sequence of method invocations. An invalid method sequence is like a valid method sequence except that the last two method names satisfy the "never" or "possible false" constraint. Thus, an invalid method sequence specifies an incorrect sequence of method invocations such that any proper prefix of the sequence is a valid method sequence. 4.1 Valid method sequences for class queue Following we list the always and the always possible true constraints for the queue example. Always constraints: (S1, S2, S7, S8, S9, S12, S19, S20, S22) Possibly True constraints: (S14, S16)

5 Next we show two method sequences one satisfying the (a) criterion and the other satisfying the (a/pt) criterion. For each method sequence, we show the sequence of constraints covered by every two consecutive methods, except that no constraints are given for method Empty. (a) Criterion 1. #.init.empty.enque.enque.deque.enque.top.top.enque.top.deque (S1, S2, S9, S7, S12, S8, S22, S20, S8, S19) (a/pt) Criterion 1. #.init.empty.enque.enque.deque.enque.top.top.enque.top.deque.deque.top (S1, S2, S9, S7, S12, S8, S22, S20, S8, S19, S14, S16) Below we list method sequences satisfying the (a-s) and (a/pt-s) test criteria respectively. Unlike the (a) and (a/pt) criteria, it is not necessary to cover always and possibly true constraints with a minimum number of sequences. (a-s) Criterion 1. #.init.empty.enque.enque.top.deque.enque 3. #.init.empty.enque.top.enque.deque (S1, S2, S9, S8, S19, S12) (S1, S2, S8, S20, S7) 2. #.init.empty.enque.deque.enque.enque.top.top.deque (S1, S2, S7, S12, S9, S8, S22, S19) (a/pt-s) Criterion 1. #.init.enque.deque.empty.enque.top 4. #.init.empty.enque.enque.top.deque.top (S1, S2, S7, S12, S8) (S1, S2, S9, S8, S19, S16) 2. #.init.empty.enque.enque.enque.deque.deque.enque.empty (S1, S2, S9, S9, S7, S14, S12) 3. #.init.enque.enque.top.empty.top.enque.deque.deque.top.deque (S1, S2, S9, S8, S22, S20, S7, S14, S16, S19) 4.2 Invalid method sequences for class queue The list below contains all possible false and never constraints for the queue example. Never constraints: (S3, S4, S6, S10, S11, S13, S18, S21) Possibly False constraints: (S15, S17) Method sequences derived from possible false and never constraints are composed of a valid method sequence that is modified to be invalid. Hence, a method sequence is created that covers several valid (always or always/possible True) constraints and one invalid constraint. Suppose we want to create a method sequence that covers the possible false constraint: pf[dque; dque] Queue (q) is empty. We first start with a valid method sequence (sequence 1 below). To make sequence 1 invalid, three additional dque operations are added to the sequence. Now sequence 2 below satisfies all of the valid constraints in sequence 1 plus the invalid possibly false constraint, S #.init.empty.enque.enque.enque.deque.deque.enque.empty-valid (S1, S2, S9, S9, S7, S14, S12) 2. #.init.empty.enque.enque.enque.deque.deque.enque.empty.deque.deque.deque-invalid ( S1, S2, S9, S9, S7, S14, S12, S15) The (a/pt-s/pf/~) test criteria combines the (a/pt-s), and (pf/~) test criteria, thus a more extensive level of coverage is created. Method sequences that satisfy the (a/pt-s/pf/~) criteria are created by adding an invalid sequence that corresponds to a possible false or never constraint to the end of method sequence derived from the (a/pt-s) criteria. Other method sequences not shown in this paper can be found in [4]. 5. Empirical Study This section presents the results of our empirical study. The objective is to evaluate the effectiveness of the method sequence generation test criteria in detecting software faults. We

6 also consider other issues such as redundancy of method sequencing constraints and variations in the size of method sequence sets. When testing individual methods of a class, the goal is to examine whether a method returns the correct results and has the desired affect on the called object [8]. Method testing is typically done by treating each method as a function and mapping some input space to some output space, selecting elements of that input space, and examining the outputs for correctness [7]. There is nearly universal agreement that the class is the smallest testable unit and that testing methods in isolations is not effective [2, 3]. In our approach we assume that using constraintbased testing is sufficient for method and intra-class testing. A valid method sequence is used to cover "always" and "possibly true" constraints, and an invalid method sequences to cover "always" and "possibly true" constraints before the end of the sequences and a "never" or "possibly false" constraint at the end of the sequences. When a valid method sequence S for a class C is used to test an implementation of C, an exception is raised if the execution of C according to S detects an error (due to error checking inside C). When an invalid method sequence S' for C is used to test an implementation of C, an exception is raised if the execution of C according to S' results in one of the following two conditions: 1) this execution detects an error before the execution of the last method of S', and 2) this execution does not detects an error during the execution of the last method of S'. Note that condition 2) implies that the implementation of C has successfully executed an invalid method sequence for C. In our empirical study, we implemented three C++ programs: 1) bound stack, 2) unbound queue, and a 3) banking example. The mutation tool Proteum [5] was used to create mutations of each program. Once the mutations were completed on the C programs the errors were translated back to C++. Method sequences are then derived from the method sequencing criteria for each of the three programs. 5.1 Types of Mutants To evaluate the effectiveness of different method sequence coverage criteria, four sets of mutants were developed for each example. Original The first set of mutants implement the original code in a self checking manner. For example, in the unbound queue, before an item is removed from the queue a check is made to determine if the queue is empty. There are no additional modifications to the source itself. Post Post-conditions defined in Larch/C++ [11] and Assertion Definition Language (ADL) [13] were used to implement the second set of mutants. In this set, post-conditions were added to the original self-checking code and rewritten in C++ and inserted into each method. Before any value is returned, post-conditions are checked to verify the global variables and data members have been properly updated before the next method is executed. By evaluating data before it is returned to another method, post-conditions can be used to check methods that return data as well as methods that do not return data. If the method contains mutation errors that affect global variables or data attributes, post-conditions will catch these errors and the program invokes an exception. Post-conditions mutants collect exceptions raised by the original code plus exceptions raised by the post-conditions. Oracle1 The third set of mutants implement a test oracle. The oracle is used to verify return values from each method. In this particular testing technique, methods that do not return values are not checked. For example, for the stack, the push method does not return a value. Hence, there is no way for the oracle to verify whether the correct value has been pushed onto the stack. In checking these types of errors, we rely on other methods (such as pop), to indicate errors in the

7 push method. Methods that return values back to the test driver program are written to a file. These return values are then compared to an oracle file for verification. If the results do not correspond with the test oracle, an exception is raised and recorded. Test oracle mutants collect exceptions raised by the original code plus exceptions raised by the oracle itself. Unlike the post-condition mutants, the test oracle will catch invalid returned values due to defects in the return statement. Oracle2 The fourth set of mutants implement post-condition checking and a test oracle. These mutants simply combine the implementations of the two techniques described above. This set of mutants collects exceptions raised by the original code self-checking code, exceptions raised by post-conditions and exceptions raised by the test oracle. 5.2 Mutation Errors The mutation tool "Proteum" allows mutants to be generated from 71 mutation operators [6]. As previously stated, "Proteum" is a C mutation tool and not a C++ mutation tool. Therefore, the mutation operators were applied to the body and functions of the C programs. These seeded faults were then transformed to the C++ examples. Further explanation of the selected mutant operators are listed in [4,6]. In transforming the mutants back to C++, several mutants were discarded due to compilation errors. We also removed functionally equivalent mutants. 5.3 Test Driver Program In order to test the methods within a class a test driver program is built for each example that initializes global variables to the appropriate values, invokes methods, executes method sequences and checks output. The test driver program initially reads pre-defined method sequences from a file that corresponds to member functions invocations. Once the object is instantiated and the method invoked, the driver program records any exceptions that are raised. All four sets of mutants were run on three datasets. Each dataset is composed of method sequences derived from specific test coverage criteria defined in Table 1. Dataset 1 The first dataset contains method sequences that correspond to relatively small data structures. Implementation of the queue and the stack were limited to five data items and the banking example created and manipulates four accounts. Dataset 1 contains method sequences that were derived to satisfy the following five criteria: (a), (a-s), (a/pt), (a/pt-s) and (a/pts/pf/~). In some instances it is impossible to cover just the always constraints. Due to the implementation of the banking example, it is impossible to cover all of the "always" constraints without covering some possible true constraints. In this case, it may be necessary to modify the criteria for deriving (a) and (a-s) method sequences to include coverage of only those possible true constraints that will enable coverage of the always constraints. Dataset 2 Dataset 2 contains method sequences that correspond to larger data structures. The maximum size of the stack and queue are increased to about twenty. To increase the size of the data structures, each constraint is covered multiple times. Dataset 2 contains method sequences to satisfy (a/pt) and (a/pt-s) criteria. In addition, method sequences in Dataset 2 are modified such that the constraints are repeated at a higher rate than sequences in Dataset 1. Dataset 3 Dataset 3 is designed to investigate placement of these non-attribute effecting methods for the Queue and the Stack class. Method sequences in Dataset 3 satisfy the same criteria as method sequences in Dataset 1, however non-attribute affecting methods are placed at the end of the sequences. These methods are used simply to report the state of the object and do not

8 affect the attributes of the objects. Dataset 3 is not run on the banking example because in the banking example there are no non-attribute affecting methods. 5.3 Data Analysis and Results For Dataset 1 we will first look at the effect of valid sequences in detecting faults. If an exception is raised by a mutant from a valid sequence, this mutant is killed. If no exception is raised, the mutant is considered live. Number of Sequences In comparing a concise set of test sequences to several sequences in all test criteria for Dataset 1, the data in Table 2 indicates a slight increase (2%-7%) in most of the criteria in the number of killed mutants as the number of sequences increases. Hence, in most cases several sequences are slightly better then 1 sequences in detecting faults. Comparing Criteria In looking at the differences in coverage criteria for Dataset 1, the results from Table 2 show that as the criterion for creating method sequences changes from (a) to (a/pt), in most cases more (3%-9%) mutants are killed for mutants that implemented either post-conditions or a test oracle. In addition, as the criterion for creating method sequences changes from (a-s) to (a/pt-s), between 3 13% more mutants are killed for mutants that implemented postconditions or a test oracle. Mutants that implement additional checking mechanisms such as a test oracle, postconditions or both post-conditions and a test oracle killed significantly more mutants than the original code alone for all test criteria. Post-conditions found between % of the errors within the program. If no additional checking is used only % of the errors are found using valid method sequences. The data indicates that test oracle1 and test oracle2 are the most effective, in that both techniques found 100% of the errors. However, since test oracle1 only implements checking of return values while test oracle2 implements both post-condition checking and checking of return values, we see that test oracle1 is just as effective as test oracle2 without the additional checking of post-conditions. Redundancy Dataset 2 addresses the issue of redundancy of sequences and it s effect on detecting faults. In examining redundancy, we compared method sequences in Dataset 1 that satisfy the (a/pt) and the (a/pt-s) criterion with the methods sequences in Dataset 2 that satisfy the same criterion. Previous research [7] has indicated that in general longer sequences may be more effective in detecting faults. The results in Table 2 indicate that in comparing these two datasets, we again see a very minor increase in the number of mutants killed. It is important to note that for the (a/pt-s) criterion, the test oracle1 and test oracle2 testing technique give us 100% coverage (all mutants are killed) without increasing the redundancy of the constraints. Hence, since full coverage can be achieved without increasing the redundancy of the constraints adding additional redundancy in the coverage of the constraints is not necessary. Non-Attribute Methods Dataset 3 addresses the issue of placement of non-attribute affecting methods. The results listed in Table 2 for Dataset 1 and Dataset 3 do not show any significant impact on placing these methods at the end or beginning of the sequence. Although no conclusive results are shown for Dataset 3, it is essential that these (non-attribute affecting) methods be tested. Hence, these methods should be included in the test sequence at least once. Invalid Method Sequences To determine the effectiveness of the method sequences, valid and invalid method sequences are executed on invalid programs. Note that both valid and invalid sequences are valid. Let M be a set of mutants, V a set valid sequences, and I a set of invalid sequences. Either a valid or an invalid sequence (or both) could kill mutants within set M. When we apply a valid

9 sequence to a mutant, this mutant is killed if an exception occurs. In looking at invalid sequences, assume that the invalid sequences are constructed by extending some valid sequences. When applying an invalid sequence (which is a valid sequence followed by an operation according to a pf or ~ constraint) to a mutant, this mutant is killed if: 1) An exception occurs before the last operation of I or 2) An exception does not occur after the completion of I. Thus, the number of mutants in M killed by the union of V and I =((the number of mutants killed by V )+(the number of mutants killed by I )) (number of mutant killed by both V & I ). The results in Table 3 show that invalid method sequences do increase the capability of detecting faults in the original code mutants. For post-conditions mutants, in most cases invalid method sequences do not increase the capability of detecting faults. 6.0 Conclusions Our intra-testing approach involves the following three steps: 1) generation of pre- and postconditions for methods in a class, 2) derivation of method sequencing constraints, and 3) generation of valid and invalid method test sequences. Step 1) requires the use of a formal specification notation. Step 2) was done manually in this paper, but it can be automated by using some logic induction tools. An alternative is to let the designer of a class manually derive method sequencing constraints according to informal specification of methods in the class. Step 3) was done manually in this paper. In [9] an algorithm is shown for test sequence generation using sequencing constraints for concurrent programs. This algorithm can be further modified to automatically generate method test sequences according to method sequencing constraints. The empirical results show that this technique of method sequence generation is very effective in detecting software faults and that in most cases generating several method sequences that cover the constraints at least once is very effective in fault detection. Testing mechanisms such as adding post-condition checking, a test oracle or a combination of both can significantly increase error detection. This work supported by a NASA fellowship and in part by NSF grant CCR References [1] A. V. Aho, J. E. Hopcroft, and J. Ullman, Data Structures and Algorithms, Addison-Wesley, Reading, MA, [2] S. Barbey, M. M. Ammann, A. Strohmeier, The Problematics of Testing Object-Oriented Software, Software Quality Management II, Building Quality into Software, Commuter Mech. Pub., Southampton, UK , [3] R. V. Binder, Testing Object-Oriented Software: A Survey, Software Testing, Verification and Reliability, Vol. 6, No 3, Sussex, England,, , September December, [4] F. J. Daniels, Producing Dependable Object-Oriented Software Using Testing & Design Patterns, Ph. D. Dissertation, Dept. of Computer Engineering, North Carolina State University, [5] M. E. Delamaro, and J. C. Maldanado, Proteum: A tool for Assessment of Test Adequacy for C Programs, Proceeding of the Conference on Performability in Computing Systems, East Brunswick, New Jersey, 79-95, [6] M. E. Delamaro, J. C. Maldanado and A. P. Mathur, Integration Testing Using Interface Mutation, ISSRE, White Plains, New York, , [7] R. Doong and P. G. Frankl, "The ASTOOT Approach to Testing Object-Oriented Programs," ACM Transactions on Software Engineering and Methodology, Vol. 3, No. 2, , April [8] S. P. Fledler, Object-Oriented Unit Testing, Hewlett-Packard-Journal, 69-74, [9] B. Karacali and K. C. Tai, "Automated Test Sequence Generation Using Sequencing Constraints for Concurrent Programs", to appear in Proc. Inter. Symposium on Software Engineering for Parallel and Dist. Sys, May [10] S. Kirani and W. Tsai, "Method Sequence Specification and Verification of Classes, JOOP, 28-38, Oct [11] G.T. Leavens, Larch/C++ Users Guide: An Interface Specification Language for C++, Iowa State Univ, [12] B. Meyer, Object-Oriented software Construction, Prentice Hall, Upper Saddle River, New Jersey, [13] [Sha94] S. Shanker and R. Hayes, Specifying and test software components using ADL, Technical Report SMLI TR-94-23, Sun Microsystems Laboratories, Inc. Mountain View, California, April [14] K.C. Tai and R. H. Carver, A Specification-Based Methodology for Testing Concurrent Programs, Proc. 5 th European Software Engineering Conference, Lecture Notes in Computer Science, Vol. 989, 1995, [15] R. H. Carver and K. C. Tai, "Use of sequencing constraints for specification-based testing of concurrent programs", IEEE Trans. Software Engineering, Vol. 24, No. 6, June 1998,

10 Criterion Always Coverage (a) Always-Several Coverage (a-s) Always/Possibly True Coverage (a/pt) Always/Possibly True-Several Coverage (a/pt-s) Possibly False/Never Coverage (pf/~) Always/Possibly True-Several/ Possibly False/Never Coverage a/pt-s/pf/~) Table 1: Method sequencing test criteria Definition The always coverage criterion requires that all always constraints be covered at least once with a minimum number of test sequences. The always-several coverage criterion requires that all always constraints be covered at least once with more than the minimum number of test sequences. The always/possibly true coverage criterion requires that all always and each possibly true constraints is covered at least once with a minimum number of test sequences. The always/possibly true-several coverage criterion requires that all always and each possibly true constraints be covered at least once in more than the minimum number of test sequences. The possibly false/never coverage criterion requires that all possibly false or each never constraints be covered once in one test sequence. The always/possibly true-several, possible false/never coverage criterion requires that all always/possibly true constraints be covered at least once in more than the minimum number of test sequences. In addition, each possibly false/never constraint must be covered in one test sequence. Dataset 1 Dataset 2 Dataset 3 Queue 155 mutants Stack 144 mutants Table 2: Comparison of test criteria for datasets 1, 2, and 3 Original Post Oracle1 Oracle2 killed live killed live Killed live killed live (a) (a-s) (a/pt) (a/pt-s) (a) (a-s) (a/pt) (a/pt-s) (a/pt) Bank 124 mutants (a/pt-s) Queue (a/pt) mutants (a/pt-s) Stack (a/pt) mutants (a/pt-s) Bank (a/pt) mutants (a/pt-s) Queue (a) mutants (a/pt) (a/pt-s) Stack (a) mutants (a/pt) (a/pt-s) Queue 155 mutants Stack 144 mutants Bank 124 mutants Table 3: Results for valid plus invalid sequences Original Post Oracle 1 Oracle 2 killed live killed live killed live killed live (a/pt-s) (pf/~) Total (a/pt-s) (pf/~) Total (a/pt-s) (pf/~) Total