Helping the Tester Get it Right: Towards Supporting Agile Combinatorial Test Design

Transcription

1 Helping the Tester Get it Right: Towards Supporting Agile Combinatorial Test Design Anna Zamansky 1 and Eitan Farchi 2 1 University of Haifa, Israel 2 IBM Haifa Research Lab, Israel Abstract. Combinatorial test design (CTD) is an effective test planning technique that reveals faulty feature interaction in a given system. CTD takes a systematic approach to formally model the system to be tested, and propose test cases ensuring coverage of given conditions or interactions between parameters. In this position paper we propose a framework for supporting agile CTD, a human-centered methodology, which takes into account the human tester s possible mistakes and supports revision and refinement. In this approach a combinatorial model of the system and test plans are constructed in an incremental and iterative way, providing the tester with the ability to refine and validate the constructions. We propose a formal framework which can be used as a theoretical foundation for the development of agile CTD support tools, and describe a use case of an envisioned tool. 1 Introduction As software systems become increasingly complex, verifying their correctness becomes more challenging. Formal verification approaches are highly sensitive to the size of complexity of software, and might require extremely expensive resources. Functional testing, on the other hand, is prone to omissions, as it always involves a selection of what to test from a potentially enormous space of scenarios, configurations or conditions that is typically exponential in nature. The process of test planning refers to the design and selection of tests out of a test space aiming at reducing the risk of bugs while minimizing redundancy of tests. Combinatorial Test Design (CTD) [6,10] is an effective test planning technique, in which the space to be tested, called a combinatorial model, is represented by a set of parameters, their respective values and restrictions on the value combinations [8]. CTD approaches can be applied at different phases and scopes of testing, including end-to-end and system-level testing and feature-, service- and application program interface-level testing. The main challenge of CTD consists of finding a small (and ideally minimal) set of test cases (a subset of the space to be tested), which ensures coverage of given conditions, or interactions between variables (such as pairs, three-way, etc.) Experiments show that a test set that covers all possible pairs of parameter

2 values can typically detect between 50 to 75 percent of the bugs in a program ([9]). Other experimental work has shown that typically all bugs can be revealed by covering the interaction of between 4 to 6 patameters [3]. One important task in CTD is the construction of a combinatorial model of a system, which includes a set of parameters, their respective values and logical restrictions on value combinations. Typically, the parameters of the model do not map directly to user inputs of the system, but are rather a high-level abstraction of them. Identifying correctly the set of parameters for the model, their values - and most importantly - the restrictions on them is a laborious task which requires abstract thinking and a considerable level of familiarity with formal logic. Another task is the planning of tests over the constructed model, which satisfy a chosen coverage requirement (e.g., pairwise coverage). The complex and error-prone nature of both of these tasks leads to the need for automatic tools supporting the human tester. While there are numerous studies on human errors (see, e.g. [1,5]), and in particular on human factors in software development ([7,4]), they have been only marginally addressed in the context of combinatorial test design. In this position paper we propose an explicitly human-centered methodology for CTD, which takes into account the possibility of the tester s error and provides the tester with systematic tools to refine and validate the constructed models and test plans. We propose the term agile test design to refer to such approach, reflecting the incremental and interative way in which models and test plans are constructed. We propose a formal framework which can be used as a theoretical foundation for the development of agile CTD tools, and describe a use case of an envisioned tool. 2 A Vision for Agile Test Design Typically, the CTD methodology is applied in the following stages. First the tester constructs a combinatorial model of the system by providing a set scenarios which are executable in the system. After choosing the coverage requirements, the second stage is constructing a test plan, i.e., proposing a set of tests over the model, so that full coverage with respect to a chosen coverage strategy is achieved. In practice, taking into account the human factor [1,5], errors are possible at both of these stages. Moreover, error discovery in the test plan may cause the tester to return to the model and refine it, and vice versa. To reflect the iterative and incremental nature of the two stages of CTD, we propose the term agile test planning. This notion is based on the assumption that the tester s activity is not errorproof: errors can happen, both in the model and the test plan, and should be taken into account. Therefore, correctness of the combinatorial model is not assumed at the stage of test planning, and the tester may go back to refining the model at any point. The basic unit manipulated by the tester is a test. A test is represented by a vector of values assigned to systems parameters. The combinatorial model of the system is a set of all tests which describe possible (or executable) behaviors

3 of the system. A test plan is a set of tests out of the model space which satisfy some chosen coverage policy (such as pairwise testing). We divide the space of tests into three basic types, according to the information available from the tester: 1. validated - these are the tests that the tester confirmed as executable (according to some chosen confirmation strategy). 2. rejected - these are the tests that the tester rejected as impossible, either by explicitly removing them from the model, or by providing a logical condition that rules them out. 3. uncertain - these are the tests for which not enough information has been provided to classify them as validated/rejected. Agile test design, therefore, can be thought of as an iterative process, the goal of which is to validate or reject each possible test, arriving at a coherent combinatorial model of the system, together with a test plan satisfying the chosen coverage. Referring to the above three types of tests as colors: green, red and yellow respectively, the goal is eliminating all yellow tests by turning them either to green or to red. This can be done by raising a series of questions to the tester, which help determine if tests are missing in the test plan (and so the combinatorial model should be expanded, turning yellow to green), or the combinatorial model contains non-executable tests (and so it should be reduced, turning some yellow tests to red). The methodology described above allows for a great amount of flexibility, both for user querying strategies, and for validation/rejection strategies. For instance, we may want to minimize the time it takes the tester to arrive to a coherent solution, or the cognitive load of the tester (by presenting him only small portions of tests in each interation). 3 The Formal Framework 3.1 Combinatorial Models and Test Plans Definition 1. (System space) A system space is a a finite set of system parameters P = {A 1,..., A n } together with their corresponding associated values {V(A 1 ),..., V(A n )}. For any value a V(A i ), we say that a has type A i, denoted by type(a) = A i. Definition 2. (Interactions, scenarios) An interaction is an element I m 1 V(A i) such that for distinct a, b I, type(a) type(b). An interaction of size n (where n is the number of system parameters) is called a scenario. Example 1. As our running example, let us consider a system in which there are two servers S 1 and S 2, which can be either active (up) or inactive (down). Moreover, there are two operations Send and Ping, which can be performed by either one of the servers. We can set the system parameters to P = {S 1, S 2, Op}, where:

4 V (S 1 ) = {up 1, down 1 } V (S 2 ) = {up 2, down 2 } V (Op) = {Send 1, Send 2, P ing 1, P ing 2 } So, e.g., an assignment S 1 = up 1, S 2 = down 2 and Op = Send 1 represents a situation where the first server is up, performing the send operation and the second is down. Moverover, {up 1, Send 0 } and up 1, down 2, P ing 0 } are interactions; the latter is also a scenario. Definition 3. (Coverage) We say that a set of scenarios T covers a set of interactions C if for every c C there is some t T, such that c t. Example 2. Let the global system space be P = {FileOps, PathName, OS}, where V(FileOps) = {open, close, read, write} V(PathName) = {relative, absolute} V(OS) = {unix, windows} Then we can define interactions I 1 = {open, relative} and I 2 = {close, absolute}. We can define further define scenarios t 1 = {open, relative, unix} and t 2 = {close, absolute, windows}. Note that {t 1, t 2 } covers {I 1, I 2 } Definition 4. (Combinatorial model) A combinatorial model E is a set of scenarios (which defines all scenarios executable in the system). Definition 5. (Test plan) A test plan is a triple P = (E, C, T ), where E is a combinatorial model, C is a set of interactions called coverage requirements, and T is a set of scenarios called tests, where T covers C. Example 3. The most standard coverage requirement in the domain of combinatorial test design is pairwise testing [10,9]: considering every (executable) pair of possible values of system parameters. In terms of the above definition, a pairwise test plan can be formulated as any pair of the form P = (E, C pair (E), T ), where C pair is the set of all interactions of size 2 which can be extended to scenarios from E. 3.2 Representing Models and Plans as Logical Theories Recall that P = {A 1,..., A n } is our set of system parameters and {V(A 1 ),..., V(A n )} their corresponding ranges of values. Definition 6. F P, the set of well formed formulas of P is defined inductively as follows:

5 A i : a F for every 1 i n and a V P (A i ). We call such formulas atomic. If ψ, F P, then ψ, (ϕ ψ), (ϕ ψ), (ϕ ψ) F P. When the parameter is clear from context, we shall write just a instead of A i : a. So examples of formulas for P defined in Example 1 are: open, relative, ((open relative) unix). Every formula of F P naturally induces a set of scenarios (i.e., assignments to system parameters, or in other words, tests) in the following sense: Definition 7. Let ψ F P and let s be some scenario. We say that s satisfies ψ, denoted by s = ψ iff: ψ = A i : a and a s. ψ = ϕ 1 ϕ 2, s = ϕ 1 and s = ϕ 2. The rest of the cases are defined similarly. For a formula ψ, we define mod(ψ) = {s s = ψ}. For a theory Γ = {ψ 1,..., ψ n }, we denote mod(γ ) = mod(ψ 1 )... mod(ψ n ). The above definition provides a useful link between logical theories and sets of tests: a set of formulas Γ naturally defines a subset of tests by mod(γ ). Thus the tester may use formulas pf the above form to specify sets of tests of interest. 3.3 Agile CTD: Iterative Uncertainty Elimination Let us now describe the framework of agile CTD more formally, using the notions introduced above. The goal of the tester is to provide a valid test plan P = (E, C, T ). By a validity we mean here that (i) T satisfies all coverage requirements in C, and (ii) T is a subset of E. A coherent solution is reached when both (i) and (ii) are satisfied, if one of them is violated our envisioned tool supports an interative resolution of such violations. It is important to note that in our framework we do not assume the availability of E, the combinatorial model, rather it is extracted when the tester specifies a set of specific test cases T, as well as some logical restrictions (in the form of formulas as defined above) on the combinatorial model, which provide only partial information about E. Once the tester is satisfied with the constructed plan, a support tool may be invoked. The tool may automatically extract the combinatorial model from the provided tests and logical restrictions. Each test in the model has a status: those appearing in the test plan are validated (green), those ruled out by the logical conditions are rejected (red), the remaining ones are uncertain (yellow). The tool then may raise a sequence of questions that help determine the status of each of the yellow tests in the model. Different sequences may be proposed according to different considerations.

6 3.4 A Possible Use Case Let us describe a possible agile test planning use case. Suppose the coverage requirement is pairwise coverage. Returning to the system from Example 1, suppose that not all test cases are executable and further restrictions should be imposed. For instance, assume that server 1 can perform both send and ping operations, while server 2 can only perform ping operations. Moreover, only one of the servers can be up at the same time. If these are the only restrictions, the set of the executable scenarios can be described (in the sense of Definition 7) by the logical theory Γ = {up 1 down 2, up 2 P ing 0 P ing 1 }. Suppose, however, that the tester erroneously thinks that all combinations are possible and provides no logical restrictions for the model at this point. This induces the whole set E 0 = V (S 1 ) V (S 2 ) V (Op) as the underlying combinatorial model. The tester further proposes the following test cases: S 1 S 2 Op down 1 up 2 Ping 0 down 1 up 2 Ping 1 up 1 down 2 Send 0 up 1 down 2 Send 1 up 1 down 2 Ping 0 up 1 down 2 Ping 1 Once the tester submits the test plan, the six tests above are colored green (validated) in the model, the rest remaining yellow as no additional restriction ruling them out have been provided. At this point the tester s mistake may be discovered, as pairwise coverage is not achieved: the following interactions remain uncovered: {up 1, up 2 }, {down 1, down 2 }, {up 2, Send 0 }, {up 2, Send 1 } This could be either due to the fact that the tester forgot to add some tests, or he intentionally left them out as they are not executable in the system. In the former case he will add further concrete test cases, which will turn green in the model. In the latter case, however, he needs to refine the combinatorial model of the system, either explicitly or by adding logical restrictions, which will make some tests red in the model. In any case, the level of uncertainty (the number of yellow-colored tests) will be reduced at each interation. To maintain scalability to large parameter sets, a useful strategy can be to consider smaller projections at each iteraction. So, e,g, the tool may focus on the projection {S 1, S 2 }, asking the tester whether he wants to add tests to cover the interactions {up 1, up 2 }, {down 1, down 2 }. A negative answer implies that some scenarios should be excluded from the model. In this case a logical condition up 1 down 2 explaining this discrepancy can be suggested to refine the model. It is an interesting direction for further investigation to investigate strategies for

7 looking for the logical conditions which would make the most sense to the tester. Suppose that the user may now accept this model refinement, leading to further yellow tests turning red. The remaining missing interactions {up 2, Send 0 }, {up 2, Send 1 } are in the projection {S 2, Op}, on which the tool may focus next. We can again ask the tester whether he wants to add tests to cover them. He replies negatively, and the logical condition up 2 P ing 0 P ing 1 explaining this discrepancy can be suggested to refine the model. 4 Summary and Future Work The tester s main tasks in combinatorial test design, which include modelling the system and test planning, are laborious and error-prone. In this position paper we presented a vision for automatic support tools for CTD testers, which support an agile iterative test design. This approach takes into account the errorprone nature of the human tester s tasks, and supports the tester in refining and correcting his outputs by proposing a series of queries. We proposed here a framework for formalization of agile CTD, which can be used as a theoretical basis for developing future automatic support tools. An immediate direction for further research is an implementation of a prototype of the proposed tool. We plan to implement it using the environment of IBM Functional Coverage Unified Solution (FoCuS [2]), which is a tool for test-oriented system modeling, for model based test planning, and for functional coverage analysis. Another challenge is proposing methods to quantify the added value such tool may provide to the human testers. A starting point here could be proposing measurable criteria for the quality of combinatorial models. These criteria could then be used to compare manually constructed combinatorial models to models constructed with the help of our tool. When developing tools to support agile CTD, there is a level of freedom when considering different minimization strategies, which in their turn induce the order and type of questions posed by the tool to the user. One possible strategy is the minimization of time to complete a test plan. In this case questions answers to which make more tests certain should be preferred. Another strategy could be minimization of the cognitive load of the tester. In this case we may propose first logical restrictions involving a small number of parameters, or capturing a small number of tests which can be vizualized on screen. Human factors have so far received little attention in combinatorial test design. It is our hope that this paper will start a discourse on the practical needs of testers in the process of test design and combinatorial system modelling. References 1. BS Dhillon. Engineering product usability: a review and analysis techniques. WSEAS TRANSACTIONS on CIRCUITS and SYSTEMS, 2:86 94, group.php?id=1871.

8 3. D Richard Kuhn, Dolores R Wallace, and Jr AM Gallo. Software fault interactions and implications for software testing. Software Engineering, IEEE Transactions on, 30(6): , Christoph Malz, Kerstin Sommer, Peter Göhner, and Birgit Vogel-Heuser. Consideration of human factors for prioritizing test cases for the software system test. In Engineering Psychology and Cognitive Ergonomics, pages Springer, Tina Mioch, Jan-Patrick Osterloh, and Denis Javaux. Selecting human error types for cognitive modelling and simulation. In Human modelling in assisted transportation, pages Springer, Changhai Nie and Hareton Leung. A survey of combinatorial testing. ACM Computing Surveys (CSUR), 43(2):11, Laleh Pirzadeh. Human factors in software development: A systematic literature review Itai Segall, Rachel Tzoref-Brill, and Aviad Zlotnick. Common patterns in combinatorial models. In Proceedings of the IEEE Fifth International Conference on Software Testing, Verification and Validation (ICST), pages IEEE, KuoChung Tai and Yu Lie. A test generation strategy for pairwise testing. IEEE Transactions on Software Engineering, 28(1): , Jian Zhang, Zhiqiang Zhang, and Feifei Ma. Introduction to combinatorial testing. In Automatic Generation of Combinatorial Test Data, pages Springer, 2014.