Automatic Test Case Generation for OCL: a Mutation Approach

Transcription

1 UNU/IIST International Institute for Software Technology Automatic Test Case Generation for OCL: a Mutation Approach Percy Antonio Pari Salas and Bernhard K. Aichernig May 2005 UNU-IIST Report No. 321 R

2 UNU-IIST and UNU-IIST Reports UNU-IIST (United Nations University International Institute for Software Technology) is a Research and Training Centre of the United Nations University (UNU). It is based in Macau, and was founded in It started operations in July UNU-IIST is jointly funded by the Governor of Macau and the governments of the People s Republic of China and Portugal through a contribution to the UNU Endownment Fund. As well as providing two-thirds of the endownment fund, the Macau authorities also supply UNU-IIST with its office premises and furniture and subsidise fellow accommodation. The mission of UNU-IIST is to assist developing countries in the application and development of software technology. UNU-IIST contributes through its programmatic activities: 1. Advanced development projects, in which software techniques supported by tools are applied, 2. Research projects, in which new techniques for software development are investigated, 3. Curriculum development projects, in which courses of software technology for universities in developing countries are developed, 4. University development projects, which complement the curriculum development projects by aiming to strengthen all aspects of computer science teaching in universities in developing countries, 5. Schools and Courses, which typically teach advanced software development techniques, 6. Events, in which conferences and workshops are organised or supported by UNU-IIST, and 7. Dissemination, in which UNU-IIST regularly distributes to developing countries information on international progress of software technology. Fellows, who are young scientists and engineers from developing countries, are invited to actively participate in all these projects. By doing the projects they are trained. At present, the technical focus of UNU-IIST is on formal methods for software development. UNU-IIST is an internationally recognised center in the area of formal methods. However, no software technique is universally applicable. We are prepared to choose complementary techniques for our projects, if necessary. UNU-IIST produces a report series. Reports are either Research R, Technical T, Compendia C or Administrative A. They are records of UNU-IIST activities and research and development achievements. Many of the reports are also published in conference proceedings and journals. Please write to UNU-IIST at P.O. Box 3058, Macau or visit UNU-IIST s home page: if you would like to know more about UNU-IIST and its report series. Chris George, Acting Director

3 UNU/IIST International Institute for Software Technology P.O. Box 3058 Macau Automatic Test Case Generation for OCL: a Mutation Approach Percy Antonio Pari Salas and Bernhard K. Aichernig Abstract Fault-based testing is a technique where testers anticipate errors in a system under test in order to assess or generate test cases. The idea is to have enough test cases capable of detecting these anticipated errors. This work presents a method of fault-based test case generation for pre- and postcondition specifications. Here, errors are anticipated on the specification level by mutating the pre- and postconditions. We present the underlying theory by giving test cases a formal semantics and translate this general testing theory to a constraint satisfaction problem. A prototype test case generator serves to demonstrate the automatization of the method. It works on OCL specifications.

4 Percy A. Pari Salas is a Fellow at UNU/IIST on leave from San Pablo Catholic University, Arequipa - Peru, where he is a lecturer in Informatics Engineering. His current research interests include software testing, specially mutation analysis, formal development methods and formal specification languages. His address is percy@iist.unu.edu. Bernhard K. Aichernig is a Research Fellow at UNU/IIST. He is also an assistant professor at the Institute for Software Technology at the Graz University of Technology in Austria. His research interests include the synergies of testing and formal development methods, techniques of refinement, and requirements engineering supported by formal specification languages. His address is bka@iist.un.edu. Copyright c 2005 by UNU-IIST, Percy Antonio Pari Salas and Bernhard K. Aichernig

5 Contents i Contents 1 Introduction 1 2 The Object Constraint Language (OCL) 2 3 Fault-based Test Case Generation Testing from Specifications Test Cases Constraint Solving Test Case Generation Algorithm The Tool The Translator OCL-subset Parser DNF Partitioning The Constraint System Generator The Constraint Handler The Constraint Theory JCHR: the CHR Java Implementation Dealing with Several Types: The Class Variable The Specialized Constraint Handler The Search Engine JASE: The Java Abstract Search Engine A Customized Search Engine Example The Triangle example The Granting Object Privileges example Related Work 36 7 Conclusions 37 A Constraint System Java Program 41

6

7 Introduction 1 1 Introduction Testing is a highly controversial subject with respect to verification. The reason is that, in general, it cannot demonstrate the absence of faults. This can only be achieved by (formal) correctness proofs. However, in practice it is in nearly all cases the only verification technique applied. Taking into account the obvious need of practitioners, Formal Methods have included testing into their realm over the last decade. This work placed testing on a solid foundation leading to a series of testing tools that work on the specification level. Solid theoretical foundation leading to tool development is a characteristic we also have into account. The present report presents a theoretical part that relates fault-based testing to the theory of designs, and a practical part that applies this new testing theory to formal specifications written in OCL. In the theoretical part we exploit a test hypothesis: We assume that we can anticipate the possible errors made by an implementer. This technique is generally known as fault-based testing and its most prominent form is mutation testing. Originally in mutation testing, a program text has been altered (mutated) in order to check if a given set of test cases is able to detect this change. In our work, we aim to mutate the specifications, in order to model errors that can happen during the development process. Furthermore, we go a step further, by not only analyzing the coverage of given test cases, but generate test cases that will cover the introduced errors. Thus, we challenge Dijkstra s famous sentence and claim: Testing can show the absence of errors, if we have a knowledge of what can go wrong. Ideas of what can go wrong, can be systematically derived from formal models of an implementation. By introducing possible faults, test cases can be generated that will ensure that this kind of fault does not survive. The theoretical part is strongly based on previous work of the second author. The second author has developed a general theory of fault-based testing [5], in which it can be formally proved that test cases will detect certain faults. This theory is based on the general concept of refinement (implementation relation) and can be instantiated to different semantic frameworks where this concept exists. The present report shows an instantiation to pre-postcondition specifications in first-order logic. The practical part of the present work describes the development of a prototype tool that implements the theory in order to apply it to model-based specifications. The tool builds on three pillars: (1) the mentioned fault-based testing theory [5] to state the problem; (2) Dick and Faivre s classical work on specification-based testing [13] to divide the problem; and (3) on Constraint Logic Programming [28] to search for solutions. The Object Constraint Language (OCL) is the target specification language used in the practical part of our work. The Object Constraint Language as part of the UML standard [21] is a language allowing the specification of formal constraints on UML models. As UML has become a de facto standard in object-oriented analysis and design it is expected that the relatively new OCL will experience a similar popularity in the near future. The presented work aims to add further value to the use of OCL by allowing it to be used in the testing process. The remaining of this work is structured as follows. In Section 2 of this report we briefly present a description of OCL as a model-based specification language. Section 3 links concepts on testing with the theory of designs and presents an algorithm that generates test cases that will find anticipated errors

8 The Object Constraint Language (OCL) 2 in a design. In Section 4 a prototype tool that implements our testing theory is described and its most important implementation details are explained. In Section 5 an example shows how the approach and the tool work and can be useful. Related work is cited and described in Section 6. Finally, in Section 7 we conclude and make suggestions for future work. 2 The Object Constraint Language (OCL) The Object Constraint Language is part of the UML 2.0 standard. It is a language allowing the specification of formal constraints in context of a UML model. Constraints are primarily used to express invariants of classes, and pre- and postconditions of operations. The classic example of a Process Scheduler [13] serves to illustrate how textual OCL specifications may amend UML diagrams. For a more detailed introduction to OCL we refer to [29]. In a process scheduler, at any time, the system may have some processes ready to be scheduled, some processes waiting for some external action before they become ready and, optionally, a single active process. Each process is identified by a unique pid (process identifier). Figure 1 shows a simple UML class diagram that models the static view of this example. Figure 1: Process scheduler s class diagram. As can be seen in the class diagram, the system has two classes, Process and Scheduler. The class Process represents the processes and only contains one attribute, his process identifier (pid). The class Scheduler represents the entire system, it contains all the functionalities and is related to the processes by three associations. The associations represent the processes ready to be scheduled, the waiting processes and the optional active process. The multiplicities at the association ends define that there are two possibly empty sets (zero or many multiplicity) of ready and waiting processes, and a set of active processes constrained to have at most one element (zero or one multiplicity). But the example is not that simple. It involves constraints that cannot be expressed in a class diagram: There is a state invariant indicating that the two sets of ready and waiting processes are always disjoint,

9 The Object Constraint Language (OCL) 3 that the active process is not ready or waiting and the only time that there is no active process is when there are no ready processes. The functionality of the system is described by the following three operations (as showed in the class diagram): NEW which introduces a new process in the system in waiting state, READY which moves a waiting process to a ready state, making it active if there is no current active process, and SWAP which swaps the currently active process for any of the ready processes and leaves the system idle if there is no ready process. It exists an additional operation INIT that puts the system into its initial state (empty of processes). An OCL specification for the system is necessary to express these requirements. This OCL specification is presented in Example 2.1 and discussed in the following. For convenience and in order to maintain a readable specification, processes (instances of the class Process) are referred to by using their process identifier pid modeled as integers. Example 2.1 context Scheduler inv: (ready->intersection(waiting))->isempty() and not ((ready->union(waiting))->includes(active) and (active = nil implies ready->isempty()) --Operations context Scheduler::Init() post: (ready->union(waiting)) = Set{} and active = nil context Scheduler::New(p:Process) pre: p <> active and not((ready->union(waiting))->includes(p)) post: waiting = waiting@pre->including(p) and ready = ready@pre and active = active@pre context Scheduler::Ready(p:Process) pre: waiting->includes(p) post: waiting = waiting@pre->excluding(p) and if active@pre = nil then (ready = ready@pre and active = p) else (ready = ready@pre->including(p) and active = active@pre) endif context Scheduler::Swap() pre: active <> nil post: if ready@pre->isempty() then (active = nil and ready = Set{}) else ( ready@pre->includes(active) and ready = ready@pre->excluding(active)) endif and waiting = waiting@pre->including(active@pre) Since classical UML and OCL use different approaches (visual model in UML and a text based specification in OCL), the relations between OCL constraints and the model elements in the diagrams must be clear. The link between OCL and a model element is called the context definition of an OCL expression.

10 The Object Constraint Language (OCL) 4 The context definition in OCL specifies the model element for which the OCL expression is defined. As shown in the example the keyword context followed by the model element s name is used to define the context. In the example two types of contexts are present, the class context (context Scheduler) and the operation context (context Scheduler::Init() for example). As the main goal of this work is centered on unit testing, classes and operations are mainly used. As pointed out before and can be seen in the OCL specification, formal constraints over classes are expressed in form of invariants. Pre- and postconditions, then represent constraints over operations in classes. In an OCL software specification an invariant is a condition that must hold always before and after the execution of an operation. Preconditions define a condition that is assumed to hold before execution of an operation. And postconditions define conditions that must hold in the object s state after an operation execution (by relating it to the state before execution). In the postcondition statements the is added to some variable names for referencing the value of the variable in the state before execution. Probably the most important characteristic of OCL resides in its mathematical foundation. Although OCL does not use mathematical notation, it has a formal mathematical semantics based on set theory and predicate logic. This semantics was first proposed by Richters [23] and then adopted by the Object Management Group that defines standards for UML. This fact is essential for this work. Indeed, it defines the domain for which the Constraint Solver must be developed and the type of constraint rules that must be implemented on it. Part of the OCL semantics is described later in this section. Another useful property of OCL is that it is a strongly typed language. As a strongly typed language, every OCL expression has a type describing the domain of the result value and a set of applicable operations. Two kinds of types can be found in OCL expressions: basic types and collection types. In OCL, predefined basic types are Boolean, Integer, Real, and String. Every type has a set of related operations that can be used in an OCL specification. Types defined in a UML model, like classes and enumerations, are also available in the OCL specification. Types derived from class declarations also have a set of operations linked to them. Query operations can be invoked over these types. Enumeration types define a set of literals and no operations are linked to them. Collections of values can be described in OCL by the collection types Set(t), Bag(t), OrderedSet(t) and Sequences(t), representing mathematical sets, multisets and lists. Collection types are parameterized by a type parameter t. The parameter t denotes the type of collection elements. OCL syntax for these types and operations can be seen in the OCL specification of the scheduler s example. It was explained before, that in the context of the class Scheduler, associations with processes (ready and waiting) define sets (a collection type). OCL defines standard operators on these sets (as on other collection types) that can be applied by using the syntax collectiontype operation(parameters) Note that the symbol is used only to invoke collection type operations. Any other operation calls (calls to operations for basic and user types) are made using the. (dot) operator. In addition to basic and collection type operations, OCL defines a set of operations called loop operations or iterators.

11 Fault-based Test Case Generation 5 3 Fault-based Test Case Generation 3.1 Testing from Specifications The vocabulary of computer scientists is rich with terms for naming the unwanted: bug, error, defect, fault, failure, etc. are commonly used. Here, we adopt the standard terminology as recommended by the IEEE Computer Society: Definition 3.1 An error is made by somebody. A good synonym is mistake. When people make mistakes during coding, we call these mistakes bugs. A fault is a representation of an error. As such it is the result of an error. A failure is a wrong behavior caused by a fault. A failure occurs when a fault executes. In this work we aim to generate test-cases on the basis of possible errors during the design of software. Examples of such errors might be a missing or misunderstood requirement, a wrongly implemented requirement, or simple coding errors. In order to represent these errors we will introduce faults into formal specifications. The faults will be introduced by deliberately changing a design, resulting in wrong behavior possibly causing a failure. In this work we restrict ourselves to model-based specifications. More precisely, we use the design calculus of the Unifying Theory of Programming [17] to assign specifications a precise semantics. Designs are a special form of predicates with a pre- and postcondition part. The free variables of a design (predicate), called the alphabet, represent state variables before (undecorated variable names) and after execution (decorated variable names) of a program. In addition, special Boolean variables ok and ok denote the successful start and termination of a program. Formally, we define Definition 3.2 (Design) Let P and Q be predicates not containing ok or ok. P Q = df (ok P ) (ok Q) A design is a relation whose predicate is (or could be) expressed in this form. Implication establishes a refinement order (actually a lattice) over designs. Thus, more concrete implementations imply more abstract specifications. Definition 3.3 (Refinement) Let D 1 and D 2 be designs D 1 D 2 = df v, w, A D 2 D 1, for all D 1, D 2 with alphabet A. Alternatively, using square brackets to denote universal quantification over all variables in the alphabet, we write [D 2 D 1 ]. Obviously, this gives the well-known properties that preconditions are weakened under refinement and postconditions are strengthened (become more deterministic): Theorem 3.1 (Refinement of Designs) [(P 1 Q 1 ) (P 2 Q 2 )] iff [P 2 P 1 ] and [(P 2 Q 1 ) Q 2 ]

12 Fault-based Test Case Generation 6 For a definition of common operators on designs, like sequential composition, conditional, choice, etc., we refer to [17]. According to Definition 3.1, faults represent errors. These errors can be introduced during the whole development process in all artifacts created. Consequently, faults appear on different levels of abstraction in the refinement hierarchy ranging from requirements to implementations. Obviously, early introduced faults are the most dangerous (and most expensive) ones, since they may go undetected into an implementation; or formally, a faulty design may be correctly refined into an implementation. Again refinement is the central notion in order to discuss the roles and consequences of certain faults and design predicates that are most suitable for representing faults. Definition 3.4 (Design Fault) Given an intended design D, and an unintended design D in which during creation an error was made. Then, we define a fault in design D as the syntactical deviation in D, if and only if refinement does not hold: D D We call D a faulty design. Consequently, not all errors lead to faults. Here, for being a fault, a possible observation of this fault must exist. Note also, the use of the term intended (unintended) in the definition, instead of correct (incorrect). This is necessary, since the latter is only defined with respect to a given specification, but faults can already be present in such specifications from the very beginning. 3.2 Test Cases We take the point of view that test cases are specifications that for a given input define the expected output. Consequently, we define test cases as a sub-theory of designs. Definition 3.5 (Test Case, deterministic) Let i be the input vector and o be the expected output vector, both being lists of values, having the same length as the variable lists v and v respectively. Furthermore, equality over value lists should be defined. t d (i, o) = df v = i v = o Although sufficient for deterministic programs, test cases derived from a specification have to take nondeterminism into account. Therefore, we generalize the notion of a test case as follows: Definition 3.6 (Test Case, general) Let i be the input vector and O a finite set containing the expected output vector(s). Both, i and o o O, are lists of values, having the same length as the value lists v and v respectively. t(i, O) = df v = i v O Previous work of the second author [3] has shown that refinement is the key to understand the relation between test cases, specifications and implementations. Refinement is an observational order relation,

13 Fault-based Test Case Generation 7 usually used for step-wise development from specifications to implementations, as well as to support substitution of software components. Since we view test cases as (a special form of) specification, it is obvious that a correct implementation should refine its test cases. Thus, test cases are abstractions of an implementation, if and only if the implementation passes the test cases. This view can be lifted to the specification level. When test cases are properly derived from a specification, then these test cases should be abstractions of the specification. Formally, we define: Definition 3.7 Let T be a set of test cases, S a specification and I an implementation, all being designs, and T S I we define T as the correct test set with respect to S, implementation I passes the test cases in T, implementation I conforms to specification S. Finding a test case t that detects a given fault is the central strategy in fault-based testing. For example, in classical mutation testing, D is a program and D a mutant of D. Then, if the mutation in D represents a fault, a test case t should be included to detect the fault. Consequently, we can define a fault-based test case as follows: Definition 3.8 (Fault-adequate Test Case) Let t be either a deterministic or non-deterministic inputoutput test case. Furthermore, D is a design and D its faulty version. Then, t is a fault-adequate test case when t D (t D ) We say that a fault-adequate test case detects the fault in D. Alternatively we can say that the test case distinguishes D and D. In the context of mutation testing, one says that t kills the mutant D. All the test cases that cover a certain fault form a fault-adequate equivalence class. Note that our definitions solely rely on the lattice properties of designs. Therefore, our fault-based testing strategy scales up to other lattice-based test models as long as an appropriate refinement definition is used. More precisely, this means that the refinement notation must preserve the same algebraic laws. 3.3 Constraint Solving The problem of generating test cases from a formal specification can be represented as a Constraint Satisfaction Problem (CSP). A constraint satisfaction problem consists of a finite set of variables and a set of constraints. Each variable is associated with a set of possible values, known as its domain. A

14 Fault-based Test Case Generation 8 constraint is a relation defined on some subset of these variables and denotes valid combinations of their values. A solution to a constraint satisfaction problem is an assignment of a value to each variable from its domain, such that all the constraints are satisfied. Formally, the conjunction of these constraints form a predicate for which a solution should be found. We define: Definition 3.9 (Solution) Let P (v, v ) be a predicate, with v and v being lists of primed and unprimed free variables. Given a pair (x, X ), with x being a list of values of same length as v and X being a finite set of value lists with the same length than v. Then, (x, X ) is solution of P iff x X P (x, x ) We call (x, X ) a maximal solution, if X is maximal. A constraint satisfaction problem is always embedded into a Constraint System. A Constraint System formally specifies the syntax and semantics of the constraints of interest. It is defined [28] as a tuple (Σ, D), CT, C) where Σ is a signature that contains all possible constraint and function symbols, D is a domain together with an interpretation of the symbols in Σ, CT is a non-empty and consistent theory over Σ, C are the allowed constraints. A constraint solver implements an algorithm for solving well-formed constraints within a CSP in accordance with a constraint theory. The constraint theory CT defines the semantics of a constraint system and is composed, in our case, of a set of transformation rules over one or more specific domains. The syntax of well-formed constraints is defined by the set of allowed constraints C. Our approach is then to embed the test generation problem modeled as a CSP into a specially designed and implemented Constraint System. But this is not a novelty because this approach has been widely explored and implemented. The novelty in our approach is the relation that we formalize between faultbased testing and constraint solving. This relation is defined by the concept of a fault-adequate equivalence class. The definition of a fault-adequate equivalence class gives us a hint on what set of constraints should be used to form the CSP in order to guarantee that the constraint solver finds a solution (a test case) capable of discovering predefined faults inside design implementations. 3.4 Test Case Generation Algorithm From the definition of fault-adequate equivalence class we have developed an algorithm to generate a test case which is inside this equivalence class. Algorithm 1 Given a design D(P re P ost) and its faulty design D (P re P ost ) as inputs. All variables in their alphabet range over finite sets. Then, an input-output test case T is generated as follows: Step 1: A test case T is searched by:

15 Fault-based Test Case Generation 9 1. finding a pair (i c, o c ) being a solution of P re P ost P ost 2. If it exists, then the test case T = t(i, O) is generated by finding a maximal solution (i, O) of P re P ost (v = i c ) Step 2: If the former does not succeed, then we look for a test case T = t(i, O) with (i, O) being a maximal solution of P re P re P ost The algorithm is designed to produce test cases such that the original D refines it (T is a positive test case). Therefore, Step 1 involves two phases: finding a solution i c, o c and finding the test case t(i, O). In these step, the solution (i c, o c ) represents a counterexample of D D, with o c being one output (from a possible non-empty set of outputs) that differ from the original. A test case t(i c, o c ) would be a negative test case reporting when a test fails. However, since we are interested in positive test cases that predict all the possible outputs according to the original specification, T includes all outputs O satisfying the postcondition. This also prevents T from reporting false negatives. Note that this algorithm is partial, since the search space over the variables is restricted to a finite domain. We can say that if no test case is identified after those three steps, then the original and the mutant specifications are equivalent (in the context of the finite domains of the CSP). Next, we show that the test case generated by the algorithm is fault-adequate. Thus, we have to show that the test case is correct with respect to D as well as it covers the fault in D. We split this analysis into two theorems. Theorem 3.2 (Test Case Correctness) Given a design D(P re P ost) and its faulty design D (P re P ost ), and a test case t(i, O) generated by Algorithm 1 t(i, O) D Proof. The proof can be split into two cases reflecting Steps 1 & 2 of the algorithm: 1. From (i, O) is solution of P re P ost (v = i c ) it follows that (i, O) is solution of P re P ost. Thus, t(i, O) D = {by Theorem 3.1} [(v = i) P re] [(v = i) P ost v O] = {(i, O) is solution of (P re P ost) and O maximal} true true = true 2. follows the same style of reasoning as the first case.

16 The Tool 10 Theorem 3.3 (Fault coverage) Given a design D(P re P ost) and its faulty design D (P re P ost ), and a test case t(i, O) generated by Algorithm 1 t(i, O) D Proof. Again, this proof is split into the two cases of the algorithm: 1. When (i, O) is solution of P re P ost (v = i c ). This test case is only generated, if (i c, o c ) is solution of (P re P ost P ost ) exists. We see immediately that i = i c and {o c } O = {} and deduce t(i, O) D = {by Theorem 3.1} [(v = i) P re ] [((v = i) P ost ) (v O)] = v ((v = i) P re ) v, v ((v = i) P ost (v / O)) = {solution (i c, o c ) is a witness for 2nd. } = v ((v = i) P re ) true = true 2. When (i, O) is solution of P re P re P ost. t(i, O) D = {by Theorem 3.1} [(v = i) P re ] [((v = i) P ost ) (v O) = v ((v = i) P re ) v, v ((v = i) P ost (v / O)) = { i is a witness to 1st. } true v, v ((v = i) P ost (v / O)) = true 4 The Tool This section describes implementation details of the prototype tool that serves to demonstrate the use of our testing theory. The tool consists of a Translator and a Constraint Solver. The Translator reads an OCL specification and translates it into Constraint Systems to be solved. The Constraint Solver is embedded into each generated Constraint System as the instantiation of two pre-built components: a Constraint Handler and a Search Engine. The graphical scheme in Figure 2 illustrates these components and their interrelations. In this section, the translator and the two components (constraint handler and search engine) are described in detail. 4.1 The Translator The Translator is the first part of our tool. It is in charge of transforming the constraints written in the input language into constraints inside a constraint system to be solved. The translation process is subdivided into: (1) the OCL parsing, (2) the DNF Partitioning and (3) the Constraint System Generation.

17 The Tool 11 Figure 2: Components of the generation tool.

18 The Tool OCL-subset Parser Our parser was built to recognize a subset of the OCL specification language. The OCL subset we used is defined formally by the grammar showed in Figure 3. This grammar is based on the general OCL grammar presented in [29]. As it can be seen, our tool recognizes and processes: invariants, preconditions and postconditions. Boolean expressions and operators are also supported, as well as if expressions, arithmetic and relational operators, and operators over sets (union, intersection, difference,... ). Moreover, it works with primitive data types (integers, reals, strings, Booleans) and collection types (sets). However, collection iterators, let expressions and links with other UML diagrams are not yet supported. oclcontext := "context" name "(" operparameters "):"vartype (stereotype)* operparameters := (vardeclaration)* vardeclaration := name ":" vartype stereotype := ("inv" "pre" "post") logicexpression logicexpression := relationalexpression (("and" "or" "implies") relationalexpression)? relationalexpression := additiveexpression ((">" ">=" "=" "<>" "<" "<=") addtiveexpression)* additiveexpression := multiplicativeexpression (("+" "-") multiplicativeexpression)* multiplicativeexpression := unaryexpression (("*" "/") unaryexpression)* unaryexpression := ("not" "-")? postfixexpression postfixexpression := primaryexpression ("->" propertycall )* primmaryexpression := name numint float string ( logicexpression ) propertycall := name "(" propcallparameters ")" propcallparameters := (logicexpression)* ifexpression := "if" logicexpression "then" logicexpression "else" logicexpression "endif" vartype := "Integer" "Real" "String" "Set" name := <STRING> numint := <INT> float := <REAL> string := " " <STRING> " " Figure 3: Subset of OCL. Our parser was generated automatically using the CUP Java Parser Generator. Semantic rules added to the generated parser rewrite the input constraints into a normalized structure (shown in Fig. 4). The classes defined in this structure conceptualize the meaning of an OCL description, and also helps the DNF partitioning process. In the following each of these classes are briefly described. OCLExpression Is an abstract class that represents any expression or the composition of expressions that results also in an expression. It is the direct result of an implementation of the Composite design pattern [15]. Conjunction Represents an expression of the form exp 1 exp 2. Disjunction Represents an expression of the form exp 1 exp 2. In this case the operator has been overloaded as shown in for representing DNF partitions of some expressions. Constraint Represents a single constraint, this is a valid constraint defined by the constraint system. In general, a constraint is a Boolean expression, but in some cases is an expression that states a calculation in addition to a equality relation. In this case, a variable is added to the list of parameters constraint(< parameters >, v). This expression is equivalent to v = constraint(< parameters >). However, the first expression states that the calculation and the equality are treated as an atomic operation with an implicit order of execution while the second states two different constraints with no order of execution.

19 The Tool 13 Disjunction 2 OCLExpression 2 1 Constraint * Conjunction Variable 0..3 OCLStereotype * OCLContext 4 Goals Figure 4: Internal structure for the OCL constraint store.

20 The Tool 14 Variable Indeed this class represents a value either known or unknown. OCLStereotype Constraints in OCL can refer to invariants, pre- and postconditions. The OCLStereotype is a container for an OCLExpression. It defines which OCL element (stereotype) the expressions refers to. OCLContext The context in OCL is the link between expressions and diagrammatic elements (classes, methods, etc.). As each context defines which expression stereotypes are allowed (e.g. pre- and postconditions are only allowed in the context of a method) our OCLContext is a container for the OLCStereotypes. Although the input model actually restricts the context to only methods, this class is important for future extensions to the tool. Goals This class is not part of the OCL model but is the link between the input model and the constraint system. It is a container for DNF partitioned sets of constraints. From the description above we can see that this structure helps to transform the input OCL expressions into a simple set of conjunctions and disjunctions of constraints. The generation of this structure also initiates the DNF partitioning process DNF Partitioning The approach of partition analysis of model-based specifications is very similar to the one used in [13]. The difference is that here preconditions, postconditions and invariants are first reduced separately without combining them (later they are used in conjunction into the constraint systems to specify test cases). As defined by Dick and Faivre, equivalence rules based on first order predicate calculus can be used to split test domain descriptions into partitions. These domain partitions are described by disjunct sets of constraints. In order to describe the rules that apply for domain partitioning, as in the work of Meudec [19], a partitioning function P is defined with signature: OCLexpression Partition. Besides, the operator is defined as: P artition P artition P artition P 1 P 2 = {c 1 c 2 c 1 P 1, c 2 P 2 } Also, for describing the partitioning rules the following notation is used: P artition = {P 1... P n } where the braces denote a set of partitions to which the distributed union operator is applied to obtain a total partition. In the following rewriting rules, let A, B, C be Boolean logic expressions:

21 The Tool 15 P (A) = {A} when no other rule applies (1) P (A B) P (A B) = P (A B) (2) P ( A B) { } P (A B) if A then B else C = (3) P ( A C) P (A B) = P ( A B) (4) P (A B) = P (A) P (B) (5) These rules are applied for rewriting specifications in order to obtain a set of disjunct invariants, preand postconditions for every operation. These rules, except for the rule (5), are hard-coded into the translator. The last rule P (A B) = P (A) P (B) is implemented by the following Partition Algorithm. Algorithm 2 Suppose we have a goal-constraints store C that is a set of constraint goals. Each constraint goal is a set of constraints. And suppose we have an expression (OCLExpression) exp to be partitioned. 1. If the expression exp is a constraint then add the constraint to each of the constraint goals in C. 2. If the expression exp is a conjunction then: (a) Apply the partition algorithm to the first child with C as the goal-constraints store. Let C be the goal-constraints store after this step is completed. (b) Apply the partition algorithm to the second child with C as the constraint store. 3. If the expression exp is a disjunction then: (a) Make a copy C copy of the goal-constraints store C. (b) Apply the partition algorithm to the first child with C as the goal-constraints store. Let C be the goal-constraints store after this step is completed. (c) Apply the partition algorithm to the second child with C copy as the goal-constraints store. Let C copy be the goal-constraints store after this step is completed. (d) Unify the constraint store C := C C copy. Note: Disjunction in this algorithm refers not to the logical operation but to a disjunct partition. An example of how the parser and the algorithm work together to get DNF partitions of OCL statements will clarify this. Suppose we have the OCL statement if a>b then result=a else result=b endif and x=true The OCLParser will read it and build in memory the Abstract Syntax Tree in Fig. 5. Note that the OCLParser has hardcoded into it the first four partition rules and the AST has only conjunctions,

22 The Tool 16 Conjunction A Disjunction Constraint eq(x,true) Conjunction Conjunction Constraint lt(a,b) B Constraint Constraint Constraint eq(result,a) ge(b,a) eq(result,b) Figure 5: AST of the OCLExpression if a>b then result=a else result=b endif and x=true disjunctions and constraints as elements. We must also note that disjunction in this tree represents an exclusive OR operation among partitions. So, the subtree labeled as A results from applying the partition rule (3) where the constraint labeled as B is equivalent to gt(a,b). All the negated constraints are solved while the AST is being built. After been generated by the OCLParser, the AST executes the Partition Algorithm that is the implementation of the rule (5) and give as output a set of constraint goals. The constraint store C is initialized containing one empty constraint goal C = { {} }. The execution of the algorithm is a recursive in-order traverse of the AST. In this example, the constraint store C resulting from the processing of the first child of the disjunction is C = { {gt(a, b), eq(result, a)} } and the constraint store C copy built from the processing of the second child is C copy = { {ge(b, a), eq(result, b)} } Having defined C and C copy, the resulting constraint store C just after the disjunction has been processed is the unification of the former two C = { {gt(a, b), eq(result, a)}, {ge(b, a), eq(result, b)} } Finally, the constraint store built from the whole OCL expression, where the conjunction after the if statement has been distributed among the partitions is C = { {gt(a, b), eq(result, a), eq(x, true)}, {ge(b, a), eq(result, b), eq(x, true)} } Once the set of goal-constraints has been generated each goal-constraint has to be embedded inside a proper Constraint System.

23 The Tool 17 class CS { \\ Constraint System initialization and Constraint Handler instantiation code... \\ Domains initialization code... \\ Variable declarations... \\ Goal constraints... \\ Search Engine instantiation/execution code... \\ Test cases generation code... } Figure 6: General structure of a generated Constraint System Java program The Constraint System Generator The Constraint System Generator generates the predicates used in Algorithm 1 from which test cases are going to be generated. So far, our tool has read the model-based specifications (the original and the mutant specifications) from the input files. These input specifications have been normalized into a structure containing just conjunctions and disjunctions. And then, the DNF partitioning process has generated a disjunct set of goal-constraints. Now, the tool combines the DNF partitions into constraint systems that represent the Fault-adequate Equivalence Class. Moreover, the tool adds to every generated constraint system a set of constraints representing the finite domain of each of the input and output variables. Domains are predefined but easily user customizable. Which variable to link with which domain, depends on the declared type of each variable. Each generated Constraint System is represented by an automatically generated Java program. This Java program would instantiate the components of the Constraint Solver in order to find a solution for the goal-constraint associated to it. The general structure of this Constraint System Java program is presented in Fig. 6 and a complete example of a generated CS program can be seen in Appendix A. In detail, the constraint system generation process takes the invariant, precondition and postcondition of the specification and combines them into several sets of constraints. Thus, let s i, j, k be indexes that refers to the DNF-partitions generated for the invariant, precondition and postcondition, respectively. Let s also be l a second index for the invariant. In general, the generated set of constraint systems is defined by Invariant@pre i P recondition j P ostcondition k Invariant l Note that the invariant is used twice, once to describe the constraints over the previous state and the second one to describe the constraints over the after state. Therefore, in the first case the model variables and auxiliary variables appearing in constraints inside the invariant change their names to correspondent. It is also useful to point out that Algorithm 1 defines which postcondition or precondition is going to be used (the original or the mutated one) depending on which step we are executing.

24 The Tool 18 This generation process faces a hard problem that is the explosion of the number of generated constraint systems, especially when working with negated pre- and postconditions. In fact, the number of combinations we can generate is exponential. Let s assume that our specification is a relatively simple one like the Triangle example in Section 5. This specification contains two (2) if statements in its postcondition and produces 9 DNF-partitions. The number of possible combinations when the postcondition is negated to form the predicate P re P ost m P ost is in over 5 millions. However, we examine this combinations and note that most of them will lead to non-feasible solutions, that is, they will be always false. These non-feasible combinations can be deleted before generating the constraint systems. In order to avoid the problems of the explosion of combinations we carry out a simplification process. This process is executed before and after combining the invariant, pre- and postcondition. The simplification consists in looking for a pattern, the conjunction of a constraint with its respective negated constraint. This means, given a set of constraints S, for each constraint c S if ( c) S then the set S can be eliminated and will not be used to generate a constraint system. This simplification process is very powerful especially after we have produced the combined constraint systems where the presence of P ost P ost m predicates lead to a great reduction (from 5 million we can reduce to 14 in the case of the Triangle example). However, we still have to do the combinations before deciding their non-feasibility and this could mean a waste of time and resources. We look at the hypotheses stated for classical mutation analysis for a hint on how to simplify the test case generation process. From the competent programmer hypotheses we can derive that the difference between the original specification and any misunderstood (mutated) specification that could be implemented is going to be very small (maybe a single constraint). Moreover, this difference is likely to be present in one (or few) rather than in all DNF-partitions. To illustrate our point let s see the following example where only postconditions are taken into account. Let s suppose the design D and its faulty design D m with their correspondent postconditions as follows A 1 A 2 A 3 A 4 A 1 A 2 A 3 X 1 B P ost = 1 B 2 A 3 B P ost C 1 C 2 A 3 A m = 1 B 2 A 3 4 C 1 C 2 A 3 X 1 D 1 D 2 D 1 D 2 In these designs, the mutation is produced when the programmer replaces the constraint A 4 by X 1. In this case the mutation spreads over all partitions where A 4 was present. If we examine the postconditions, we see immediately that when the predicate P ost m P ost will be constructed the second partition B 1 B 2 A 3 would always be non-feasible. The reason is because any combination of P ost would contain B 1 B 2 A 3. The same would apply to the fourth partition. Therefore, the only valid combinations would be formed from the partitions where a mutation is present. Let P ost and P ost m be sets of partitions, then these mutated partitions are identified by removing from P ost m all the p partitions such that p P ost. This simplification procedure has a side effect. It is capable of identifying refinement. In fact, after the simplification it can be the case that P ost m will become empty. This case only occurs when D D m. In this case, we could say we have identified a refinement and no case will be capable of distiguish D from D m. Although the last simplification procedure is helpful in reducing the number of combinations for P ost m P ost, the problem of the explosion in P ost still remains (when no refinement has been identified). As each combination in P ost must state that none of the partitions could be evaluated to true then the solution it is not in reducing the number of partitions in P ost but reducing the number of constraints on

25 The Tool 19 each partition. So far, by the previously described simplification procedure we have identified those partitions in P ost m in which a mutated statement (predicate) is present. Let s suppose that a single partition has been identified (we can show that if multiple partitions have been identified the same simplification procedure can be applied by taking one partition each time) then any solution of the predicate representing this partition must be a test case of P ost m. Of course, we are not interested in just a test case of P ost m but a test case capable of distinguishing P ost m from P ost. We know that this test case, if it exists, must be an element of the set of all solutions of P ost m, thus P ost m must evaluate to true and so all the constraints on it. Based on this knowledge we can state a simplification procedure that will reduce the number of possibilities we have for constructing the predicate P ost m P ost from P ost m (that we assume must be true) and a combination of the constraints (negated) in the partitions of P ost. Let s generalize and call c m any constraint in P ost m and c i any constraint in the ith partition of P ost (we will call this partition P ost i ). The simplification procedure now is stated in such way that each partition P ost i will be replaced by P ost i where P ost i = {c i c i P ost m }. In words, each partition P ost i would maintain only those constraints c i which negated version c i belongs to P ost m. If it is the case that none of the constraints in the partition i satisfy these condition then all the constraints must be kept, this is, the simplification will not take place and this partition will not be reduced. This simplification procedure reduces the set of combinations to a unitary set when all partitions have been simplified, otherwise the number of combinations is o j=1 nc j where o is the number of non-simplified partitions and nc j is the number of constraints in the j-th partition that was not simplified. Once the simplification procedures have been executed, the constraint systems that still remains as feasible possibilities are translated into correspondent Java programs. 4.2 The Constraint Handler The constraint handler collects the constraints that arrive from a running program. It puts them into a constraint store, tests their satisfiability, simplifies, and if possible solves them. The constraint handler is composed by a series of simplification and propagation rules over the constraints and basic types The Constraint Theory Below, transformation rules for constraints involving set domains and order relations are presented. These rules are presented in the language of generalized substitutions following the approach in [10]. Domain/Set constraint rules The constraint solver described in this work relies on the fact that variables have established domains, i.e. a variable has a defined value or variable s possible values are defined by a inclusion relation in a set. The equality relation is used to define a value of a variable and one way of defining a set of values is element enumeration. Simplification and propagation rules over equality and inclusion into sets (enumerations) constraints are showed in the following.