Automatically Extracting Mock Object Behavior from Design by Contract TM Specification for Test Data Generation

Transcription

1 Automatically Extracting Mock Object Behavior from Design by Contract TM Specification for Test Data Generation ABSTRACT Stefan J. Galler and Andreas Maller and Franz Wotawa Institute for Software Technology Graz University of Technology Inffeldgasse 16b/II, 8010 Graz, Austria Test data generation is an important task in the process of automated unit test generation. Random and heuristic approaches are well known for test input data generation. Unfortunately, in the presence of complex pre-conditions especially in the case of non-primitive data types those approaches often fail. A promising technique for generating an object that exactly satisfies a given pre-condition is mocking, i.e., replacing the concrete implementation with an implementation only considering the necessary behavior for a specific test case. In this paper we follow this technique and present an approach for automatically deriving the behavior of mock objects from given Design by Contract TM specifications. The generated mock objects behave according to the Design by Contract TM specification of the original class. Furthermore, we make sure that the observed behavior of the mock object satisfies the pre-condition of the method under test. We evaluate the approach using the Java implementations of 20 common Design Patterns and a stack based calculator. Our approach clearly outperforms pure random data generation in terms of line coverage. Categories and Subject Descriptors D.1.5 [PROGRAMMING TECHNIQUES]: Object-oriented Programming; D.2.5 [SOFTWARE ENGINEERING]: Testing and Debugging data generator General Terms Algorithms, Verification Keywords test data generation; automated unit testing; mock object; Design-by-Contract; Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. AST 10 May , Cape Town, South Africa Copyright 2010 ACM /10/05...$ INTRODUCTION Testing is a very important but resource-demanding part of every software development process to ensure software quality. The automation of this process reduces costs while improving software quality at the same time. Our work focuses on automatically generating test input data for JUnit tests based on the provided Design by Contract TM specification. In particular we are interested in generating values that satisfy the given pre-condition for non-primitive type parameters of the method under test. Consider, the add() method in Figure 1 as the method under test. To satisfy the pre-condition a Stack<Double> object has to be constructed that returns at least a value greater than or equal to two when size() is called. Consider the actual java.util.stack implementation: it provides one method to put an element on the stack, one to remove an element from the stack and three methods to observe the current state of the stack without changing it. Therefore, to generate a Stack<Double> object, which satisfies the given pre-condition requires at least two successive method calls to push() at the very end of the object construction method call sequence. This means for a random approach such as JET [3] or a state space exploring technique such as PKorat [14], that it generates a pre-condition satisfying Stack<Double> object only in one out of 25 cases. For both, the first and second method call, only one out of five available methods is suitable. Using more advanced random approaches like the adaptive random testing, may improve this probability slightly. On the other hand, real-world software may have more complex pre-conditions. Therefore, even more runs are required to generate test input data that satisfies the given pre-condition of the method under test. Generating objects that cannot be used costs time and we try to avoid it. Therefore, we suggest to replace all non-primitive parameters of the method under test with Mock Objects. Our approach uses Design by Contract TM specifications to automatically configure the mock object such that the return values of all methods of the mocked object satisfy their post-condition and the pre-condition of the method under test. The approach increases overall line coverage, since it is able to generate objects that satisfy the pre-condition of the method under test immediately. It does not have to construct multiple objects to generate one that fits, such as random has to. The main contributions of this paper are:

2 class Stack<T> size()==@old(size())+1 ) void push(t elem) ) T peek() size()==@old(size()) 1 ) T ) int size () {... class AdditionOperator stack.size()>=2 && otherparameter.isvalid()==true && stack.peek().doublevalue()<10000 ) void add(stack<double> stack, Point otherparameter) { Double valueone = stack.pop(); Double valuetwo = stack.pop(); stack.push(valueone+valuetwo); Figure 1: Running example. Through out the paper we use the add() method of the AdditionOperator class as the method under test. To test the add() method a Stack<Double> object with at least two elements is required. Our approach is able to generate such an object in less time than random requires. The stated specifications is tailored for the running example. 1. a new approach for test input data generation of nonprimitive data types that satisfies a given pre-condition 2. automatic extraction of mock object behavior from Design by Contract TM specification 3. evaluation of the approach on two case studies, a student project and an implementation of the well-known Design Patterns by Gamma et al. [5]. Using the presented approach in automated unit test generation combines the benefits of mocking (currently used only in manual written unit tests) and automated test generation: generates less meaningless tests, thus a higher line coverage can be achieved generated tests separate bugs in the method under test from those in the environment, i.e., the implementation of the test input data, since they get mocked removes system dependencies (e.g., network connections, databases, file system) reduces object construction overhead (method calls to transform the object into a required state, construction of objects that are required by the constructor or any other method call on the parameter object) This paper continues as follows: the concept of Design by Contract TM and Mock Objects are introduced in Section 2. Section 3 describes the approach in detail. Its evaluation on two case studies is presented in Section 4. After the discussion of related work in Section 5 we draw our conclusions in Section PRELIMINARIES 2.1 Design by Contract TM Bertrand Meyer propagated Design by Contract TM [11] with the Eiffel programming language. In addition, to typical interface, class and method declarations, Design by Contract TM allows the programmer to specify the semantics of object oriented software components. Similar to legal contracts between two partners, a programmer can specify what a method requires from its caller and what it guarantees after execution. These contracts are written as conjunction/disjunction of Boolean expressions that are checked at runtime. The three main concepts of Design by Contract TM are [10]: 1. pre-condition: The pre-condition is evaluated before the method is called. It specifies the expected value range of the methods parameters and/or the publicly observable state of the object. The method is only executed when the caller satisfies the pre-condition. 2. post-condition: The post-condition is evaluated directly after the method body is executed and before returning to the caller. It specifies the guaranteed behavior of the method. Whenever the client satisfies the precondition the method has to satisfy the post-condition. 3. class invariants: A class invariant specifies a property that applies for all instances of the class through out their lifetime. All class invariants are checked before and after a method execution. The invariant may be violated within execution time of a method. In general, a pre-condition violation is considered to be a bug in the caller, whereas a post-condition violation is a bug in the methods implementation. The main advantages of Design by Contract TM are an always up-to-date formal specification of the behavior of a method; reduction of development time due to easier bug localization; and more robust software components due to the possibility of automating the verification and/or testing process. Multiple libraries exist for adding Design by Contract TM features to many state of the art programming languages, such as JML [7] or Modern Jass [12] for Java. Eiffel is still the only programming language with built-in support for Design by Contract TM and the term is a trademark of Eiffel Software Inc., California. For our case study we use Modern Jass [12] for Java. Modern Jass supports Design by Contract TM specifications written as Java 5 annotations 2.2 Mocking Mackinnon, Freeman and Craig introduced Mock Objects [8]. Mock Objects replace code with a dummy implementation. The Mock Object configuration defines among other things which parameters may be passed to a method, and which values the method returns when it is called. But the actual implementation is not executed. This makes tests independent from system components, such as databases, network connections or file systems.

3 Mock Objects reduce object construction overhead in unit tests since a Mock Object does not require any constructor parameter, whereas the mocked object may have required other objects passed to the constructor. 3. APPROACH The idea of the presented approach is to replace all nonprimitive parameters by Mock Objects. The Mock Objects are configured to behave according to the Design by Contract TM specification of the mocked type. The return value of methods called on the mock satisfy the post-condition of the called method. It might be the case that the real implementation returns a different value, but the client of that particular object has to deal with any return value that satisfy its post-condition. The presented approach generates iteratively an object for each non-primitive parameter type of the method under test. It takes the type of the parameter and its specification as input and executes the following steps: 1. the mock object for the given type is instantiated 2. the required behavior of the mock object is derived from the Design by Contract TM specification of the given type 3. the Mock Object is configured through the API of the used mock library to behave accordingly 4. the generated Mock Object is returned and used as test input. The required behavior of the mock object is determined based on the Design by Contract TM specification of the corresponding type and the given pre-condition of the method under test. The Design by Contract TM specification of the type defines the principal behavior of the mock object. The pre-condition strengthens this specification further, to guarantee that the mock object reacts as if it is in a state that satisfies the pre-condition. Therefore, we configure all methods of the Mock Object, such that they return a value according to the conjunction of the pre-condition of the method under test and the postcondition of the mocked method. The detailed description of this configuration process is presented in Subsection 3.2. The presented approach leads to higher code coverage, in terms of covered lines, since it generates test input data that satisfy the pre-condition of more methods under test. 3.1 Supported Specification Syntax Figure 2 defines the syntax of the pre-condition currently supported by the implementation. The pre-condition is a conjunction/disjunction of Boolean expressions. Each of the Boolean expressions may consist of a left-hand side, a comparison operator and a right-hand side. The latter two are optional in case that the left-hand side evaluates to a boolean value. The left hand-side is a method call or a single statement of multiple successive method calls, e.g., stack.pop().tostring(). The right-hand side defines the comparison value for the return value of the last method call, in the example the call to tostring(). All other method calls on the left-hand side are considered to return a valid instance, i.e., not null. The currently implemented syntax does not support a method call on the right-hand side. The pre-condition of PRECOND := BEXPR BEXPR && PRECOND BEXPR PRECOND BEXPR := var! = null var MCALL var MCALL OP value MCALL :=.methname(arglist).methname(arglist) MCALL ARGLIST := ɛ var var, ARGLIST OP := ==! = < <= > >= var := [a za Z][a za Z0 0] Figure 2: Grammar of the supported pre-condition syntax. The pre-condition (PRECOND) is build up by Boolean expressions (BEXPR). Each of them defining the return value of a method call (MCALL). The comparison of two method calls is not supported by the current implementation. our running example in Figure 1 is matched by the specified syntax. 3.2 From Design by Contract TM to mock behavior Automatically extracting Mock Object behavior from Design by Contract TM specification for test data generation requires the class type to generate and the Design by Contract TM specification. The algorithm is shown in Figure 3. Each step is explained in detail in the following paragraphs. build a map. First, a map containing all non-void methods of the mock object is created. Second, to each method the corresponding post-condition is mapped. Figure 4 shows a summary of the map for the java.util.stack<double> class of the running example (Figure 1). derive requested state. To derive the requested state for the mock object, the pre-condition of the method under test (which includes constraints on the mock object) is transformed, through rewriting of the specification by following a number of rules. Therefore, for each method of the mocked object a subcondition of the pre-condition is extracted, such that the sub-condition constraints only the according method. A sub-condition in that sense, is a conjunction/disjunction of Boolean expressions, with equal left-hand sides. The conditions m 1 and m 2 in Figure 5 are two Boolean expressions matching the BEXPR rule from Figure 2. Both impose constraints on the same method m - one of those present in the map. Condition o refers to any other boolean expression. The given rules are applied repeatedly until no rule is applicable anymore. No rule is applicable anymore when the pre-condition does not include any Boolean expression that references the currently processed method. Basically the rules are applied to the overall Boolean expression iteratively for each method m in the map. The result is a Boolean expression where all left-hand sides equal m, i.e., imposing constraints only on method m. Any Boolean

4 1. build a map: method post-condition (postcond) of method 2. derive requested state of the mock object from the pre-condition of the method under test (precond) (a) split pre-condition in conjunction of subconditions (subcond) where each Boolean expression in one sub-condition references always the same method: subcond 1 &&... && subcond n = precond (b) update map to: method subcond && postcond 3. generate return value that satisfies the specification 4. generate Java code Figure 3: From Design by Contract TM to mock behavior. An outline of the algorithm used to generate Java source code that configures a mock object, which satisfies a given pre-condition, and can be used as test input data. java.util.stack<double> method constraint pop() size()==@old(size())-1 =null Figure 4: Map content after first step of transformation process. The map contains for each non-void method an entry: method post-condition. m 1 && o && m 2 m 1 && m 2 (1) m 1 o && m 2 m 1 && m 2 (2) m 1 && o m 2 m 1 (3) m 1 o m 2 m 1 (4) m 1 m 2 m 1 (5) Figure 5: Extraction rules. Applying those rules iteratively as long as no rule is applicable anymore results in a set of sub-conditions. Each sub-condition imposes constraints on one and the same method. m 1 and m 2 are two Boolean expressions constraining the same method, o constraints a different method. java.util.stack<double> method constraint 2 pop() size()==@old(size())-1 =null Figure 6: Map content after the second step of the transformation process. expression o, which left-hand side differ from m 1 can be removed. Therefore, rules 1 and 2 only keep m1 and m2. Rules 3, 4 and 5 are similar to 1 and 2, but in those cases even m2 can be removed since the overall Boolean expression is satisfied as soon as m1 is satisfied. Following those rules guarantees, that if all sub-conditions are satisfied the original pre-condition is satisfied too. Each generated sub-condition is then conjuncted with the already present specification of the corresponding method in the map. The resulting map of this step in the transformation process is given in Figure 6. generate return values. For each method in the map return values are generated, that satisfy the specification present in the map. This is done by calling the value generator component of the test data generation tool and passing the specification as argument. Depending on the required return type, different value generators are either called with the given specification or called until the generated value satisfies the given sub-condition (or a maximum number of trials is reached). In our case a value for primitive data types is generated randomly and checked against the specification. If it does not hold, a new value is generated randomly. For non-primitive data types the value generator component constructs a mock object recursively (up to a manually set depth). The provided specification is the sub-condition eliminated by all Boolean expression that do not constraint any return value of method calls on the requested type itself. For this elimination the same algorithm as described in Figure 5 is applied, this time with m 1 and m 2 referencing the method called on the object under generation. In addition, a new temporary variable has to be introduced, referencing the required return value. In the specification the method call on the object under generation in m 1 and m 2 has to be replaced by the temporary variable. Given the pre-condition of the add() method in Figure 1 and generating the return value for method call peek(), the provided specification to the value generator component is stack.peek().doublevalue()< before introducing the temporary variable. Finally, the passed specification is tmp1.doublevalue()< where tmp1 is of type Double. Figure 7 shows the map updated with exemplary return values. generate Java code. The last step of the transformation process generates the actual Java code. After instantiation of the Mock Object,

5 java.util.stack<double> method return value size() int: 5 pop() Double: 2.4 peek() Double: 2.4 Figure 7: Map content after the third step of the transformation process. Stack stack = EasyMock.createMock(Stack.class); int v0 = stack.size (); EasyMock.expectLastCall().andReturn(5).anyTimes(); stack.push(easymock.anydouble()); EasyMock.expectLastCall().atLeastOnce(); Double v1 = stack.pop(); EasyMock.expectLastCall().andReturn(2.4).anyTimes();... EasyMock.replay(stack);t Figure 8: Resulting Java code. Automatically generated mock behavior configuration for the running example from Figure 1. for each method of the mocked object the corresponding EasyMock API call is generated. In case the method has parameters for each parameter a parameter matcher is instantiated. A parameter matcher - as they are called in EasyMock- specifies the expected value range of the parameter. Currently, no constraints on parameters are considered since those are checked by the Design by Contract TM tool anyway. Thus, for a parameter of type Double the Easy- Mock.anyDouble() matcher is generated. For all other types similar API calls are available. Each method is expected to be called any times. If the method returns an object, the andreturn() API call is added as well. The generated code for the running example is presented in Figure 8. Calling method size() returns always the value 5. push() may be called with any valid Double but does not return anything. pop() returns a Double instance of value Limitations and Future Work This section focuses on limitations of the approach not on limitations inherited by the use of the EasyMock library (e.g., no final classes can be mocked) or other implementation based limitations. Therefore, two properties of the presented approach have to be discussed in more detail: 1. static state of the generated mock object 2. return value may not depend on parameter values static state of the generated mock object. The presented approach does not work for implementations that expect state changes of the mock object nor Design by Contract TM specifications that specify the state of the mocked object after method execution. Since the generated mock object for the Stack in Figure 9 always returns the same value for each call to method size() - one that satisfies the pre-condition of the method under test - the given implementation leads to an endless stack.size()>1 ) int sumup(stack stack) { int tmp = 0; while(stack.size()>0) { tmp += stack.pop(); return tmp; Figure 9: Example implementation of a method that expects a state change of the mock object. Currently the generated test results in an endless loop. public class Math x>0 && y ) int multiply(int x, int y) { return x y; Figure 10: Example specification of a weak postcondition for a method that returns the result of the addition of two integer values passed as parameters. This is caused by the static behavior configuration of the mock object. The Mock Object configuration does not consider any internal state changes caused by method calls. Our future work will focus on extending the Mock Object by dynamic state information. The initial state of themock Object is generated by the presented approach. But afterwards the Mock Object records all method calls. Return values of method calls on the Mock Object are not configured statically but take into account the history of method calls. return value may not depend on parameter values. Consider a method under test that requires the Math object in Figure 10 as test input. The presented approach uses the specified post-conditions to generate a return value that satisfies it, at test generation time. The presented approach does not work, whenever the return value of the method depends on parameters to the method. The value of the parameter may not be defined before test execution time, therefore the test data generation algorithm may not generate a return value for the method at test generation time. Future work will focus on an approach to generate the return value at test execution time. By introducing a return value generator component that is called from the mocked method to return a value that satisfy the post-condition at runtime, similar to the symbolic mock object approach from Tillmann and Schulte [16]. 4. CASE STUDY AND RESULTS Using test data that is automatically generated by extracting Mock Object behavior from Design by Contract TM specification improves line coverage on our case studies by at least 20%. Both, test generation and test execution time, increase only slightly. 4.1 Case Studies We evaluate our approach on two different case studies.

6 Table 1: Case studies source code measurements. This table shows the size of the case studies in terms of number of classes, functions and non commenting source statements (NCSS). attribute StackCalc design pattern classes methods testable methods NCSS Table 2: Amount of specifications. The table lists the amount of specifications written for each of the case studies. attribute StackCalc design sum Both case studies were implemented without knowledge that they will be annotated with Design by Contract TM specification and used as case studies later. Therefore, we can guarantee that both case studies are not tailored to this evaluation. 1. StackCalc : This case study was developed by two students as a course work. It implements a stack based calculator with a command line input/output interface. Multiple mathematical operators are implemented that operate on the Stack. 2. Design Pattern : This case study is available online from the homepage of the Design Patterns in Java [9] book. The implementation consists of all Design Patterns introduced by Gamma et al. [5]. We use this case study because well-designed software heavily builds upon Design Patterns. We believe that the approach is useful in more general software components as well when we can show an improvement in the testability of Design Patterns. The Design by Contract TM specifications were added to the case studies by the authors without changing any implementation or design details. To get an impression of the case studies in terms of size and amount of specifications, Table 1 shows some common software metrics and Table 2 lists the amount of specifications for both case studies. All methods annotated are not tested (e.g., getter methods in our case studies). 4.2 Evaluation Process We compare the presented approach to random data generation. Therefore, we use multiple servers with the same hardware and software configuration (AMD Athlon 64bit Dual Core; each 2,2GHz; 2GB RAM). Both, our approach and the random approach, is executed with 25 different parameter value combinations. Each of these configurations is executed 30 times for each approach. The two alternating parameters are: 1. mutating probability: This parameter defines the probability that after executing a method call on a generated object, another method is chosen to change the objects state (mutate the object state). This parameter is only used by the random approach and may be one of the following values: 0.0, 0.2, 0.4, 0.6, attempts: This parameter defines how many tests are generated for each method of the case study. We used the following values: 1, 2, 3,4, 5. We claim that using automatic generation of mock behavior from Design by Contract TM specification increases line and branch coverage compared to the random approach. This is due to the ability of the presented approach to generate input data that satisfies the given pre-condition for more methods. Using more sophisticated methods usually takes more resources. Since the approach is only applicable to industrial projects when the increase of test generation time and execution time is worth it, we also evaluate the approaches with respect to test generation time and test execution time. 4.3 Detailed Results The presented approach leads to a significant increase in line and branch coverage for both case studies. The improvement by 50% for the StackCalc case study (Table 3) is due to the design structure. About half of the methods under test operate on the Stack (e.g., sum up, multiply, subtract or divide the last two elements on the stack) and therefore have very similar pre-conditions. They require at least two elements on the stack. The random approach was not able to generate a Stack object with two elements in hardly any test run. Whereas, the presented approach always succeeded in generating a Stack object that satisfies the given pre-condition. The enormous increase in branch coverage can be tracked down to the same issue. All methods operating on the stack use conditional statements. As soon as the test input data satisfies the pre-condition of the method under test, at least one conditional statement is executed. The Design Pattern case study, as the more general one in terms of software design, still shows an increase in line coverage by 20% as presented in Table 4. Since this case study includes all design patterns this increase cannot be explained by a special software design anymore. One may have suggested that the coverage decreases due to the fact, that mock objects replace real implementation. This is not the case, because each class is tested anyway, but separately. Only when the class is used as a parameter for a method the implementation is mocked. As expected the presented approach requires slightly more time for both, test generation and test execution. Tables 5 and 6 show in column one of each approach the required time to generate all tests divided by the amount of methods for which the approach tried to generate a test. Column two shows the required time divided by the amount of actually generated tests. This is caused by heavy use of Java reflection and the overhead of instantiating a Mock Object. But

7 Table 3: StackCalc: achieved coverage. Line coverage increased by 50% for the stackcalc case study. This is mainly due to the fact that the random approach hardly never constructed a Stack object with two elements. No methods that operate on the Stack (e.g., multiply, add, substract) could be tested. RANDOM MOCK attempts line branch line branch 1 11% 1% 17% 8% 2 11% 1% 17% 9% 3 12% 1% 18% 10% 4 14% 2% 21% 12% 5 2% 2% 17% 10% avg. 12% 1% 18% 10% Table 4: Design Pattern: achieved coverage. Line coverage increased by 20% only. Since the design pattern are widely used and very general, we claim that our approach is helpful in testing an arbitary program. RANDOM MOCK attempts line branch line branch 1 33% 13% 41% 20% 2 35% 15% 42% 22% 3 35% 16% 41% 22% 4 39% 18% 45% 24% 5 35% 17% 40% 23% avg. 35% 16% 42% 22% Table 5: Stackcalc: required time per test. RANDOM MOCK att. gen./ gen./ exec. gen./ gen/ exec. sum succ. sum succ avg with increasing size of the case study the premium decreases relatively to the overall generation time. In addition, to the significant increase of line coverage of the system under test the presented approach inherits all advantages from using Mock Objects. The Design Pattern case study uses Java File objects. The random approach messed up the file system by generating thousands of randomly named files. They had to be deleted manually after the test generation process. The presented approach did not create a single file, since the File object was mocked, the createfile() method of it was actually never executed. It is to mention that one class of the StackCalc case study could not be tested due to the limitation shown in Figure RELATED WORK This section should give an overview of recent work on test data generation to show where current research efforts focus on. Both, Tillmann et al. [15] and Saff et al. [13], use Mock Table 6: Design Pattern: Required Time on average per test. RANDOM MOCK att. gen./ sum gen./ succ. exec. gen./sumgen/succ. exec avg Objects to generate test data input. They require in addition to the program under test, a set of system tests respectively unit tests. The given test set is executed and the interaction of the method under test with its parameters is recorded. This information is used later in two different scenarios: The next time the test set is executed, the parameter objects are replaced by Mock Objects to reduce test execution time. Automatically generated unit tests use Mock Objects as test input which are configured to exactly behave as recorded earlier. Both approaches do not guarantee that the behavior of the Mock Object corresponds to the intended one - since no formal specification is present. In addition, they may find errors, that are caused only by a change in the method call sequence, even tough the semantics did not change at all. Related approaches that use different technologies for test data generation can be categorized into random based approaches, state space exploring based approaches and approaches based on evolutionary methods. Cheon [3] and Ciupa et al. [4] work on random based approaches. The key idea of Cheon is to construct objects incrementally. First choosing randomly a constructor for instantiating the requested object. Afterwards, transforming the object state by calling a random number of methods on it. This random approach is used for our case studies. Ciupa extended adaptive random testing [2] from Chen for object types. ARTOO [4] works with two sets of test data: the candidate set and the applied data set. The candidate set is filled with randomly generated values. Values used for test generation are moved from the candidate set to the applied data set. The value from the candidate set which has the highest distance value to the already applied values is used the next time. Therefore, Ciupa introduced the notion of an object-distance, which is calculated recursively based on its members and the distance in the inheritance tree of the two compared objects. Both approaches depend on randomly constructed class instances, which they store in a pool for later usage. ARTOO uses a more sophisticated algorithm to choose which object is used next from the pool. But in any case, the sequence of method calls to transform the object state is chosen randomly. Therefore, it is very unlikely that one of the objects satisfy a given pre-condition. Kurshid et al. [14] and Howe et al. [1] are working on state space exploring techniques. The former approach explores the whole state space - up to a given bound. This approach is very expensive in terms of object constructions to find one that satisfies the given pre-condition. The latter approach

8 uses AI planner to directly find a method sequence to a state that satisfies the given pre-condition. This approach is very promising, but still requires human interaction - in contrast to our approach, which is fully automated. Mark Harman [6] works on search-based algorithm for test data generation. It is based on the theory of evolution. The problem of constructing non-primitive objects is modeled as search problem, where the fitness function determines how good the data is with respect to achieving a given problem. Techniques such as crossing-over and mutation help in generating new sets of data based on an initial (random) data set. 6. CONCLUSION In this paper we presented an approach for automated generation of mock behavior from Design by Contract TM specifications. Using automatically generated Mock Objects in unit testing allows test generation tools to easily generate data that satisfies a given pre-condition. The evaluation of the presented approach for two case studies shows an increase above 20% in terms of line coverage. At the same time the test generation time and the test execution time increased only slightly. Especially for the larger case study the premium of using automated mock behavior generation was very low relative to the overall execution time. In addition, all advantages of using Mock Objects for testing are inherited: generated tests report only errors in the method under test, not in the implementation of test input data called by the method under test; reducing generation overhead for instantiating complex objects (e.g., objects that require multiple other objects to be instantiated before); being independent from system objects such as databases, files and network connections. Our future research will focus on automated generation of a dynamic mock behavior. This will eliminate the two limitations of the approach: no changes of the mock objects state are possible at test execution time; dealing with parameter references in post-conditions. The former can be removed by introducing a virtual state of a mock object. The mock objects initial state is generated as presented in this paper but afterwards the interaction with the mock object is recorded and stored. State discovering method calls (usually getter methods) may then return different values depending on the pre-sequence of method calls. Acknowledgment The research herein is partially conducted within the competence network Softnet Austria ( and funded by the Austrian Federal Ministry of Economics (bm:wa), the province of Styria, the Steirische Wirtschaftsförderungsgesellschaft mbh. (SFG), and the city of Vienna in terms of the center for innovation and technology (ZIT). Department of Computer Science The University of Texas at El Paso, 500 West University Avenue, El Paso, Texas, USA, [4] I. Ciupa, A. Leitner, M. Oriol, and B. Meyer. Artoo: Adaptive random testing for object-oriented software. In International Conference on Software Engineering 2008, May [5] E. Gamma, R. Helm, R. Johnson, and J. Vlissides. Design Patterns. Addison-Wesley Reading, MA, [6] M. Harman, F. Islam, T. Xie, and S. Wappler. Automated test data generation for aspect-oriented programs. In AOSD 09: Proceedings of the 8th ACM international conference on Aspect-oriented software development, pages , New York, NY, USA, ACM. [7] G. T. Leavens, A. L. Baker, and C. Ruby. Preliminary design of jml: a behavioral interface specification language for java. SIGSOFT Softw. Eng. Notes, 31(3):1 38, May [8] T. Mackinnon, S. Freeman, and P. Craig. Endo-testing: unit testing with mock objects. pages , [9] S. J. Metsker and W. C. Wake. Design Patterns in Java [10] B. Meyer. Applying design by contract. Computer, 25(10):40 51, October [11] B. Meyer. Object Oriented Software Construction. Prentice Hall PTR, Englewood Cliffs, NJ, first edition, [12] J. Rieken. Design by contract for java - revised. Master s thesis, Department für Informatik, Universität Oldenburg, April [13] D. Saff, S. Artzi, J. H. Perkins, and M. D. Ernst. Automatic test factoring for java. In ASE 05: Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering, pages , New York, NY, USA, ACM. [14] J. H. Siddiqui and S. Khurshid. PKorat: Parallel Generation of Structurally Complex Test Inputs. In Software Testing Verification and Validation, ICST 09. International Conference on, pages , [15] N. Tillmann and W. Schulte. Mock-object generation with behavior. In ASE 06: Proceedings of the 21st IEEE International Conference on Automated Software Engineering (ASE 06), pages , Washington, DC, USA, IEEE Computer Society. [16] N. Tillmann and W. Schulte. Unit tests reloaded: parameterized unit testing with symbolic execution. In Software, IEEE, volume 23, pages 38 47, Los Alamitos, CA, USA, July IEEE Computer Society Press. 7. REFERENCES [1] A. K. Amschler Andrews, C. Zhu, M. Scheetz, E. Dahlman, and A. E. Howe. AI Planner Assisted Test Generation. Software Quality Journal, 10(3): , November [2] T. Y. Chen, H. Leung, and I. K. Mak. Adaptive random testing. pages [3] Y. Cheon. Automated random testing to detect specification-code inconsistencies. Technical report,