Story Driven Testing - SDT

Transcription

1 Story Driven Testing - SDT Leif Geiger Software Engineering, University of Kassel Wilhelmshöher Allee Kassel, Germany leif.geiger@uni-kassel.de Albert Zündorf Software Engineering, University of Kassel Wilhelmshöher Allee Kassel, Germany zuendorf@uni-kassel.de ABSTRACT In the last years, SCESM community has studied a number of synthesis approaches that turn scenario descriptions into some kind of state machine. In our story driven modeling approach, the statechart synthesis is done manually. Many other approaches rely on human interaction, too. Frequently, the resulting state machines are just the starting point for further system development. The manual steps and the human interaction and the subsequent development steps are subject to the introduction of errors. Thus, it is not guaranteed that the final implementation still covers the initial scenarios. Therefore, this paper proposes the exploitation of scenarios for the derivation of automatic tests. These tests may be used to force the implementation to implement at least the behavior outlined in the requirements scenarios. In addition, this approach raises the value of formal scenarios for requirements elicitation and analysis since such scenarios are turned into automatic tests that may be used to drive iterative development processes according to test-first principles. Categories and Subject Descriptors D.2 [Software Engineering]: [Processes, Tools] General Terms Scenarios, Test-First Principle, Code Generation 1. INTRODUCTION This paper extends the ideas of [5]. For the new contribution of this paper cf. the conclusions. Our approach uses sequences of UML scenario diagrams for modeling so-called use case stories. We use such stories to drive our software development process. Especially, stories are used for test generation. Therefore we call our approach Story Driven Testing (SDT). SDT is included into the following iterative, use case driven software development process: A use case is identified. The use case is described by a textual scenario description. The textual scenario is turned into a UML scenario description. For each step of the textual scenario, a UML collaboration diagram is provided. This collaboration diagram models the object structure employed to realize this step and how this object structure evolves within this step. Alternatively, intermediate steps may be modeled by sequence diagrams. From the collaboration diagrams used above, it is straight forward to derive a conceptual class diagram, cf. [3]. From the UML scenario description, we automatically generate the implementation of a JUnit test. This test executes use case actor activities in a given example situation and checks for proper system response. In our approach, we use so-called story diagrams for test specification. Then we employ the Fuja code generators to turn these story diagrams into usual Java code, cf. [9]. System reactions are to be implemented within the methods called by the actor. Thus the Java developer has now the task to implement the methods employed in the current scenario or he/she may have to adapt already existing methods in order to cover the new scenario, too. Our process covers simple scenario-sed requirements elicitation and state-of-the-art object-oriented analysis sed on UML scenario diagrams. Usually, such analysis is only loosely coupled to the design and implementation activities. In our approach, the results of the analysis activities are directly turned into JUnit tests driving the design and implementation. From the Java developer s point of view, now analysis diagrams have a real value for his/her work. The developers will soon use analysis scenarios as an easy means for writing tests. From the project management s point of view, the generated JUnit tests keep analysis documentation and the actual implementation consistent. 2. SCENARIOS TO TESTS As running example, we use one use case from the Paderborn shuttle system case study, cf. Figure 1.

2 Invoice // The shuttle agent has an unpaid invoice for 100$. Its nk account has a lance of 42$. :ShuttleAgent :BankAgent :Invoice costs done:order :Account lance == 42 // The shuttle agent informs the nk agent about the invoice. 1: pay () lance == 42 Figure 1: Paderborn shuttle system scenario // The nk agent adds the invoice to the shuttle agents account. It checks wether the account is valid and if there is already a reminder. Because there is no reminder, the nk agent is waiting. For our approach, the information provided in the scenario of Figure 1 is insufficient. In order to generate code, we need to know, the relationship between the shuttle agent and the nking agent and which object structures are employed to represent orders, s, invoices and reminders. This information is usually added in the analysis phase by turning scenarios into so-called story boards, cf. Figure 2. The first activity of our scenario models the start situation as an UML object diagram. During test derivation, this model of the start situation is turned into a JUnit setup method, cf. Figure 3. Such a setup method models the creation of an object structure corresponding to the start situation of the considered UML scenario. From such method specifications the code generators of the Fuja environment generate usual Java source code, cf. [9]. For setup methods this is easy, we just generate new statements for the objects that are to be created and calls to setter methods creating the links and initializing the attributes. Note, that for debugging purposes (cf. chapter 4) and to split the testing into different methods, the current test objects needs references to all created objects. This is modeled using several object X links from object this, cf. figure 3. Each activity of the UML scenario is marked either by a stickman indicating that it is initiated by some use case actor or by two toothwheels indicating that it is part of the system reaction. In the test generation, the operations belonging to the use case actor are turned into operations of the JUnit test method. The second activity of Figure 2 starts the execution of the scenario by invoking pay() on the nking agent. As indicated by the stickman, this activity is executed by the use case actor. Accordingly, this activity becomes the first step of a testinvoice method generated for our JUnit test, cf. Figure 4. Fuja s code generator interprets such method activities as an object pattern that has to be matched again the runtime object structure. This is achieved by generating appropriate search loops and checks. 1: check() 2: checkreminder() time := // The shuttle agent sends a reminder for the invoice. // The reminder is added to the invoice and it is checked. 1: checkreminder() lance == 142 :Invoice 1: remind () reminders // The shuttle agents account has lance of 142$. lance == 42 time := lance == 42 lance == 42 r1 :Reminder time := time >.gettime() + 7 time <.gettime() + 14 // The reminder was in time, so the invoice is processed and the shuttle agent is informed. 1: transfercompleted() lance == 42 lance := 142 Figure 2: Story board for the invoice scenario

3 In the activities of Figure 4 all objects are still known from the setup method. Thus, the generated code just checks for attribute conditions and for the existence of the depicted links. If one of these conditions is violated, the generated code raises a JUnit exception and the JUnit test fails, cf. [5]. If the object pattern matches, the depicted method calls are executed and the test proceeds with the next activity. Story board activities marked by a toothwheels icon model expected responses of system methods to actions within the considered example situation. For example activity 3 of Figure 2 models that in the depicted situation method pay() is expected to create a link and a link and to set the time attribute of the object to In addition, the check() operation and the checkreminder() operation are to be called. Note, there may be multiple scenarios invoking the pay() method in different situations. Now the system developer has to implement the pay() method such that all requirements are met and such that it works for the general case. InvoiceTest::testInvoice (): Void // The shuttle agent informs the nk agent about the invoice. // The shuttle agent sends a reminder for the invoice. 1: remind () 1: pay () lance == 42 time := lance == 42 InvoiceTest::setUp (): Void // The shuttle agents account has lance of 142$. // The shuttle agent has an unpaid invoice for 100$. Its nk account has a lance of 42$. :ShuttleAgent :Invoice total := 100 done:order costs object_ object_ object_done this object_ object_ :BankAgent time := :Account lance := 42 :Invoice lance == 142 Figure 4: Actor activities and intermediate checks Figure 3: Setting up the mple situation After calling a method, we want to be able to check the resulting object structure for certain mandatory effects. Such checks may be modeled in Fuja s story diagram notation. As an example, activity 4 of Figure 2 models the object structure expected as result of the pay() call issued in activity 2. Similarly, activity 7 of Figure 2 models the object structure expected as result of the remind() call issued in activity 4. Such checks become part of the generated test method, cf. activities 2 and 3 of Figure 4, respectively. We have adapted Fuja s code generation such that if it is not possible to match the depicted object structure a specific JUnit test exception is raised. Thus, the JUnit test would fail and the cause of failure would be reported, cf. Figure 5 where the last scenario activity has not been matched. Figure 5: The result situation is not met If the modeled (intermediate) situation is reached and the corresponding check is passed, we may model some new actor steps. The actor may change the current situation e.g. by creating new objects and links. For example, activity 4 of Figure 2 changes the time attribute of the nk agent

4 object to In additon, an actor step may invoke new system methods as e.g. the remind() call shown in activity 4 of Figure 2. This is turned into corresponding actions in the test method, cf. activity 2 of Figure EXPLOITING SYSTEM REACTIONS So far we have turned actor activities in corresponding test activities. We did not yet exploit the story board activities that model system reactions. Actually, these steps have a somewhat vague semantics. One may consider them as some kind of comment outlining conceptual steps that guide the implementation of the considered system operation. In this case one would probly not require, that the actual implementation performs every single object structure modification exactly as it is depicted in the scenario. It might be okay, if e.g. some attribute changes are omitted and some helper operations are not called or have different names. However, if a close match between scenarios and system implementation is desired, our test generator turns system reaction activities in so-called assertstep operations, cf. e.g. Figure 6. These operations are again checks, whether the actual runtime object structure matches the object structure modeled in the scenario. One difference is, that we check for the situation that is achieved after performing the depicted object structure changes. For example, method assertstep5 checks whether system operation remind() called in activity 4 of Figure 2 has successfully created a reminder object attached to the object. InvoiceTest::assertStep5 (): Void // The reminder is added to the invoice and it is checked. reminders lance == 42 r1 :Reminder time == time >.gettime() + 7 time <.gettime() + 14 the reminder object only temporarily. In activity 6 of Figure 2 the is payed and afterwards, the object is discarded. Thus, after termination of method remind(), the check assertstep5 would fail due to the missing object. Accordingly, the assertstep checks have to be called during the execution of system reactions. Fortunately, the system reaction activities show explicitly, which actions are expected to happen during system reaction. In our example, we expect that a reminder object is created and attached to the already existing object. At this point, we assume that the employed objects provide some listener concept. In Fuja this is easily achieved through some flag for the code generation [8]. In other systems, this may easily be achieved e.g. using aspect weaving. Provided with a listener concept, we extend our setup method by an implicit subscription of the this object as listener for attribute and link changes. During test execution, if a change event occurs, we execute the assertstep operations and protocol their success. In addition, activity 5 of Figure 2 shows a call of operation check() on the object. If we consider thus calls mandatory, we ask Fuja or some aspect weaver to instrument the corresponding system method with a call notification. These notifications are protocolled, too. Provided with a protocol of successful and failed assertstep checks and of internal method calls, we are able to extend the JUnit test result by the following information: If the test is passed, and all assertstep checks have been passed in the correct order and if the internal operations have been called at appropriate points in time, then everything is fine. If some of the checks have never been passed or some internal operation is never called, then the system implementation differs from the scenario descriptions. This may either be turned into a warning or into a JUnit test failure. If the test fails, the protocol of the assertstep checks may help to localize the fault. If the first two assertstep checks are passed but the third fails, then the scenario seems to be implemented up to that step. Using an agile test driven software development process, developers may work along the scenario and implement the general behavior for one step after the other. After completing one step, the corresponding assertstep check should be passed. Thus, the developer gets a step related feedck on his progress in system implementation. Figure 6: Asserting the creation of a reminder The problem with the assertstep operations is, when should they be called. The testscenario operation invokes some system method. When this method returns, it is to late to test for some intermediate state that may exist only at some point in time during the execution of the system method. For example, in our scenario, method remind() creates 4. DEBUGGING AIDS In case of a JUnit test failure, we already exploit the scenario description of the system steps in order to identify, which scenario steps have been executed successfully and where the implemented behavior deviates from the given scenario. In addition to this, we provide runtime object structure visualization with the help of our DOBS application, cf. Figure 7.

5 The SCENT approach [7] derives statecharts from natural language scenarios. Test case generation is then done using path traverl within these statecharts. Again this approach lacks of support for complex object structures. In our process, a scenario is in most cases just a sequence of activities. When deriving the behavioral specifications all these sequences are merged together into a system statechart manually. So, path analysis techniques might be useful in our process to identify paths in the implementation not yet covered by scenarios to achieve completeness. This is future work. Figure 7: Visualizing the runtime object structure Figure 7 shows the actual runtime object structure when the JUnit test has failed in the third activity of Figure 4 resulting in the error shown in Figure 5. The error mesge in Figure 5 tells that the lance of account is not equal to 142. DOBS shows that the lance of account is actually still 42. Thus, the implementation has executed the scenario almost complete but failed to change the lance. If this does not suffice to identify the problem, the developer may restart the JUnit test in debugging mode. This enables stepwise execution of the scenario while at each step the changes to the runtime object structure are directly visualized in DOBS. Due to our experience, this combination of UML scenario diagrams and DOBS visualization of runtime data is a perfect debugging aid. 5. COVERAGE Fuja has build-in support for the JCoverage tool to combine the JUnit tests with a code coverage analysis. This coverage analysis may reveal certain parts of the implementation that are not required for the story board scenarios. This might be a hint on additional cases that have already been considered in the implementation but that are not yet documented by corresponding story board scenarios. Such a mechanism may provide us with some notion of completeness for story board documentations. Note, that coverage analysis is done on source code level. Coverage analysis on model level is a part of our future work. 6. RELATED WORK There are several other approaches which deal with scenariosed testing. The Rational Quality Architect [1] for example uses sequence diagrams to define scenarios. From these sequence diagrams they generate test cases and test drivers. At test runtime, the signal flow is traced and protocolled using a sequence diagram which is then compared to the scenario sequence diagram. The use of sequence diagrams make sense when dealing with complex signal flow, like e.g. in protocols. We think that for modeling applications with complex object structures, story boards are better suited. Another benefit of story boards is, that the developer may model complex post conditions using an object diagram, see e.g. the last activity of figure 2. This is hardly done with sequence diagrams. More likely, one would use OCL expressions for this purpose. Here, we claim, that our graphical notation is easier to read than OCL post conditions. The TOTEM approach [2] uses again sequence diagrams for specification of scenarios. The test generation also takes inter-scenario relationships modeled in use case diagrams into account. We model inter-scenario relationships implicitly in the story boards by calling the methods of other scenarios. Unfortunately, this relationships are not yet considered at test generation. In [6] it is suggested to use collaboration diagrams for automatic test generation, too. Their approach uses one collaboration diagram to model the execution of an operation. From such a collaboration diagram a Mesge Sequence Path is generated. This Mesge Sequence Path can then be compared with the trace of the corresponding test execution. 7. CONCLUSIONS This paper extends our first approach of [5] by more flexible scenario descriptions and by the more powerful exploitation of system activities through assertstep methods that are called by change listeners monitoring changes to the initial object structure. These features are currently implemented. In addition, in [4] we have adapted the Fuja code generation for JUnit tests. Within a JUnit test the code throws JUnit test exceptions instead of usual search exceptions. This gives the test methods a slightly different semantics: usually, if a pattern matching fails, the generated code proceeds with subsequent activities. The failure may only be exploited by special transition guards. In JUnit tests, a failure in matching a pattern aborts the JUnit test with a failure. Our process is tool supported by the XProM plug-in of the Fuja environment [5, 9]. Fuja s XProM plug-in provides an html sed editor for analysis documents. This editor enables the integrated editing of use case diagrams, text scenarios and UML scenario diagrams. In addition, the XProM plug-in contains our generator for JUnit tests. We already use the XProM plug-in with great success since three years for educational purposes and for one year in an industrial project. Due to our experiences, the scenarios get high attention by the students due to their value for the subsequent development steps. It has turned out, that our detailed scenarios help the students a lot in designing their system. In addition, the scenarios prepare the field for a test driven development process. Combined with a versioning system for UML models that supports optimistic locking, this enables very good team collaboration on the implementation of scenarios. The debugging aids provided by DOBS combined with the explicit object structures in the scenarios has proven to be extremely valuable.

6 8. REFERENCES [1] Rational software corporation: Rational quality architect realtime edition user s guide [2] L. Briand and Y. Labiche. A uml-sed approach to system testing. In 4th International Conference on the Unified Modeling Language (UML), pages , Toronto, Canada, [3] I. Diethelm, L. Geiger, and A. Zündorf. Systematic story driven modeling. In Workshop on Scenarios and State Machines: models, algorithms, and tools; workshop at ICSE 2004, Edinburgh, [4] L. Geiger. Automatische JUnit Testgenerierung UML-Szenarien mit Fuja. Diploma Thesis. University of Braunschweig, [5] L. Geiger and A. Zündorf. Transforming graph sed scenarios into graph transformation sed junit tests. In Applications of Graph Transformations with Industrial Relevance (AGTIVE), Charlottesville, Virginia, USA, [6] J. Offutt and A. Abdurazik. Using uml collaboration diagrams for static checking and test generation. In 3th International Conference on the Unified Modeling Language (UML), pages , York, UK, [7] J. Ryser and M. Glinz. Using dependency charts to improve scenario-sed testing. In 17th International Conference on Testing Computer Software (TCS2000), Washington D.C., USA, [8] C. Schneider, A. Zündorf, and J. Niere. Coobra - a small step for development tools to collaborative environments. In Workshop on Directions in Software Engineering Environments, Scotland, UK, [9] A. Zündorf. The fuja toolsuite