Model-Based Extreme Testing: Theory and Practice

Transcription

1 Model-Based Extreme Testing: Theory and Practice W.E. Howden CSE, UCSD R.T. Kim CSE, UCSD P.P. Tran CSE, UCSD Abstract A new testing technique is described that combines aspects of extreme programming, model-based testing and capture-playback tools. It helps to solve several of the problems associated with extreme programming and model-based testing. A support tool called MOCA is described, and some of the underlying theory of MOCA behavioral models is presented. 1. Introduction An approach to testing, called model-based extreme testing (MXT), is described that combines ideas from extreme programming, model-based testing, and captureplayback testing tools. It capitalizes on the strengths of extreme and model-based testing, and helps to overcome some of their potential problems. In extreme programming (XP) [1], testing plays a critical role. Basic concepts include: early construction of test cases, automated testing, using tests as a kind of program specification, and incremental development of both code and tests. There are several potential problems in using just raw test cases as a specification. One is that they may be at too low a level of detail. Another is their possible failure to provide an overall, integrated picture of the program under test. Model-based testing [e.g. 3] is a systematic approach to functional and acceptance testing that uses a program specification model to guide test data selection and to measure the adequacy of a set of tests. The basic idea is to "cover" the model with a set of tests. For example, it might be required that for each state model transition, there should be at least one test that causes that transition to be covered, i.e. that causes that aspect of the program's behavior to be observed. Potential problems in the application of model-based testing include the lack of a specifications model. In addition, model-based testing is limited by its focus on what are described below as "functional" defects, as opposed to other kinds of defects. In MXT, we use the Rational Robot captureplayback tool [11] to generate event-state traces during program testing. These traces are then used to synthesize abstract models of the kind used in model-based testing. A model a) summarizes the functional coverage of the tests carried out so far, b) provides a test-based (partial) functional specification of the current (partial) system, and c) can be used to suggest additional defect-oriented tests that might be performed for the (partial) system that it describes. Important practical considerations in MXT include the use of models that are good test summaries and that can be generated and updated incrementally. It is also important to use a kind of model that facilitates the development of an underlying theory of model validity. Important theoretical considerations in MXT involve key issues that arise in model design and modelsynthesizing algorithms. One of the goals in model design is to use models that abstract, condense, and generalize the information from a set of traces. This implies the need to characterize the nature of abstraction. In addition, if we generalize from a set of tests, then the question of the validity of the generalization procedure arises. Previous work on extreme and agile testing, [e.g. 6] emphasizes early testing and test automation. Discussions of visualization and XP testing [e.g. 7] have been concerned with building tools for regression testing and for reporting percentages of unit and functional tests that have passed and failed. We believe this paper describes the first combined application of model-based, capture playback, and XP testing concepts. Related earlier work on model synthesis [e.g. 2, 8, 9, 13, 14] that considered some of same basic problems faced in the development of the MXT approach are discussed below in the Related Work section. Section 2 of the paper contains a description of the MOCA (Model Oriented Capture Analysis) model and the MOCA model generation tool. Section 3 describes the model builder procedure, and Section 4 gives an

2 introduction to a theory of behavioral models that can be used to characterize the validity properties of MOCA models and the MOCA model generation algorithm. Section 5 describes ways in which MOCA models can be used to support a testing effort and introduces the concept of a model-based test wizard. Section 6 is a discussion of related research and Section 7 contains conclusions and a description of ongoing and future work. 2. MOCA models 2.1. Basic model properties Model-based testing may involve different kinds of models, but the most common is a state model. MOCA models are a special kind of state model, and to distinguish them from specification state models we refer to them as behavioral models. The goal is to build a model that describes the set of possible behaviors that a program could exhibit, based on its observed behavior for the current set of tests. MOCA models are simple tree-like structures, containing a special "loop-back" notation for cycles. They are designed to facilitate incremental development, be easy to draw and re-draw, permit behavioral descriptions at multiple levels of abstraction, and to have a regular enough structure to enable the development of an underlying theory. In the work described in this paper, the programs under test were GUI-oriented applications. The events in the generated MOCA state models correspond to user input actions and the states to the resulting observable behavior (if any). The examples in the paper were generated during the testing of a simple dating system called DatingSystem.3 (DS). In this program, the first window that is displayed gives a Start/End option. If the Start button is clicked, a Login window is presented. When a user logs on, based on the logon name, the user is identified as being a dating system member, an administrator, or an unauthorized user. In the case of a dating system member, the system presents a Member Functions Choice dialog, allowing the member to choose between getting a date and changing member data. In the case where the get-a-date option is selected by a user/member, the user enters desired datee characteristics and the dating system searches the database for a match. If one is found, the datee details are given. If no match is found, a sorry message appears. In the case where the user is an administrator, the Admin Functions Choice dialog appears, allowing the user to choose between adding and deleting a member. Messages are then displayed indicating the outcome of the attempted actions. If the user is unauthorized, a warning message is given. Whenever one of the above functions is completed, the system returns to the initial window with the Start/End buttons, allowing the user to terminate a session or to repeat the log on. MOCA models have the following informal interpretation. For each path in a model, it is assumed there is some set of initial conditions (e.g., in DS, the data base of dating system members) such that if the events corresponding to the transitions in the model are followed, then the program behavior described by that path will occur. Figure 1 contains the model that was generated by MOCA from a set of tests for DS. In this case, the model was constructed from four separate tests and contains five basic paths. The top path corresponds to a test where the user was an administrator who added a member. In the second path the user/administrator deleted a member. In the third path, the user logged on as a dating system member and changed his/her datee properties in the database. In path four the logged on user/member chose the get-a-date option, and the properties of a discovered date were returned. In each of these tests, the user terminated the session after performing the desired function. This resulted in the behavior described by the fifth, bottom path, showing a transition from the DatingSystem.3 state (which contains the Start/End buttons) to the End termination state. The state node appearing at the end of each of the paths, except the End path, indicates a repeated occurrence of the DatingSystem.3 state, and is underlined. Underlining is the loop-back notation in a MOCA model, indicating transfer of control in the model back to the previously occurring instance of the loopback state on the path to the loop-back. This feature of a MOCA model is further discussed in the following section on generalization and loops. 2.2 Behavior generalization and loops The model in Figure 1 was generated from a set of four Rational Robot traces, whose identities are listed in the left part of the MOCA window. Because of the loopback notation, the model actually describes an infinite number of possible behaviors, corresponding to the infinite number of possible paths in the model. For example, a user could repeatedly logon as a dating system member and use the get-a-date option before ending a session by choosing End in the Start/End window. Figure 1 models both these and all other different possible combinations of potential system behavior. The same range of possible behavior would be exhibited in the model regardless of the way it first appeared in a test. For example, if these four basic kinds of behavior had occurred in succession in a single test instead of in four separate tests and traces, the model

3 Figure 1: Typical MOCA model would be the same. The only difference would be that only one trace would be listed on the left side of the screen. The behavior generalization that is incorporated in Figure 1 occurred because the tester identified the DatingSystem.3 state as a context-free state. This means it is a kind of "re-set" state where the system is returned to a previous state, so that all of the behavior that was possible in a previous occurrence of the state is also possible in the repeated occurrence. Whenever the MOCA model-builder finds such a state in a trace, it generates a model sub-path that terminates in a loopback state.

4 2.3. Abstract Models As indicated above, the model in Figure 1 was generated from a set of four Robot traces. A set of Robot traces can be thought of as a concrete behavioral model. Since all traces for a given program will start with the same initial state node, a trace set can be thought of as a primitive tree model where the traces all have a common root node. MOCA is capable of generating three kinds of abstract models, at increasing levels of abstraction: Abbreviated Robot (AR), Control Set (CS), and Window (W). The model in Figure 1 is a W model. A W model shows the frame captions for the GUI window frames that occur during a set of tests, along with the events that cause transitions from one window to the next. Some of the detailed events that occur in a test, such as individual mouse movements, are abstracted out by Robot and do not occur in the Robot traces. Others, such as use of an edit box in a window to enter text, are abstracted out by MOCA in forming a W model. The fact that such a control was used would be recorded in a CS model, and the details of its use would be summarized in an AR model. An AR model is a kind of "cleaned up", or "normalized", Robot trace. The user of the MOCA tool can go back and forth between different levels of abstraction for the same set of Robot traces. 3. MOCA model-builder The MOCA model-building tool was designed as an add-on to Rational Robot [11]. The user identifies a set of tests, and then runs the program that is being tested under Robot using those tests. Robot creates a test script for each test, which is stored in a script file. These are the traces used to build a MOCA model. The user of the MOCA model-builder can cause a model to be built from a collection of traces or have a path added to the model one test-trace at a time. There are three basic model-building actions that the modelbuilder must perform: abstraction, common prefix overlay and context-free generalization. When a Robot test trace is added to the current MOCA model it is abstracted to create a model path, which is then added to the current model. The process of adding the path involves generating a set of paths called the "context-free extension" of the path. Each path in the extension is added to the model using common prefix overlay. Common prefix overlay is simply the process of causing all paths that have a common initial segment to share that initial segment in the tree-shaped MOCA model. The context-free extension of a path P is a set of paths whose behavior is assumed to be possible, based on two factors. The first is the observed behavior in P and the second is the assumption that one or more states in P are context-free. Recall that a state is context-free if any observed behavior that was initiated in one instance of the state is observable starting at any other instance. This means if we see such a state for the second time on a path, it is valid to put a loop-back to the first. In addition, that first instance could be the starting point for any of the behaviors that were observed to follow the second state instance in the path. The context-free extension algorithm works as follows. Suppose that P is an observed path to be added to a model M. If P has no repeated instances of a context-free state, then the path can be simply added to M, using common prefix overlay. Suppose that P has a repeated instance of a context-free state, and that x is the first of these. Then P must have the form P = AxBxC for some paths A, B and C, where B is such that the second instance of x after B is the first repeated instance of x in P. In this case we add the path AxBx to M, using common prefix overlay. Recall that x is the loop-back notation, as in Figure 1. We then apply the procedure recursively to the path AxC. Since the algorithm halts when there are no repeated instances of a context-free state in a path, and because it reduces the number of such pairs of occurrences by one in each recursive application of the algorithm, it must terminate. 4. Theoretical foundations There are a number of concepts that were introduced in the discussion of the MOCA model that can be treated more formally. In some cases this is done just to clarify, and in others it is necessary to characterize and prove important validity properties. We will consider the following four ideas here: abstraction, context-free behavior, context-free completion, and model-builder correctness Abstraction As described above, MOCA supports 3 levels of abstraction. Different kinds of model abstraction can be devised. One property that some kinds of abstraction have, but not others, is defined as follows. Definition Suppose that M 1 and M 2 are two models, and that M 1 is an abstraction of M 2. Then the abstraction satisfies the path preservation property if the paths in M 2 form a set of equivalence classes and every path in M 1 corresponds to one of the equivalence classes of paths in M 2. The path preservation property was found to be important for the synthesis of understandable, consistent MOCA models of functional program behavior.

5 One of the most common kinds of state model abstraction is submodel abstraction, in which a state in the more abstract model is mapped to either a single state or a submodel in the less abstract model. If one of the states in a path P in the abstract model maps to a submodel in the less abstract model then the path P maps to a set of paths in the less abstract model. Submodel abstraction has the path preservation property, and includes W-abstraction, in which the submodels correspond to the more detailed actions that are performed inside an instance of a window. Another kind of abstraction is projection. Suppose that we follow an approach in which states in models correspond to different possible sets of values for a set of state variables. Then, in projection, a subset of the variables is ignored and states that have the same values for the remaining variables are grouped together in an abstract state. Transitions from one abstract state S 1 to a second abstract state S 2 are derived from the transitions between the concrete states grouped into S 1 and the concrete states grouped into S 2. In some approaches, selftransitions are used when two concrete states that were linked by a transition are grouped together into the same abstract state. Projection abstraction can result in models in which there are paths in the abstract model that have no corresponding path in the less abstract model, and therefore does not satisfy the path preservation property. Restricted kinds of projection do satisfy the path preservation property. In contiguous subpath projection, states in a concrete model that form a contiguous subpath can be projected to form an abstract state, but this is really just another form of submodel abstraction. Other kinds of abstraction that have the path preservation property can also be considered. Formal approaches to the study of model relationships include simulation/bi-simulation, as in [10]. This kind of relationship does not guarantee the path preservation property, which we found necessary for MOCA model development Context-free behavior The nodes in a behavioral state model are considered to be state instances, rather than states. This is because two nodes may have the same label, i.e. state variable values. Even though they have the same label, they may be involved in different kinds of behavior, i.e. the state of which they are instances is context-sensitive. Duplicated states in a MOCA model occur because of the tree-like structure of the model, in which states with the same label are not "merged" unless they are part of a common prefix. Even if we gave up on having a tree-like model, we could not necessarily merge other states having the same label. There may be factors in their context, which is determined by the paths leading up to their occurrences, that will cause the system to act differently in these states. Also, they may "look" the same in the current set of tests, but there may be other possible behaviors that we have not seen yet that would appear in other tests. Merging them could result in models that contain non-existent behavioral paths. Important situations in which state instances with the same label can appear that are not really the "same" state, are side-trip and abstraction. Side-trip contextsensitivity occurs when common code or data is re-used in different functional contexts. In the case of DS, the message feature is implemented using the singleton pattern. This means that whenever a message is displayed, regardless of the functional context, its occurrence looks the same in a Robot trace and hence in the abstract model paths that are derived from the trace. However, different instances may lead to different subsequent behavior Abstraction context-sensitivity occurs when state instances look the same because distinguishing details have been abstracted out. Suppose that at the beginning of a DS session, a user simply presses the Enter button in the Login window without entering a name. Then the system will respond with an Unauthorized User message. Alternatively, suppose the user enters a valid member name, selects the get-a-date option, performs the associated actions, and when the system returns to the DatingSystem.3 state, clicks the Start button, which leads to the Login state/window again. If the user clicks the Enter button this time, without entering any name in the edit box, the response will not be the same as it was above, because the previously typed in name is saved. Instead of an Unauthorized User message, the system will respond with the Member Function Choice window. This is because in DS the typed-in name is saved and reused if no new name is entered at the second login. These two instances of the Login state lead to different system behavior, indicating that the Login state is context-sensitive. When a tester declares that a state x is context-free, this means that we are assuming the following. Suppose that we have a complete set of behavior paths for all possible initial conditions for some program. Construct a tree T from these paths using common prefix overlay. Suppose that x' and x" are two instances of a state x in the paths. Consider the subtrees S' and S", rooted at x' and x". When we say that x is context-free, we mean that S' and S" must be matching trees. Note that behavior trees could have an infinite number of paths. We can distinguish between local and extended context-free behavior of a model state. If the trees S' and S" that are described above are restricted to paths of length 1 and to the initial length-1 subpaths of longer paths, we refer to them as their root node's local behavior. If all instances of a state have the same local behavior, the state is locally context free. The complete

6 behavior trees, as defined above, correspond to extended context-free behavior. This formal treatment of "reset" states, using the concept of a context-free state, makes results possible such as the following, which give deeper insight into loop synthesis and model generalization. The proofs for the following and other theorems in this section can be found in [4], which also contains additional MOCA theory. Theorem 1 Suppose that M is a model in which all states have local context-free behavior. Then all states will have extended context-free behavior Context-free completion In the above discussion of the MOCA model construction procedure, we introduced the concept of the context-free extension of a path P. The idea was that when a new model path was derived from a Robot trace, that both P and a set of associated paths should be added to the current MOCA model. The associated paths correspond to the additional behavior that is assumed to be possible in the system under test, given the observed behavior and related context-free state assumptions. This concept can be treated more formally and used to establish "soundness" properties of the MOCA modelbuilder. Definition Suppose that PS is a set of paths, and that one or more states x have been declared to be context-free. Then the context-free completion PS' of PS is the set of paths that is defined as follows. PS' contains all of the paths in PS, plus all paths that can be formed from any two paths P 1 and P 2 in PS' using the following construction. Suppose that P 1 and P 2 contain instances of the common context-free state x and have the forms P 1 = H 1 xq 1 P 2 = H 2 xq 2, where H 1 /H 2 is the initial segment of P 1 /P 2 up to some instance of x, and Q 1 /Q 2 is the tail segment of P 1 /P 2 after that instance of x. Then the path H 1 xq 2 is also assumed to be in PS'. When the MOCA model-builder, MB, is applied to a path P, it generates a set of paths EP that contains P, which is added to the current model. This will result in a model that contains not only the paths EP, but additional paths that are indirectly generated by interactions between the existing paths MP in M and the added paths EP. We might wonder if these extra paths "make sense", i.e if the model-builder is "sound" in some reasonable way. Theorem 2 MB is sound in the sense that all of the paths that it creates when it adds a path P to a model M are contained in the context-free completion of the set of paths MP {P}. The proof of theorem 2 depends on a variation of the context-free completion construction, which is defined below. Definition Suppose that PS is a set of paths that contain instances of one or more context-free states. Suppose Q is a path that is constructed from paths in PS as follows. Q starts out at the beginning of some path P, and follows it up to an instance of a context-free state x. At this point it "crosses over" to some other path P' that contains an instance of x, and starts following P' from that instance until it reaches an instance of a context-free state y (which may be an instance of a different state or of the same state x). At this point it crosses over to a path P" that contains an instance of y, and starts following P" from that point forward. Q continues this process of following one path in PS then crossing over to another at an instance of a context-free state either indefinitely or until it reaches the end of one of the paths it is following. This process of constructing Q is called a random path walk. The concept of a random path walk can be used to prove Theorem 2. We give an idea of the general proof strategy here. The details are given in [4]. First, we need the following. Theorem 3 Suppose that a path Q is constructed from a set of paths PS using a random path walk through paths from PS. Then Q is in the context-free completion of PS. Then we show the following. Suppose that M is a model and that P is a new path that was added to M using the MOCA model-building procedure. Let M' be the new model, and let P' be a path in M' that was not in M and is not P. It is possible to show that P' can be constructed from P and the original paths in M using a random path walk. Theorems like the above tell us that MB is not "over-generalizing", but they do not tell us what the set of paths in a model M "look like". It is possible to prove additional theorems that shed some light on this question. Theorem 4 Suppose that we have a set PS of paths having the form of the regular expression Ax( B i x)d = AxB 1 xb 2 x...b n xd, 1 i n, which is constructed from a set of subpaths A j, 1 j k, i.e. each B i = some A j. Let PS' be the context-free completion of this set of paths, let PS" be the set of paths in the MOCA model that is generated by the MOCA model-builder from this

7 set of paths, and let P"' be the set of paths that is described by the following regular expression: Ax((ΣA j )x) * D, 1 j k. Then PS' = PS" = PS"'. The paths in the DS example have the structure of the set of paths PS"' in Theorem 4. This kind of path structure will be observed in a system with a single toplevel mechanism for repeated usage. Theorems can also be proved for more complex structures involving nested loop-back architectures Abstractions and Context-Free Behavior Earlier, the Login window in the DS program was used in an example of how different instances of a state may have different behavior. The instance that occurs when a user first logs on can behave differently from an instance that occurs after the user has completed some function and returned to the Login state. In the above discussion, it was actually the local behavior that was different: in the first case, pressing the Enter button without entering a name will transit to one state, whereas performing the same action in the second case transits to a different state. The example is a little misleading, because whether Logon is context-free does not depend on whether or not these particular uses of Logon act differently, but on the total possible behavior of the Logon state in the specified contexts. As it turns out, this depends on the level of abstraction of the model in which the behavior is being observed. In this case, the Logon state is not contextfree at the AR level of abstraction, but is at the W level. The reason why Logon is context-free at the W-level is because W-level abstractions do not display behavior that takes place "inside" a window, only the transitions from one window to the next. This means that it is possible, while performing actions inside a second instance of Logon, to simulate the behavior of any first instance. For example, suppose that when the Login dialog is displayed the second time in a path, with its edit box still containing the entry from the first occurrence, the tester were to backspace over the entry and then click the Enter button. This will have the same effect as clicking Enter without entering any text in the edit box in a first occurrence. At the W-level of abstraction, the model will not contain any of the backspace actions, so the different kinds of possible observable behavior at the W level will be the same for all first and second occurrences of Login. However, in the AR model, the fact that entries were made in the edit box will appear as events in the AR paths and the instances of Login in these two contexts will not have the same behavior, so that the AR-level Login state is not context-free. Figure 2 contains fragments of MOCA-generated model paths that illustrate this example. In the left AR path, a legal member logged on and carried out some member function, indicated with ellipses. After this the member selected the Start option to keep using the system and when the Login box appeared for the second time, simply clicked the Enter button. The system used the retained name and displayed the Member Function Choice dialog, which was then used to select the get-adate option. In the AR path on the right, the user simply clicked the Enter button in the initial Login box without entering a name, and the resulting behavior was the display of a message, after which the user ended the session. At the AR level of abstraction there is no way for the behavior of the initial instance of the Logon state seen in the right diagram to appear in the context of the second Logon in the left diagram. At the W-level of abstraction, the behavior model for the case on the right of Figure 2 would be almost identical to that given for the AR model. For the case on the left, the behavior would be similar to what is seen there, but there is a way to cause the behavior of the logon in the right path to also appear in the context of the second Logon instance in the left path. The dependence of the occurrence of context-free behavior on abstraction levels is not surprising, since at some level of abstraction all states are context-free. This is trivially true since we could simply define a kind of abstract model that had a single state. The transition from context-sensitive to context-free behavior is one of the factors in determining a useful abstraction. 5. Test data selection and the MOCA wizards In traditional model-based testing, test data is selected that will "cover" a model in some way. For example, the goal may be to choose a test set that will cause each model transition to be covered at least once. Because a MOCA model is constructed from tests (i.e. the Robot test traces), coverage occurs automatically. In model-based extreme testing, models are used both to summarize the possible system behavior that can be inferred from a set of tests and to help generate tests. But the ways in which they are used to guide test selection is not quite the same as in model-based testing. In the MOCA approach to testing, we identify three general categories of defects: functional, special-case, and combinational. Functional defects correspond to basic system functionality, which we expect to correspond to some part of a model. We assume that when a defect like this occurs the program fails for all uses of this function. Defects like this will be discovered during model formation, provided that the functionality in question was used to build the model. Special-case defects include the familiar boundary or extreme-range

8 Figure 2: Sample AR - level abstractions defects. Examples of special-case defects found in the Dating System (DS) included the following. If a member/user fails to select any datee characteristics in the datee properties list boxes, and then presses the OK button to cause the DS system to search for a date, DS will fail. For defects such as these, it is reasonable to formulate general test selection rules, many of which appear in the literature. We can often view the occurrence of functional defects as the result of failing to implement some functionality, and special-case defects as the result of failing to implement that functionality correctly. Combinational defects involve (often rare) combinations of conditions that result in a program failure. It is seldom useful to try to construct generic testing rules for these kinds of defects since there would be too many of them and they would seldom be effective or applicable. Examples of this kind of defect that were found in DS included the following, which occurs due to peculiarities in data structure indexing arithmetic. If an admin/user, during a session, tries to delete a nonexistent member, and the admin user has not previously in this session deleted an existing member, then the program will fail. It will not fail if the deletion of a nonexistent member was preceded by a legal deletion. The MOCA testing strategy involves the use of wizards that help the user generate tests for each of the above three kinds of defects. First consider functional defects. These could occur if the current (partial) model is incomplete or inconsistent with respect to the current (partial) system. It could be incomplete if there are branches and state/window instances possible in the current (partial) system that are not included in the model. It could be inconsistent if, due to context-free assumptions, the model contains paths that do not occur in the code. A wizard could help detect these potential problems by looking at sources of information outside the model. One way of doing this for a GUI-oriented program would be to determine if the window transitions that are possible in the code match the window transitions that appear in the current model. For special-case defects, a wizard could examine the states and events that appear in an abstract model and then examine the concrete traces used to build the model to see if additional tests are needed to test special-case conditions. Examples include examining all the instances of a window that contains a control, such as an edit box, to make sure there is at least one test instance where that control was never used. E.g. the enter button was clicked without entering text. The current MOCA tool contains a simple wizard that provides basic test development assistance for detecting certain kinds of functional and special-case defects. Let W be a window/state in a Control State model M, let W C be the set of controls appearing in the definition of W, and let W i, 1 i n, be the set of instances of W. Let U i be the set of controls in W C that were used in W i. Then the user can click on any state in the

9 displayed model M, and MOCA will return the following: W C, U i and W C ~U i. The last term corresponds to the kind of information, described above, that might be useful for detecting possible functional defects corresponding to missing parts of the MOCA model. The formula given below could be used to assist in the identification of unexplored special-case tests: U i, 1 i n. This formula identifies controls that were used in every instance of a window, i.e. there were no instances in which they were not used. In the case of combinational defects, we have followed a legacy defect approach. The idea is to maintain a database of past defects, organized by functionality and application. The tester examines the database, looking for relevant entries. If one is found, a test is constructed and run. MOCA contains a feature that allows the tester to copy legacy defect descriptions into a defect description database that is associated with a current model. The MOCA user can then define one or more tests for an associated legacy defect and store the associated Robot traces along with the defect description. When a tester thinks of a new test, based on suggestions from the Wizards, the test can be run under Robot. At the user's discretion the trace behavior that is generated by Robot for the test can be converted to an abstract path (or set of abstract paths, as determined by the model building procedure) and added to the current MOCA model. Presumably, the tester will do this if the test uncovers new kinds of model behavior, otherwise it will be unnecessary. 6. Related work Research on state model synthesis has previously been described by several authors [e.g. 2,8,9,13,14]. In general, this previous work involves the synthesis of models from descriptions of component interactions in a distributed system. The interaction descriptions are projected on to the individual components, to get a description of their individual behavior, and the projections are used to construct finite state models. Some of the basic ideas in the work described here previously appeared in [2]. In this previous work, the interacting components were processes. A tree of traces of their compound behavior was captured, which in turn was projected to construct a tree of behavior traces for each process. As in our work, the models that were derived from the traces could be used for a variety of purposes, including test case summaries. The MOCA work described in this paper differs from that in [2] in several ways. The MOCA project was applied to functional testing of application programs, not distributed systems, and was concerned with system functionality rather than component interaction. In addition, a commercial playback tool was used for generating traces. The MOCA models have labeled states as well as labeled event transitions, which provide a richer semantic context than the models in [2], motivating the consideration of context-free behavior and state instances. Also, we considered multiple levels of abstraction and distinguished between functional abstraction and loop synthesis, which is a generalization as opposed to an abstraction issue. Other differences involve the type of model used in MOCA and its associated model-building procedure, which appear to be unique to MOCA. In [2], behavior trace trees similar to MOCA state behavior trees are used to identify equal states and to synthesize loops associated with state repetition. This would not work in the MOCA context for the following reason. Suppose, as in DS trace paths, there is a state having multiple occurrences where the observed behavior subtrees are different for each instance because the tester tested a different system feature each time. Automatic recognition of states based on common behavior trees would not be successful. In MOCA, the user hypothesizes that some state has the potential for the same behavior, i.e. the state is context-free, and this is used as the basis for loop synthesis. Additional differences between the work in this paper and in [2] include the theoretical results presented above, which are based on the concepts of context-free states and context-free completion introduced in this paper. Later work in [8,9,13,14] was concerned with generating state models for components by projecting sequence diagrams [5,12] on to their components and then constructing a state model for each component. In this work the sequence diagrams are specifications, rather than traces, which makes it somewhat different from the work described in [2] and in this paper. In the work involving synthesis of models from sequence diagrams, states are assumed to occur in between message events. This is done in different ways and additional information is sometimes used, such as message preconditions [14], or user-defined message sequence chart conditions [8]. In general, a loop is synthesized if the same state occurs in a component's projection trace. This can be modified in several ways in order to avoid the generation of "false loops" or "overgeneralization". In [9] the analyst can reject or accept model structuring and synthesis suggestions. In [14] additional conditions must be satisfied before a loop can be recognized. The idea in MOCA, that states with the same label may be instances of the same state but should only be used for synthesizing loops if they are contextfree, leads to a different approach to loop synthesis.

10 However, like [8,9] it follows a kind of interactive approach to the problem. In summary, the contributions of the work described here include: the development of the MOCA model and model-building procedure, the formalization of the concepts of context-free behavior and context-free completion, the characterization of validity criteria for a model-building procedure, and the associated validity proofs for the MOCA model-builder. 7. Conclusions and future research The MOCA project was a successful proof of concept. The MOCA-style model was found to be: intuitively appealing, much easier to draw than a conventional state model, and well suited to incremental development. Additional behavior corresponding to new tests/traces can be easily added to a model. As more functionality is added to a program, the model can easily be augmented with the new behavior that was observed in the additional tests. We were able to generate an interesting underlying theory for MOCA behavioral models based on the concepts of context-free behavior and context-free completion. The recursive formulation of the MOCA model-building procedure made it possible to formally reason about its operation and argue its correctness. There are several fruitful areas for further development. There is no claim that MOCA is now at the level of a commercial testing tool, either in terms of its scope or its reliability. The current MOCA tool is designed for limited-feature, Windows-based GUI programs. One future direction is to expand the MOCA tool's capabilities and to carry out extensive experiments with a broad range of industrial programs. The theory section in this paper contains a sample of the kinds of results that have been proved. These are relevant both to model-based extreme testing and to model synthesis in general. There are a variety of additional kinds of model and model-builder properties that could be investigated and formalized with additional theorems and definitions. We are also considering the application of the basic MOCA theory to more complex kinds of models, such as state charts. The current MOCA test wizards are very simple versions of the kinds of functional, special-case, and combinational wizards that might be constructed. They work by inserting Visual Basic commands into Robot traces where they collect information about windows child controls. More powerful wizards might directly access source code or source code models such as document object models. The approach to wizards taken in the MXT project follows a current research trend in which static and dynamic analyses are combined. In this case, dynamic analysis is used to build a model that is based on actual observed behavior. Static analysis is used to detect possible problems in the model, and to suggest additional tests. Test wizard design is a research area of its own, and is open to extensive investigation. The current, simple MOCA wizard that is used for functional and specialcase defect test generation is currently being upgraded to include additional functionality. It is also possible to consider other kinds of applications, and work is now in progress on the application of MOCA to distributed and web-based systems. This work involves a number of additional interesting practical and theoretical issues. 8. References [1] Beck, K., Extreme Programming Explained, Addison Wesley, [2] Boigelot B. and Godefroid, P., Automatic Synthesis of Specifications from the Dynamic Observation of Reactive Programs, TACAS '97, July, 1997, pp [3] El-Far, I. K., Whittaker, J.A., Model-based Software Testing, Encyclopedia of Software Engineering, ed. J.J. Marciniak, Wiley, [4] Howden, W.E., Kim, R.T., and Tran, P.P, Model Oriented Capture Analysis (MOCA), CSE Technical Report, UCSD, July, [5] ITU-T.Z. 120, Message Sequence Charts, (MSC), ITU, [6] Jeffries, R.E., Extreme Testing, Software Testing and Quality Engineering, March/April, [7] Kelliher, T., Blezek, D., Lorensen, W., Miller, J., The Frost Extreme Testing Framework, Proceedings, Software and Internet Quality Week, 9T2, SRI, May, [8] Krüger, I., Grosu, R., Scholz, P., and Broy, M., From MSCs to StateCharts, DIPES 98, Kluer, [9] Makinen, E. and Systa, T. An Interactive Approach for Synthesizing UML Statechart Diagrams from Sequence Diagrams, OOPSLA 2000, ACM, [10] Milner, R., Communication and Concurrency, Prentice Hall, [11] Rational Software Corporation, Users Guide for Rational Robot, Rational Software Corporation, [12] Rumbaugh, J., Jacobson, I., Booch, G., The Unified Modeling Language Reference Manual, Addison Wesley, [13] Vasilache S. and Tanaka, S., Synthesizing Statecharts from Multiple Interrelated Scenarios, Proceedings of the International Symposium on Future Software Technology (ISFST2001), Zheng Zhou, China, Nov. 5-8, 2001, pp [14] Whittle J., Schumann J., Generating Statechart Designs from Scenarios, Proceedings of the 22nd International Conference on Software Engineering, ACM, 2000, pp