Test Case Generation from UML State Machines

Transcription

1 Test Case Generation from UML State Mahines Dirk Seifert To ite this version: Dirk Seifert. Test Case Generation from UML State Mahines. [Researh Report] <inria v2> HAL Id: inria Submitted on 23 Apr 2008 HAL is a multi-disiplinary open aess arhive for the deposit and dissemination of sientifi researh douments, whether they are published or not. The douments may ome from teahing and researh institutions in Frane or abroad, or from publi or private researh enters. L arhive ouverte pluridisiplinaire HAL, est destinée au dépôt et à la diffusion de douments sientifiques de niveau reherhe, publiés ou non, émanant des établissements d enseignement et de reherhe français ou étrangers, des laboratoires publis ou privés.

2 Test Case Generation from UML State Mahines Dirk Seifert Loria Université Nany 2 Campus Sientifique, BP 239 F Vandoeuvre lès Nany edex Dirk.Seifert@Loria.fr ABSTRACT In this paper we desribe a omprehensive approah for onformane testing of embedded reative systems. Based on a formal speifiation, namely UML state mahines, we automatially generate test ases and use them to hek the funtional onformane of a system under test. Our test ases inlude not only stimuli to trigger the system under test, they also inlude possible orret observations to automatially evaluate the test ase exeution. In ontrast to lassial Harel Stateharts, state mahines behave asynhronously, whih makes automati test ase generation a hallenge. The TEAGER Tool Suite implements the automati generation, exeution and evaluation of test ases and proves the appliability of our test approah. 1. INTRODUCTION The impat of embedded systems in our everyday life is steadily growing. They are present not only in very speifi ontexts but also in nearly every eletrial devie we use. In general, embedded systems omprises of hardware and software omponents interating with a speialized tehnial environment via sensors and ators. The main reason for their suess is the ombination of speifi or high-performane hardware with the flexibility of software. The software is responsible for ontrolling the hardware and software omponents and for alulating reations as responses to reeived events. It is remarkable that users unonditionally trust in the orret funtioning of suh systems. This is true not only for safety ritial systems like an anti-lok brake system in a ar but also for omparatively simpler systems like a ellular phone. The development should satisfy this onfidene. Erroneous systems annoy the ostumers and are a high ommerial risk in mass ustomization. Moreover, size and omplexity of nowadays systems whih have to be developed, demand for improved and automated proesses: for development as well as for quality assurane. Model-based software development bases on setting up models of the system to be onstruted. This approah has proved to be use- This work has partly been done in the researh group Softwaretehnik at the Tehnishe Universität Berlin. ful, beause it allows developers to first elaborate the most important properties of the software before proeeding with the implementation. Nowadays the Unified Modeling Language (UML) [24] is widely used to model systems and to guide development proesses. The UML omprises of several diagram types to speify the struture and the behavior of a system or system omponent. State mahines are used to either desribe the disrete reative behavior (behavioral state mahines) or to desribe the usage protool (protool state mahines). In this paper we refer to behavioral state mahines. We use them to speify the states a system an take and ations it an exeute during its lifetime in response to external and internal events. Due to the disrete reative harater of state mahines and the possibility to ompletely desribe the disrete behavior of a system, state mahines are appropriate to model embedded systems. We intend to use state mahine models not only for development but also to support an automated quality assurane proess. In Setion 2 we introdue the syntax and semantis of state mahines we need in this paper by means of an example. In Setion 3 we present our test approah. In this approah we automatially generate test ases out of a state mahine speifiation. The test ases inlude not only the stimuli to trigger the system under test, they also inlude the possible orret reations. The latter allows an automati evaluation of the test ase exeution. We desribe the underlying theory, the developed test ase generation algorithm, how approximation tehniques are used to develop a pratial approah and how the test ase generation and exeution an be ontrolled and evaluated. In Setion 4 we desribe the TEAGER Tool Suite whih ompletely implements the introdued approah. Finally, in Setion 5 we onlude our work, disuss related work and give an outlook to ongoing and future researh. 2. STATE MACHINES UML state mahines [24] are an objet-oriented extension of the lassial Harel-Stateharts [15]. In this paper we use behavioral state mahines to desribe the sequene of states a system or system omponent an take and the ations it exeutes when hanging these states. State mahines are mathematial models with a graphial representation: the nodes depit simple or omposed states of the system and the labeled edges depit transitions between these states. Composite states allow to hierarhially and orthogonally struture the model, thus reduing the graphial omplexity. Labels express onditions under whih transitions an be taken and the ations whih will be exeuted when the transition is taken. Events are used as triggers to ativate transitions and an be parameterized to exhange data. Optional, every state mahine has a data spae whih an be read and manipulated by the state mahine during

3 boolean incdfull = false; boolean intapefull = false; integer trakcount; Car Audio System Audio Player power Off power CD Player On d_insert / incdfull = true; trakcount = d_insert.1 bak Tuner Mode bak P1 P2 tape_ejet [ not in("cd Full") ] / tuner_plays sr [ not in("cd Full") ] / tuner_plays CD Empty CD Full sr [ in("tape Full") ] / tape_plays d_ejet / incdfull = false; bak P4 P3 bak Tape Mode d_ejet play sr [ in("cd Full") ] / d_plays CD Mode CD Playing play sr [ not in("tape Full") ] / tuner_plays bak Bakward Spooling Tape Playing bak sr [ in("tape Full") ] / tape_plays bak Forward Spooling Tape Player Tape Empty tape_insert / intapefull = true; tape_ejet / in TapeFull = false; Tape Full Next Trak Former Trak sr [ in ("CD Full") ] / d_plays tape_ejet [ in("cd Full") ] / d_plays Figure 1: State mahine speifiation for the Car Audio System. exeution. More preisely, it is possible to read the data to desribe speifi onditions when a transition an be taken or to manipulate the data and exhange information within the ations. The general struture of a transition onsists of a soure state, a trigger event, an optional guard in square brakets, an optional ation sequene separated from the previous elements by a slash, and a target state. With the optional guard a fine-grained ondition to enable the transition an be desribed depending on the system s state. Hene the ativation of the soure state, the trigger event and the fulfilled guard ondition onstitute the ondition whih must hold to enable the transition. An ation an either be a statement manipulating the data spae or the generation of new events. The ation sequene and the subsequently ative target state onstitute the effet of the transition. In this paper we use a substantial subset of state mahines to study automated test ase generation and evaluation based on state mahines. We assume the reader to have some basi knowledge of transition systems. In the following we briefly desribe state mahines by means of an example. Afterwards we disuss semanti issues whih make automated test ase generation a hallenge. A omplete and detailed desription as well as a preise definition of the semantis (inluding the integration of omplex data) an be found in [18]. 2.1 Example To demonstrate the state mahine notation we use a state mahine model speifying the behavior of a simple sound devie in a ar. Figure 1 shows this model. The requirements for suh sound devie ould be as follows: It should be possible to turn the Car Audio System on and off. When turned on, it should play one of three different audio soures, namely radio, tape or ompat dis, respeting the presene of a tape or a ompat dis. It should be possible to hange between available soures. Furthermore, it should be possible to swith between four radio stations, to spool a tape bakward or forward, or to selet the previous or the trak of a ompat dis. Abstrating from any physial devies we introdue the following events to model the required behavior: power, sr (to swith between the different soures),, bak and play. Additionally we introdue events signaling the insertion and the ejetion of a tape or a ompat dis as well as events to signal system reations. Furthermore, we use data variables to store detailed information about the urrent state. For example, we use an integer variable trakcount to store the number of titles of an inserted ompat dis. Figure 1 shows a state mahine model of the sound devie inluding the underlying state spae. At the highest level of abstration the model onsists of an orthogonal state omprising three regions. The two regions CD Player and Tape Player model the information if a tape or a ompat dis is inserted into the system or not. The more omplex region Audio Player models the ontrol of the system. The region is refined by two states: Off and On. Initially the system is assumed to be swithed off, expressed by the small arrow leaving a bullet and ending at the Off state. When the event power is proessed the system is swithed on and starts to play the radio (again expressed by a small arrow). The omposite state On is refined into states modeling the three signal soures. The transitions between these states desribe the hanges between the soures as reation

4 to an event sr. For example, when the system is in Tuner Mode and a tape and a ompat dis are inserted into the system (i. e. both in-prediates are true) and the event sr is proessed, the system an either swith to the tape mode or swith to the ompat dis mode beause both transitions are enabled and an fire. All three substates of Audio Player are further refined to desribe the partiular behavior in reation to the events, bak and play in eah state. 2.2 State Mahine Semantis The semantis of state mahines is adapted from the STATEMATE semantis [17, 16] to fit into the objet-oriented paradigm. As desribed above a state mahine an be refined by simple omposite and orthogonal states. Simple omposite states ontain exatly one region and orthogonal states ontain at least two regions. In every region only one substate an be ative at a time. The state whih is entered by default when a region is entered is marked by an arrow emanating from a filled irle. The hierarhial ordering of states forms a tree struture with a region as the root node, simple states at the leave nodes and in between (alternating) omposite states and regions. Due to orthogonal regions a state mahine an be in several states at a time. We all the set of all ative states a onfiguration. For the same reason it is possible that more than one transition an fire at a time; one in every ative orthogonal region. We all the set of all jointly firing transitions at a time firing transition set (FTS in short). Due to the hierarhial struture of state mahines it an happen that two transitions are enabled for firing on different hierarhy levels of a state. Taking both would lead to a onfiguration whih is not well-formed. A similar situation arises if a transition leaves an orthogonal region. In this ase the transition annot fire together with an enabled transition in another orthogonal region. In both ases the transitions are said to be in onflit with eah other. Suh situations are identified if two transitions leave idential states in the state hierarhy. The UML desribes a two step proess to resolve suh onflits. In the first step a priority sheme is used. Transitions emanating from a state deeper in the state hierarhy has priority over the other transition. Thus the more refined transition is taken. 1 Nevertheless, not all onflits an be resolved using this priority sheme. In the seond step only transitions are seleted whih are not in onflit to eah other respeting maximal progress of the system. A so-alled transition seletion algorithm selets all maximal sets T T of enabled transitions fulfilling the following requirements: t : T enabled(t,,e,d) (1) t 1,t 2 : T t 1 t 2 t 1 t 2 (2) t : T \ T enabled(t,,e,d) t : T t t t t (3) First, all transition in the firing transition set must be enabled regarding the urrent onfiguration, the trigger event and the urrent data assignments. Seond, all transitions in the set are mutually onflit free (expressed by the operator). Third, there is no enabled transition outside the set whih is onflit free with the transitions in the set or with higher priority than a transition inside the set. Thus, transitions with the highest priority are taken and maximal sets are hosen. Result of the transition seletion algorithm is a set of firing transition sets (PPT). 1 This differs from lassial Stateharts. But it reflets the objetoriented inheritane behavior. Eah set represents a valid firing transition set. It is important to mention that for exeution one suh set is arbitrarily hosen, and that the order in whih the transitions are fired is arbitrarily hosen, too. In onsequene, all set hoies and transition permutations form the set of all possible semanti steps of the state mahine at a time. This is important if we want to ompute the possible orret behavior for an input sequene to evaluate the test exeution. In opposite to the lassial Stateharts, the event proessing takes plae in a so-alled run-to-ompletion step. This asynhronous event proessing demands the proessing of the previous event to be ompletely finished before the event an be proessed. Therefore it is neessary to buffer reeived events in an event store. Consequently, the ourrene of an event and its proessing are asynhronous, i. e. take plae at different times. It follows immediately that a possible (observable) reation of the system also takes plae asynhronously. The semanti model of state mahines builds on the semanti steps a state mahine an exeute during its lifetime. Suh a step moves the state mahine from one semanti state to another semanti state while reeiving events from and emitting events to the environment. A semanti state (alled a status) omprises of three omponents: a onfiguration (a set of ative states), an event queue, and the variable assignments. We depit the omponents of a status in double square brakets [[, q, d]] and a semanti step as follows: [[,q,d]] in,out [[,q,d ]] (4) Please note that the hosen set of firing transitions and the exeution order of these transitions an be identified (if neessary) from this representation. Assuming a state mahine to be input enabled (f. the setion) a semanti step an be desribed as follows. We have to distinguish two situations. First, the situation when the event store does not ontain any events: q =<> q = (q,e in ) [[,q,d]] E in, [[,q,d]] During the step, only the events reeived from the environment (E in ) are added to the event store ( (q,e in )). The ative onfiguration and the data assignments are left unhanged. Seond, the situation when the event store ontains events for proessing: q ran (q,e) = (q) = ( \ t:t exits(t)) t:t enters(t) A seq perm({t : T effet(label(t)(e))}) (d,e gen ) = performall( /A seq )(d) (E int = E gen E SM ) (E out = E gen E env ) q = (q E int ) E in [[,q,d]] E in,e out [[,q,d ]] During the step, the trigger event will be seleted from the event store ( (q)). The onfiguration results from leaving all states the transitions exit, and entering all states the transitions enter. Next, an exeution order for the firing transition set is hosen (perm), and the effet of this transition sequene is alulated (5) (6)

5 (performall). The effet inludes the new data assignments (d ) and the sequene of newly generated events (E gen ). Finally, this event sequene is proessed. The generated internal events (E int ) and the events reeived from the environment (E in ) are added to the event store. The remaining external events (E out ) are sent to the environment. Now we an desribe the exeution of a state mahine based on this definitions as a onatenation of semanti steps. We all suh a sequene of semanti steps a omputation: [[ 1,q 1,d 1 ]] in 1,out 1 [[2,q 2,d 2 ]] in 2,out 2 in n 1,out n 1... [[ n,q n,d n ]] All formal definitions of the used state mahine semantis an be found in [18]. 3. TEST CASE GENERATION We use the formal exeution model from the previous setion as the bases for the definition of our automated test approah. Only suh mathematial preise models with a lear interpretation offers the basis for automated proesses. To define suh a proess we need to fix some last open points. In the UML (and in our semantis, too) not all semanti details are fixed. Suh points are alled semanti variation points. Semanti variation points have been introdued to avoid unneessary restritions on semanti details. Instead, there should be some spae for different realizations. 2 A user of the semantis has to instantiate these variation points before working with the semantis. For our test approah the most interesting semantis variation points are: the nature of the event store, events not enabling any transition, the seletion poliy of possible firing transition sets, and the exeution order of the transitions in a hosen set. For our test approah we have to instantiate the first two semanti variation points. We do not instantiate the latter two but leave them uninstantiated. Thus the test approah works orretly for different implementations of a state mahine speifiation. Preisely, we neither want to restrit how to hoose a possible set of firing transitions (if there is more than one) nor do we want to restrit the order these transitions will be exeuted. This is different for the event store. In order to be able to alulate the possible orret behavior allowed by the state mahine speifiation, we need to know the nature of the event store, or with other words, we have to deide for a speifi nature. In most pratial ontexts a FIFO queue is used to store events for further proessing. Hene we assume an unbounded reliable FIFO queue as event store. Seond, we assume that events that do not enable a transition when they are proessed are just deleted and the event from the event store will be proessed. This implies that the state mahines do not blok. Tehnially they are alled input enabled. In summary, the result of the disussion about semanti variation points is twofold: first, an event queue and events to be omitted are introdued into the semanti model of state mahines. Seond, we need to respet different firing transition set seletions and exeution strategies in a test approah. 3.1 Conformane Relation for State Mahines Before we desribe how to generate test ases based on our semantis definitions we desribe the general test setting. As mentioned 2 Note that many problems with the UML semantis arise from that point. On the one hand some of these points are not obvious in the semantis and on the other hand deisions taken by the users of the semantis are often not propagated to the outside. seqe in PCO IN Environment Test Envirionment State Mahine SM PCO OUT seqe out Figure 2: Abstrat Test Arhiteture for Embedded Systems. in the introdution, an embedded system omprises of hardware and software omponents. Thus we have to treat the system under test (sut) as a blak box. We only require the sut to have so-alled points of ontrol and observations (po). Thus it is possible to ontrol the sut from the outside, i. e. to send inputs, and to observe the outputs of the sut. Figure 2 shows the abstrat test arhiteture. As a onsequene of this arhiteture only the inputs to the sut and the outputs of the sut are visible in the environment and thus for the tester. This partiularly implies that the event queue is not visible from the outside. Thus we need to restrit the test proess to the observable parts of a system under test and must respet internal details, whih influene the possible behavior. To generate test ases for a blak box sut from a state mahine speifiation, we need to extrat the observable parts of the omputations we defined for the semanti model of state mahines. These are the events reeived from the environment and the generated events sent to the environment. Corresponding to the omputation defined above we yield an observable omputation by extrating and onatenating these events: in 1 out1... inn 1 outn 1 (7) An observable omputation is a sequene of two types of events. Preisely, the sequenes of reeived events and the sequenes of generated events are onatenated. 3 The set of all observable omputations form our observable exeution model of state mahines. A prerequisite to evaluate automatially whether a sut onforms to a speifiation is a formal definition of onformane. To define onformane, we use the notion of implementation relations. De Niola and Hennessy studied various possible haraterizations of onformane [9, 8]. Brinksma and Tretmans studied various implementation relations for synhronous transition systems [4, 22]. In general, relevant implementation relations are based on the same idea of an external observer. In this idea an implementation I onforms to its speifiation S, if and only if all observations obs any external observer o an make on the implementation an be related to the observations this observer an make on the speifiation: I o S o : O obs(i,o) obs(s,o) (8) To get an appliable relation you need to define the type of observers (O), whih observations these observers an make (obs), and how to relate these observations ( ). In our test approah we use sequenes of inputs to the system under test as observers. The observations these observers an make are the resulting outputs, 3 We assume the event store to be a queue so that reeived events will be stored one after another in sequene. Furthermore, transitions and ations on transitions are exeuted in sequene, whereat generated events are stored in a sequene.

6 i. e. the generated events, of the system under test. The relation we use to ompare observations of the system under test with the observations of the speifiation is set inlusion ( ). Thus we an argue that a system under test onforms to its speifiation if and only if the output sequenes for all possible input sequenes are inluded in the set of all output sequenes of the speifiation for the same input sequene: I out S σ : seqe S out(i,σ) out(s,σ) (9) Following the idea of Tretmans [22] we restrit the set of possible inputs to that of the speifiation. The set of outputs we alulate from the set of observable omputations of a speifiation: out(s,σ) == {δ : otraes(s) σ = δ E in δ E out } (10) Preisely, the set of all observations out(s,σ) for S with input sequene σ results from all observable omputations of S (otraes(s)) for whih σ denotes the input sequene (σ = δ E S ) and δ E env denotes the resulting output sequene. Now we have a preise meaning of onformane and a guideline how to ompute test ases. Based on the speifiation we need to alulate the traes of the state mahine for all possible inputs and extrat the possible orret observations. For testing the sut we need to stimulate the system under test with the partiular inputs, observe the outputs and ompare them to the pre-alulated possible orret observations. That means to hek for their existene. Obviously a problem arrises when thinking about pratial testing: the set of inputs is infinitely large 4 or pretty huge. 3.2 Seleting Inputs for Test Case Generation When testing in pratie only we are interested in relevant and interesting test ases to advantage the quality assurane proess, and to use time and omputation power at an optimum. Therefore, we generate a test ase for a prior seleted input sequene. This twostep proess learly separates the input seletion problem from the test ase generation problem. Thus it is possible to use different seletion strategies with the same generation proess and it allows to adapt the input seletion proess to different test aims or to different projet stages. In the TEAGER Tool Suite we implemented several input seletion strategies. The strategies range from using given fixed input sequenes to using speifi models desribing the environment. The former allows so alled speial value testing and is used for very speifi test aims like the overage of a ertain path or state. The latter allows to model varied behavior of an environment. We use probabilities for inputs to model different environments. The most general one is an environment in whih all inputs an happen at any time with the same probability (uniform distribution). In a more speifi environment different probabilities are assigned to the inputs (a prior distribution). Thus the ourrene of speifi inputs an be influened. We also use a variant of this strategy where we adapt the probabilities one an input is hosen (dependant distribution). For every input a weight is assigned and deremented if the input is seleted. If all weights are equal to zero the initial assignments will be used. With this strategy we ensure that eventually every event is hosen. The most expressive way to desribe the environment is to model it with probabilisti state mahines. Using state mahines allows to model dependenies among inputs in 4 If we think of embedded systems as non-terminating systems. a a 01 b b b a b b Figure 3: Stepwise State Spae Exploration for [a,b,]. a sequene. It also allows to ompletely reassign input probabilities depending on the assumed state of the system under test. For example, the probability of dialing a number before lifting the reeiver of a telephone is ertainly different from the probability of dialing after lifting the reeiver. In summary, we use different omplex strategies to desribe assumed environments to selet relevant and interesting inputs. 3.3 Test Case Generation Algorithm With the deision to onsider a finite set of finite sequenes of inputs we an alulate all possible orret observations for these inputs. We use this information to be able to automatially evaluate the test exeution. Considering omplex data during the test ase generation proess is not sope of the present paper and we skip the orresponding details here. In the urrent implementation data are hosen randomly while generating test ases. The problem of test ases with data and whih speifi data to hoose is part of ongoing researh. To alulate the possible orret observations we stepwise explore the state mahine s state spae for the given input. The hallenge here is to orretly onsider all semanti subtleties. We do this in a two step algorithm: First, we initialize the state mahine with its initial status, i. e. with its initial onfiguration and an empty queue. Then we insert the first input event to the event queue. Now we apply a semanti step to this onfiguration: first, we alulate all possible firing transitions sets. For every transition set and every possible exeution order of the transitions inside these sets we alulate the resulting status. It is important to note that we alulate a fix-point for this set. That means, that no new status an be reahed from any alulated status. Thus we yield a set of all reahable status inluding all intermediate status for the first event. To store the intermediate status is important for handling possible interleavings of input and internally generated events. Seond, we insert the event to every reahable status in the previously alulated set. By doing so we respet possible interleavings of events in the event queue. Then we again alulate all reahable status for this input and proeed in the same way for the other inputs. We alulate the graph of all exeution paths whih inludes the reahable status. Figure 3 and 4 show suh graphs. Only this stepwise alulation of all reahable status ensures that all possible exeution paths for the given input are alulated. This inludes all non-determinism in the speifiation (modeled and arising from the semanti model of state mahines) and effets from proessing events asynhronously. Example. Let us assume an internal event i. Proessing this event from the queue [a] will produe a new internal event j. Event i will be generated in response of input event a. For the step we want to proess the input sequene a b. The problem is, that while testing we annot observe the queue of the system under test. So we do not know how event b will interleave with the internal a 01 b

7 [...] [x,y,z]... [...] [...] [x,v,w] [u,v,w] i x u v y v j w z w k pass inonlusive [...]... [i,j,k] [...] Figure 4: Hull with alulated sequenes of observations. events. So we first insert event a into the queue and alulate that there are three reahable status with different queues: [i], [j], []. The first queue results from just proessing a. The seond results from proessing a and i. The third results from proessing a, i and j. By inserting input b into all the queues we prepare for respeting all possible interleavings. The resulting queues are [i,b], [j,b], [b] and during the step [b,j] whih properly respets one possible interleaving. Due to inserting the event to all reahed status, event b will also be inserted to the queue [a]. This results in the queue [a,b] refleting the situation that the environment triggered both events before the system under test proessed the first one. Figure 3 shows the priniple of the stepwise state spae exploration for the input sequene [a,b,]. After proessing all events from the input sequene we an identify among the set of all reahed status those status whih are finally reahed by proessing the omplete input sequene. These nodes on the hull of the exeution graph are so-alled quiesent. That means that their event queue is empty and thus they annot proeed without a new input from the environment. Figure 4 shows an exeution graph with the status on the hull. We now extrat from these status the observations, whih would be emitted when exeuting a partiular path. These observations are the events whih the state mahine sends to the environment. All observation sequenes omprise the possible orret observations we an make when triggering the system under test with the input sequene. Our idea is now to treat all observations as an alphabet for a language. The alulated observation sequenes form aepted words of these language ausing the test exeution to pass. All other sequenes ause the test exeution to fail. We now just need to build an aeptor for the alulated observation sequene and use them as our test orale to automatially evaluate the test exeution. Before we an do that we need to solve one problem whih an arise when alulating the observation sequenes. We argued that we alulate a fix-point for the set of reahable status. Due to the fat that the state mahine an generate (internal) events and produe internal infinite loops the alulation of the fix-point does not terminate in any ase. 5 Figure 4 shows suh a situation in the lower left orner. To solve the problem we limit the number of steps to an upper bound. Tehnially, every reahed status has got a ounter 5 Here we subsume the problem that the time to alulate the fixpoint is unaeptable high. Figure 5: Aeptane graph with an inonlusive test verdit. for the number of steps neessary to reah this status. If a ounter reahes a speified upper bound we mark this status and abort further proessing of this status. As a onsequene we alulate two types of observation sequenes. One whih ould be alulated within the given bound, and one whih ould not. The latter type ould be interpreted as follows: all observations made so far are orret, but not all observations ould be alulated. Hene, after proessing all observations, we have no further information to ompare the output of the system under test. We an neither say that further observations are orret nor an we say that they are not orret. We only an stop testing the system under test with this input sequene and give an inonlusive test verdit. This verdit says that all observations so far are orret but that we stopped proessing the urrent exeution path further. It would also be possible to deide for a pass or a fail verdit. But introduing a third verdit allows a finer distintion of differently aused test exeutions results. As a onsequene we have to distinguish the two sets of possible observation sequenes. The aeptane graph we build out of these sets omprises two aepting nodes. One for all observation sequenes whih ould ompletely be generated and one for all observation sequenes whih were bounded. The aeptor itself is a deterministi finite automaton aepting both sets of observation sequenes. A test ase exeution finishing in one of these nodes results in a pass or an inonlusive verdit. All observations not overed by the aeptane graph result in a fail verdit. Figure 5 shows an aeptane graph for the observations of Figure 4. Algorithm 1 shows the ontrol struture of the test ase generation algorithm. The loop will be exeuted as often as inputs should be sent to the system under test in the test ase. The inner while-loop ontrols the fix-point alulation of reahable status. While there are newly generated status the simulation step is suessively repeated to alulate all reahable status. If there are no newly generated status the algorithm proeeds with the input event. The results of the loop are a set of all ompletely alulated observation sequenes and a set of all inompletely alulated observation sequenes. Out of these sets an aeptane graph will be alulated. Algorithm 2 shows the alulation of the suessive status for the alulated status in the previous step. First, the state mahine is initialized with the onfiguration from the status and the trigger event is seleted from the orresponding event queue. Then, all possible firing transitions sets and all possible transition exeution orders are exeuted to estimate the resulting status and the generated events. This inludes: saving reahed onfiguration, adding internal events to the input queue, and saving generated events whih should be sent to the environment. The latter events are the possible orret observations whih we use to build the aeptane graphs.

8 input : state mahine: sm output: an aeptane graph sm.onfiguration initial onfiguration result initial simulation node inonlusives while trigger < input length do trigger generate a new trigger store forall node result do node.queue trigger store {node} steps 0 while result steps < limit do temp simulationstep(result) steps steps + 1, result forall node temp do if steps = limit then inonlusives {node } else store {node} result {node} result store; generateaeptanegraph(result,inonlusives) Algorithm 1: Test Case Generation: Control Struture. input : set of simulation nodes: input output: set of new generated simulation nodes: result result forall node input do if node.queue <> then sm.onfiguration node.onfiguration event node.dequeue forall T : sm.getfts(event) do permutations permute(t ) forall firing_transitions: permutations do effets [] forall t: firing_transitions do effets fire(t) temp node temp.onfiguration sm.onfiguration forall effet: effets do forall ev: effet do if ev / E SM then temp.observation ev else temp.queue ev; return result result {temp } sm.onfiguration node.onfiguration Algorithm 2: Test Case Generation: Simulation Step. Both: the suessive status and the generated events will be stored in a new simulation node. The set of all new simulation nodes will be returned as the result of the simulationstep. A test ase omprises of the input sequene to stimulate the system under test and an aeptane graph to automatially evaluate the exeution of this test ase. The length of a test ase and the number of test ases an be influened by the seletion poliy of input sequenes as explained above. The generated test suite is sound. That means that no orret systems under test will be rejeted due to a test ase. Instead, the test verdit fail will only be assigned if the observation of the system under test annot be explained by the possible orret observations of the speifiation (see the onformane relation for state mahines). This is true beause we alulate all possible exeution paths to generate the sets of possible orret observations. With unlimited omputation power and time the presented algorithm is able to ompute a omplete test suite, whih is apable to exatly differentiate between orret and inorret implementations. The presented algorithm has exponential omplexity. The exponential omplexity arises from the branh fator introdued by the different sets of firing transitions, the different possible exeution orders of transitions, and the neessity to onsider possible interleavings in the event queue. Thus the effort to alulate a test ase grows with the length of the input sequene and indiretly by the number of internally generated events (f( x)). The branh fator is bounded by the finite number of transitions and the finite number of events (). Thus we an approximate the effort A to generate a test ase for a given input sequene of length x as follows: A(x) e (x+f( x)) (11) The exponential effort is visualized in Figure 6 by the doubly dotted urve. 3.4 Combining Test Sequenes When testing non-terminating embedded systems it is also interesting to exeute longer input sequenes. To redue non-determinism in the speifiation is not possible without any further knowledge about the system under test. Thus we onentrate on the asynhronous event proessing. The lion s share of the alulation effort results from respeting all interleavings of the input sequene with internal generated events. Now we an argue that it is not neessary to onsider all of these interleavings. For example, in pratie it is the ase that the system under test immediately starts to proess the first reeived input. It usually does not wait until "ten" events are reeived from the environment. With the distane of these events the probability falls that an internally generated event (as a onsequene of proessing, for example, the first input) interleaves with the tenth generated event. Based on this idea we developed various strategies to redue the alulation effort. To demonstrate the ore idea we implemented a strategy where we introdue so alled observation points. Observation points are points in time where we give the system under test enough time to alulate its reation. Compared to our semanti model of state mahines the system under test reahes a status in whih the event queue must be empty. So no more reation an be produed for the given input. This is the same situation as explained above for the general algorithm to alulate the possible orret observations all the status on the hull are quiesent. Continuing after suh an observation point now means: to enqueue the input to all (non-inonlusive) status on the hull of the previously alulated exeution graph (note that for these status the

9 Computation Time shows the general struture of a ombined test ase. When reahing a pass node in an aeptane graph for an input sequene we an ontinue to trigger the system under test with the input sequene and hek the newly generated output of the system under test at the observation point. First experiments with this stati strategy showed that if we an introdue suh observation points for the system under test this strategy works quite well. But further researh and experiments are needed to investigate more elaborate (dynami) strategies. We espeially think of using knowledge about the system under test like speifi properties of the used buffer to store events, introdue probabilisti strategies to handle possible interleavings or to observe the memory onsumption of the system. Length of the Input Sequene Figure 6: Linearization of the exponential Complexity. event queue is empty). We also reset the olleted possible observations. We an do so beause at an observation point we assume the system under test to have ompletely alulated its reations. These reations will be heked by the last proeeding aeptane graph. 6 Now we an proeed to alulate the possible orret observation sequenes for the omplete input sequene. The redution in the omputation effort results from the fat that we do not onsider possible interleavings resulting from events in the first input sequene with events in the seond input sequene. Figure 6 visualizes this effet. We now repeatedly alulate only the first part of the exponential urve. The overall alulation effort follows from adding the efforts needed to alulate the observations for the individual input sequenes. The average effort has a linear gradient depited by the dotted line. Compared to the effort for proessing one input sequene with the length of the sum of all sub-sequenes this is an enormous redution in the alulation effort. Now the effort for ombined test sequenes still grows exponentially with the length n of the partiular input sequenes but linear with the number x/n of ombined sequenes and onsequently with the length x of the overall input sequene: A omb (n,x) e (n+f(ñ)) x (12) n The onsequene of this redution is that now the possible orret behavior is over-approximated. We do not alulate all possible observation sequenes for the omposed input sequene. Instead, we approximate them in the way that we treat more behavior to be orret, i. e. more observation sequenes to be orret. Therefore, this kind of a test ase is weaker beause it is not able to detet all errors a test ase with only one input sequene would detet. But the test ase is still sound. We do not rejet orret systems under test with this strategy. The over-approximation follows from the fat that observation sequenes from different aeptane graphs an be ombined in any possible order. This would not be possible for a omplete input sequene. Depending on the used testing strategy we an now parameterize how test ases should be generated and ombined. On the one hand by the effort we need to proess the total ount of inputs, and on the other hand by the redution apability when splitting the input sequene into smaller parts. Figure 7 6 An improvement of this strategy would be to ollet possible orret observations for more than one observation point. 3.5 Evaluating the Test Proess When a test suite is generated with the algorithm above and a system under test is tested with this test suite we would like to know how extensively we tested the system under test. The number of test ases and the length of the input sequenes in the test ases only onditionally allow to draw onlusions related to that question. Still today the question is hard to answer. The mostly used approah is to measure the overage of different (strutural) elements of the system under test or the speifiation. For general ode this is ommon pratie. The used riteria are usually based on ontrol flow or data flow information in the ode or on funtional desription in the speifiation. With our test approah we address embedded systems omposed of hardware and software omponents. You an apply well known tehniques to measure overage in the software omponents, but our impression is that this is not suffiient for suh systems. To measure overage in the hardware omponents is usually not possible. The only way to regard the whole system is to use the speifiation. Thus we need to develop meaningful riteria for state mahines. Our urrent work introdues different riteria based on strutural elements of state mahines, like states and transitions, and on semanti elements, like onfigurations and sets of firing transitions. First results show that espeially semanti riteria are able to evaluate the behavior in a meaningful manner. For the future we partiularly address suh semanti riteria based on the semanti model of state mahines and their relation to seleted input sequenes. A still open and interesting question is whether it is possible to use those riteria to ontrol the test ase generation proess, viz. to measure overage while generating test ases and to selet the inputs aording to this overage. 4. TOOL SUPPORT To evaluate and to show the pratiability of our approah we implemented the TEAGER Tool Suite. Figure 8 shows the general arhiteture of the TEAGER Tool Suite. TEAGER onsists of an environment to automatially generate and exeute test ases, and additionally of an environment to exeute state mahine speifiations. The latter we use to analyze the exeution behavior and the testability of a state mahine, and to measure overage on a state mahine speifiation to evaluate generated test suites. The Test Case Generation and Driver omponent ontains the Test Case Generator and the Test Driver. The Test Case Generator we use to automatially generate test ases out of a state mahine speifiation. For seleted inputs a loaded state mahine speifiation will be exeuted step by step to ompute the possible orret observation sequenes. Based on them an aeptane graph as the test orale is generated. Input sequenes and aeptane graphs will be stored for eah test ase in separate files for later exeu-

10 obs2 obs1 input1 input2 inputn... obs1 obs2 Pass obs3 obs3 obs1 Inonlusive Pass Figure 7: General Struture of a ombined Test Case. tion. The Test Driver in turn loads saved test ases and exeutes them. The exeution inludes both: stimulating the system under test and omparing the observation to the omputed possible orret behavior in the aeptane graphs. The ommuniation with the system under test takes plae over a soket onnetion using pre-implemented adaptors. This onept offers a flexible way to onnet the system under test. It also offers the possibility to use our State Mahine Exeutor as a system under test stub. Thus we an analyze the exeution behavior of state mahine speifiation or measure the overage of a used speifiation. The omplete test ase generation proess is parameterized to have maximal ontrol over the struture of test ases and the effort needed to alulate them. First you an speify the number of test ases to be generated, the length of input (sub-)sequenes and the number input sequenes to be ombined in a test ase. Then you an speify the way input sequenes are generated. As an example, we implemented different probabilisti strategies whih desribe possible environments (f. Setion 3.2). It is also possible to speify the sequene of events as a preamble to generate test ases related to speifi parts of the speifiation. After onfiguring all parameters the test ase generation works ompletely without any user interation. For test ase exeution you an ontrol the frequene at whih inputs are sent to the system under test. To avoid a fixed timing we use a Gaussian distributed trigger rate with a mean value and deviation to be speified by the tester. The tester also speifies the number a test ase should be repeated, and the poliy how several exeution results should be ombined. Exeuting a test ase several times is espeially neessary when dealing with non-deterministi systems. Every exeution an ause the system to exeute a different path for the same input. However, we need to hek all resulting observations. How often test ases should be exeuted when dealing with different non-determinism annot be fixed in advane. Thus the given number is a so-alled test hypothesis. The term hypothesis expresses that we assume the number high enough to test the system under test adequately. The timeout value whih an be speified is also an test hypothesis. This upper time bound speifies how long the test driver should wait for a desired observation. Usually, this time bound is higher than the reation of the system. So the system under test has enough time to produe the reation for an input. The timeout value diretly influenes the test ase exeution behavior sine we use it to implement our observation points. Up to this bound all reations an be observed and (as important as the previous fat) no other reations an be observed. A value too short would ause unneessary false negatives; a value too high would unneessarily slow down the test exeution. For the ombination of different exeution results of the same test ase different strategies are imaginable. Atually we use three different strategies: MUST requires every test exeution to pass. STRONG_MAY requires at least one test exeution to pass, whereat the test exeution will be repeated (up to the number of test repetitions) until the first test ase passes. WEAK_MAY requires that no test exeution fails. The State Mahine Exeutor exeutes a state mahine and thus allows an exploration of misellaneous properties like the testability or the overage of a speifiation whih we use to evaluate the quality of a generated test suite. Here it is partiularly important to avoid a fixed exeution timing. For UML state mahines the soalled zero time assumption does not hold. Instead, it is assumed that exeuting a transition onsumes time. To respet this and espeially to be able to investigate effets of the used asynhronous ommuniation we also use a probability based sheme for the exeution times of transitions. The tester an speify the mean value and the deviation for a Gaussian distribution whih is used to selet an exeution time of every transition exeution. Thus effets of different timings an be tested. First experiments with the TEAGER showed, that we an meet the state explosion problems introdued through the semantis of state mahines with our approximation strategy. The generation and exeution proess is parameterized. This allows the appliation of different testing strategies and to have maximal ontrol over the omplete proess. For more information about the TEAGER Tool Suite, its individual omponents, and the used parameters, we refer the interested reader to our web site [19]. 5. SUMMARY AND OUTLOOK Testing benefits from the fat that the real system is brought to exeution. Thus, the interation of the real hardware and the real software an be evaluated. It aims in falsifiation, i. e. to show inonsistenies between the speifiation and developed system. Testing is appliable at different levels of abstration and at different stages of the development. With our approah UML state mahines an be used in the quality assurane to serve as a speifiation for the desired reative behavior of the system. It is possible to selet relevant and interesting inputs for a test ase and to alulate the possible orret observations for given inputs. They allow to automatially evaluate test exeutions whih is in general a diffiult and time onsuming task. Applied approximation makes the generation proess pratial, whereat it is possible to ontrol this proess depending on the time and omputation power to invest. Espeially the modularization of the different task in automatially generating test ases makes the approah interesting for further researh. All disussed strategies are implemented as modules of the tool suite. Thus, different strategies for seleting inputs, for om-