Test Case Generation from UML State Machines

Transcription

1 Test Case Generation from UML State Mahines Dirk Seifert Loria Université Nany 2 Campus Sientifique, BP 239 F Vandoeuvre lès Nany edex Dirk.Seifert@Loria.fr inria , version 2-23 Apr 2008 ABSTRACT In this paper we desrie a omprehensive approah for onformane testing of emedded reative systems. Based on a formal speifiation, namely UML state mahines, we automatially generate test ases and use them to hek the funtional onformane of a system under test. Our test ases inlude not only stimuli to trigger the system under test, they also inlude possile orret oservations to automatially evaluate the test ase exeution. In ontrast to lassial Harel Stateharts, state mahines ehave asynhronously, whih makes automati test ase generation a hallenge. The TEAGER Tool Suite implements the automati generation, exeution and evaluation of test ases and proves the appliaility of our test approah. 1. INTRODUCTION The impat of emedded systems in our everyday life is steadily growing. They are present not only in very speifi ontexts ut also in nearly every eletrial devie we use. In general, emedded systems omprises of hardware and software omponents interating with a speialized tehnial environment via sensors and ators. The main reason for their suess is the omination of speifi or high-performane hardware with the flexiility of software. The software is responsile for ontrolling the hardware and software omponents and for alulating reations as responses to reeived events. It is remarkale that users unonditionally trust in the orret funtioning of suh systems. This is true not only for safety ritial systems like an anti-lok rake system in a ar ut also for omparatively simpler systems like a ellular phone. The development should satisfy this onfidene. Erroneous systems annoy the ostumers and are a high ommerial risk in mass ustomization. Moreover, size and omplexity of nowadays systems whih have to e developed, demand for improved and automated proesses: for development as well as for quality assurane. Model-ased software development ases on setting up models of the system to e onstruted. This approah has proved to e use- This work has partly een done in the researh group Softwaretehnik at the Tehnishe Universität Berlin. ful, eause it allows developers to first elaorate the most important properties of the software efore proeeding with the implementation. Nowadays the Unified Modeling Language (UML) [24] is widely used to model systems and to guide development proesses. The UML omprises of several diagram types to speify the struture and the ehavior of a system or system omponent. State mahines are used to either desrie the disrete reative ehavior (ehavioral state mahines) or to desrie the usage protool (protool state mahines). In this paper we refer to ehavioral state mahines. We use them to speify the states a system an take and ations it an exeute during its lifetime in response to external and internal events. Due to the disrete reative harater of state mahines and the possiility to ompletely desrie the disrete ehavior of a system, state mahines are appropriate to model emedded systems. We intend to use state mahine models not only for development ut also to support an automated quality assurane proess. In Setion 2 we introdue the syntax and semantis of state mahines we need in this paper y means of an example. In Setion 3 we present our test approah. In this approah we automatially generate test ases out of a state mahine speifiation. The test ases inlude not only the stimuli to trigger the system under test, they also inlude the possile orret reations. The latter allows an automati evaluation of the test ase exeution. We desrie the underlying theory, the developed test ase generation algorithm, how approximation tehniques are used to develop a pratial approah and how the test ase generation and exeution an e ontrolled and evaluated. In Setion 4 we desrie the TEAGER Tool Suite whih ompletely implements the introdued approah. Finally, in Setion 5 we onlude our work, disuss related work and give an outlook to ongoing and future researh. 2. STATE MACHINES UML state mahines [24] are an ojet-oriented extension of the lassial Harel-Stateharts [15]. In this paper we use ehavioral state mahines to desrie the sequene of states a system or system omponent an take and the ations it exeutes when hanging these states. State mahines are mathematial models with a graphial representation: the nodes depit simple or omposed states of the system and the laeled edges depit transitions etween these states. Composite states allow to hierarhially and orthogonally struture the model, thus reduing the graphial omplexity. Laels express onditions under whih transitions an e taken and the ations whih will e exeuted when the transition is taken. Events are used as triggers to ativate transitions and an e parameterized to exhange data. Optional, every state mahine has a data spae whih an e read and manipulated y the state mahine during

2 oolean incdfull = false; oolean intapefull = false; integer trakcount; Car Audio System Audio Player power Off power CD Player On d_insert / incdfull = true; trakcount = d_insert.1 ak Tuner Mode ak P1 P2 tape_ejet [ not in("cd Full") ] / tuner_plays sr [ not in("cd Full") ] / tuner_plays CD Empty CD Full sr [ in("tape Full") ] / tape_plays d_ejet / incdfull = false; ak P4 P3 ak Tape Mode d_ejet play sr [ in("cd Full") ] / d_plays CD Mode CD Playing play sr [ not in("tape Full") ] / tuner_plays ak Bakward Spooling Tape Playing ak sr [ in("tape Full") ] / tape_plays ak Forward Spooling Tape Player Tape Empty tape_insert / intapefull = true; tape_ejet / in TapeFull = false; Tape Full inria , version 2-23 Apr 2008 Next Trak Former Trak exeution. More preisely, it is possile to read the data to desrie speifi onditions when a transition an e taken or to manipulate the data and exhange information within the ations. The general struture of a transition onsists of a soure state, a trigger event, an optional guard in square rakets, an optional ation sequene separated from the previous elements y a slash, and a target state. With the optional guard a fine-grained ondition to enale the transition an e desried depending on the system s state. Hene the ativation of the soure state, the trigger event and the fulfilled guard ondition onstitute the ondition whih must hold to enale the transition. An ation an either e a statement manipulating the data spae or the generation of new events. The ation sequene and the susequently ative target state onstitute the effet of the transition. In this paper we use a sustantial suset of state mahines to study automated test ase generation and evaluation ased on state mahines. We assume the reader to have some asi knowledge of transition systems. In the following we riefly desrie state mahines y means of an example. Afterwards we disuss semanti issues whih make automated test ase generation a hallenge. A omplete and detailed desription as well as a preise definition of the semantis (inluding the integration of omplex data) an e found in [18]. 2.1 Example To demonstrate the state mahine notation we use a state mahine model speifying the ehavior of a simple sound devie in a ar. Figure 1 shows this model. The requirements for suh sound devie ould e as follows: sr [ in ("CD Full") ] / d_plays tape_ejet [ in("cd Full") ] / d_plays Figure 1: State mahine speifiation for the Car Audio System. It should e possile to turn the Car Audio System on and off. When turned on, it should play one of three different audio soures, namely radio, tape or ompat dis, respeting the presene of a tape or a ompat dis. It should e possile to hange etween availale soures. Furthermore, it should e possile to swith etween four radio stations, to spool a tape akward or forward, or to selet the previous or the trak of a ompat dis. Astrating from any physial devies we introdue the following events to model the required ehavior: power, sr (to swith etween the different soures),, ak and play. Additionally we introdue events signaling the insertion and the ejetion of a tape or a ompat dis as well as events to signal system reations. Furthermore, we use data variales to store detailed information aout the urrent state. For example, we use an integer variale trakcount to store the numer of titles of an inserted ompat dis. Figure 1 shows a state mahine model of the sound devie inluding the underlying state spae. At the highest level of astration the model onsists of an orthogonal state omprising three regions. The two regions CD Player and Tape Player model the information if a tape or a ompat dis is inserted into the system or not. The more omplex region Audio Player models the ontrol of the system. The region is refined y two states: Off and On. Initially the system is assumed to e swithed off, expressed y the small arrow leaving a ullet and ending at the Off state. When the event power is proessed the system is swithed on and starts to play the radio (again expressed y a small arrow). The omposite state On is refined into states modeling the three signal soures. The transitions etween these states desrie the hanges etween the soures as reation

3 inria , version 2-23 Apr 2008 to an event sr. For example, when the system is in Tuner Mode and a tape and a ompat dis are inserted into the system (i. e. oth in-prediates are true) and the event sr is proessed, the system an either swith to the tape mode or swith to the ompat dis mode eause oth transitions are enaled and an fire. All three sustates of Audio Player are further refined to desrie the partiular ehavior in reation to the events, ak and play in eah state. 2.2 State Mahine Semantis The semantis of state mahines is adapted from the STATEMATE semantis [17, 16] to fit into the ojet-oriented paradigm. As desried aove a state mahine an e refined y simple omposite and orthogonal states. Simple omposite states ontain exatly one region and orthogonal states ontain at least two regions. In every region only one sustate an e ative at a time. The state whih is entered y default when a region is entered is marked y an arrow emanating from a filled irle. The hierarhial ordering of states forms a tree struture with a region as the root node, simple states at the leave nodes and in etween (alternating) omposite states and regions. Due to orthogonal regions a state mahine an e in several states at a time. We all the set of all ative states a onfiguration. For the same reason it is possile that more than one transition an fire at a time; one in every ative orthogonal region. We all the set of all jointly firing transitions at a time firing transition set (FTS in short). Due to the hierarhial struture of state mahines it an happen that two transitions are enaled for firing on different hierarhy levels of a state. Taking oth would lead to a onfiguration whih is not well-formed. A similar situation arises if a transition leaves an orthogonal region. In this ase the transition annot fire together with an enaled transition in another orthogonal region. In oth ases the transitions are said to e in onflit with eah other. Suh situations are identified if two transitions leave idential states in the state hierarhy. The UML desries a two step proess to resolve suh onflits. In the first step a priority sheme is used. Transitions emanating from a state deeper in the state hierarhy has priority over the other transition. Thus the more refined transition is taken. 1 Nevertheless, not all onflits an e resolved using this priority sheme. In the seond step only transitions are seleted whih are not in onflit to eah other respeting maximal progress of the system. A so-alled transition seletion algorithm selets all maximal sets T T of enaled transitions fulfilling the following requirements: t : T enaled(t,,e,d) (1) t 1,t 2 : T t 1 t 2 t 1 t 2 (2) t : T \ T enaled(t,,e,d) t : T t t t t (3) First, all transition in the firing transition set must e enaled regarding the urrent onfiguration, the trigger event and the urrent data assignments. Seond, all transitions in the set are mutually onflit free (expressed y the operator). Third, there is no enaled transition outside the set whih is onflit free with the transitions in the set or with higher priority than a transition inside the set. Thus, transitions with the highest priority are taken and maximal sets are hosen. Result of the transition seletion algorithm is a set of firing transition sets (PPT). 1 This differs from lassial Stateharts. But it reflets the ojetoriented inheritane ehavior. Eah set represents a valid firing transition set. It is important to mention that for exeution one suh set is aritrarily hosen, and that the order in whih the transitions are fired is aritrarily hosen, too. In onsequene, all set hoies and transition permutations form the set of all possile semanti steps of the state mahine at a time. This is important if we want to ompute the possile orret ehavior for an input sequene to evaluate the test exeution. In opposite to the lassial Stateharts, the event proessing takes plae in a so-alled run-to-ompletion step. This asynhronous event proessing demands the proessing of the previous event to e ompletely finished efore the event an e proessed. Therefore it is neessary to uffer reeived events in an event store. Consequently, the ourrene of an event and its proessing are asynhronous, i. e. take plae at different times. It follows immediately that a possile (oservale) reation of the system also takes plae asynhronously. The semanti model of state mahines uilds on the semanti steps a state mahine an exeute during its lifetime. Suh a step moves the state mahine from one semanti state to another semanti state while reeiving events from and emitting events to the environment. A semanti state (alled a status) omprises of three omponents: a onfiguration (a set of ative states), an event queue, and the variale assignments. We depit the omponents of a status in doule square rakets [[, q, d]] and a semanti step as follows: [[,q,d]] in,out [[,q,d ]] (4) Please note that the hosen set of firing transitions and the exeution order of these transitions an e identified (if neessary) from this representation. Assuming a state mahine to e input enaled (f. the setion) a semanti step an e desried as follows. We have to distinguish two situations. First, the situation when the event store does not ontain any events: q =<> q = (q,e in ) [[,q,d]] E in, [[,q,d]] During the step, only the events reeived from the environment (E in ) are added to the event store ( (q,e in )). The ative onfiguration and the data assignments are left unhanged. Seond, the situation when the event store ontains events for proessing: q ran (q,e) = (q) = ( \ t:t exits(t)) t:t enters(t) A seq perm({t : T effet(lael(t)(e))}) (d,e gen ) = performall( /A seq )(d) (E int = E gen E SM ) (E out = E gen E env ) q = (q E int ) E in [[,q,d]] E in,e out [[,q,d ]] During the step, the trigger event will e seleted from the event store ( (q)). The onfiguration results from leaving all states the transitions exit, and entering all states the transitions enter. Next, an exeution order for the firing transition set is hosen (perm), and the effet of this transition sequene is alulated (5) (6)

4 inria , version 2-23 Apr 2008 (performall). The effet inludes the new data assignments (d ) and the sequene of newly generated events (E gen ). Finally, this event sequene is proessed. The generated internal events (E int ) and the events reeived from the environment (E in ) are added to the event store. The remaining external events (E out ) are sent to the environment. Now we an desrie the exeution of a state mahine ased on this definitions as a onatenation of semanti steps. We all suh a sequene of semanti steps a omputation: [[ 1,q 1,d 1 ]] in 1,out 1 [[2,q 2,d 2 ]] in 2,out 2 in n 1,out n 1... [[ n,q n,d n ]] All formal definitions of the used state mahine semantis an e found in [18]. 3. TEST CASE GENERATION We use the formal exeution model from the previous setion as the ases for the definition of our automated test approah. Only suh mathematial preise models with a lear interpretation offers the asis for automated proesses. To define suh a proess we need to fix some last open points. In the UML (and in our semantis, too) not all semanti details are fixed. Suh points are alled semanti variation points. Semanti variation points have een introdued to avoid unneessary restritions on semanti details. Instead, there should e some spae for different realizations. 2 A user of the semantis has to instantiate these variation points efore working with the semantis. For our test approah the most interesting semantis variation points are: the nature of the event store, events not enaling any transition, the seletion poliy of possile firing transition sets, and the exeution order of the transitions in a hosen set. For our test approah we have to instantiate the first two semanti variation points. We do not instantiate the latter two ut leave them uninstantiated. Thus the test approah works orretly for different implementations of a state mahine speifiation. Preisely, we neither want to restrit how to hoose a possile set of firing transitions (if there is more than one) nor do we want to restrit the order these transitions will e exeuted. This is different for the event store. In order to e ale to alulate the possile orret ehavior allowed y the state mahine speifiation, we need to know the nature of the event store, or with other words, we have to deide for a speifi nature. In most pratial ontexts a FIFO queue is used to store events for further proessing. Hene we assume an unounded reliale FIFO queue as event store. Seond, we assume that events that do not enale a transition when they are proessed are just deleted and the event from the event store will e proessed. This implies that the state mahines do not lok. Tehnially they are alled input enaled. In summary, the result of the disussion aout semanti variation points is twofold: first, an event queue and events to e omitted are introdued into the semanti model of state mahines. Seond, we need to respet different firing transition set seletions and exeution strategies in a test approah. 3.1 Conformane Relation for State Mahines Before we desrie how to generate test ases ased on our semantis definitions we desrie the general test setting. As mentioned 2 Note that many prolems with the UML semantis arise from that point. On the one hand some of these points are not ovious in the semantis and on the other hand deisions taken y the users of the semantis are often not propagated to the outside. seqe in PCO IN Environment Test Envirionment State Mahine SM PCO OUT seqe out Figure 2: Astrat Test Arhiteture for Emedded Systems. in the introdution, an emedded system omprises of hardware and software omponents. Thus we have to treat the system under test (sut) as a lak ox. We only require the sut to have so-alled points of ontrol and oservations (po). Thus it is possile to ontrol the sut from the outside, i. e. to send inputs, and to oserve the outputs of the sut. Figure 2 shows the astrat test arhiteture. As a onsequene of this arhiteture only the inputs to the sut and the outputs of the sut are visile in the environment and thus for the tester. This partiularly implies that the event queue is not visile from the outside. Thus we need to restrit the test proess to the oservale parts of a system under test and must respet internal details, whih influene the possile ehavior. To generate test ases for a lak ox sut from a state mahine speifiation, we need to extrat the oservale parts of the omputations we defined for the semanti model of state mahines. These are the events reeived from the environment and the generated events sent to the environment. Corresponding to the omputation defined aove we yield an oservale omputation y extrating and onatenating these events: in 1 out1... inn 1 outn 1 (7) An oservale omputation is a sequene of two types of events. Preisely, the sequenes of reeived events and the sequenes of generated events are onatenated. 3 The set of all oservale omputations form our oservale exeution model of state mahines. A prerequisite to evaluate automatially whether a sut onforms to a speifiation is a formal definition of onformane. To define onformane, we use the notion of implementation relations. De Niola and Hennessy studied various possile haraterizations of onformane [9, 8]. Brinksma and Tretmans studied various implementation relations for synhronous transition systems [4, 22]. In general, relevant implementation relations are ased on the same idea of an external oserver. In this idea an implementation I onforms to its speifiation S, if and only if all oservations os any external oserver o an make on the implementation an e related to the oservations this oserver an make on the speifiation: I o S o : O os(i,o) os(s,o) (8) To get an appliale relation you need to define the type of oservers (O), whih oservations these oservers an make (os), and how to relate these oservations ( ). In our test approah we use sequenes of inputs to the system under test as oservers. The oservations these oservers an make are the resulting outputs, 3 We assume the event store to e a queue so that reeived events will e stored one after another in sequene. Furthermore, transitions and ations on transitions are exeuted in sequene, whereat generated events are stored in a sequene.

5 inria , version 2-23 Apr 2008 i. e. the generated events, of the system under test. The relation we use to ompare oservations of the system under test with the oservations of the speifiation is set inlusion ( ). Thus we an argue that a system under test onforms to its speifiation if and only if the output sequenes for all possile input sequenes are inluded in the set of all output sequenes of the speifiation for the same input sequene: I out S σ : seqe S out(i,σ) out(s,σ) (9) Following the idea of Tretmans [22] we restrit the set of possile inputs to that of the speifiation. The set of outputs we alulate from the set of oservale omputations of a speifiation: out(s,σ) == {δ : otraes(s) σ = δ E in δ E out } (10) Preisely, the set of all oservations out(s,σ) for S with input sequene σ results from all oservale omputations of S (otraes(s)) for whih σ denotes the input sequene (σ = δ E S ) and δ E env denotes the resulting output sequene. Now we have a preise meaning of onformane and a guideline how to ompute test ases. Based on the speifiation we need to alulate the traes of the state mahine for all possile inputs and extrat the possile orret oservations. For testing the sut we need to stimulate the system under test with the partiular inputs, oserve the outputs and ompare them to the pre-alulated possile orret oservations. That means to hek for their existene. Oviously a prolem arrises when thinking aout pratial testing: the set of inputs is infinitely large 4 or pretty huge. 3.2 Seleting Inputs for Test Case Generation When testing in pratie only we are interested in relevant and interesting test ases to advantage the quality assurane proess, and to use time and omputation power at an optimum. Therefore, we generate a test ase for a prior seleted input sequene. This twostep proess learly separates the input seletion prolem from the test ase generation prolem. Thus it is possile to use different seletion strategies with the same generation proess and it allows to adapt the input seletion proess to different test aims or to different projet stages. In the TEAGER Tool Suite we implemented several input seletion strategies. The strategies range from using given fixed input sequenes to using speifi models desriing the environment. The former allows so alled speial value testing and is used for very speifi test aims like the overage of a ertain path or state. The latter allows to model varied ehavior of an environment. We use proailities for inputs to model different environments. The most general one is an environment in whih all inputs an happen at any time with the same proaility (uniform distriution). In a more speifi environment different proailities are assigned to the inputs (a prior distriution). Thus the ourrene of speifi inputs an e influened. We also use a variant of this strategy where we adapt the proailities one an input is hosen (dependant distriution). For every input a weight is assigned and deremented if the input is seleted. If all weights are equal to zero the initial assignments will e used. With this strategy we ensure that eventually every event is hosen. The most expressive way to desrie the environment is to model it with proailisti state mahines. Using state mahines allows to model dependenies among inputs in 4 If we think of emedded systems as non-terminating systems. a a 01 a Figure 3: Stepwise State Spae Exploration for [a,,]. a sequene. It also allows to ompletely reassign input proailities depending on the assumed state of the system under test. For example, the proaility of dialing a numer efore lifting the reeiver of a telephone is ertainly different from the proaility of dialing after lifting the reeiver. In summary, we use different omplex strategies to desrie assumed environments to selet relevant and interesting inputs. 3.3 Test Case Generation Algorithm With the deision to onsider a finite set of finite sequenes of inputs we an alulate all possile orret oservations for these inputs. We use this information to e ale to automatially evaluate the test exeution. Considering omplex data during the test ase generation proess is not sope of the present paper and we skip the orresponding details here. In the urrent implementation data are hosen randomly while generating test ases. The prolem of test ases with data and whih speifi data to hoose is part of ongoing researh. To alulate the possile orret oservations we stepwise explore the state mahine s state spae for the given input. The hallenge here is to orretly onsider all semanti sutleties. We do this in a two step algorithm: First, we initialize the state mahine with its initial status, i. e. with its initial onfiguration and an empty queue. Then we insert the first input event to the event queue. Now we apply a semanti step to this onfiguration: first, we alulate all possile firing transitions sets. For every transition set and every possile exeution order of the transitions inside these sets we alulate the resulting status. It is important to note that we alulate a fix-point for this set. That means, that no new status an e reahed from any alulated status. Thus we yield a set of all reahale status inluding all intermediate status for the first event. To store the intermediate status is important for handling possile interleavings of input and internally generated events. Seond, we insert the event to every reahale status in the previously alulated set. By doing so we respet possile interleavings of events in the event queue. Then we again alulate all reahale status for this input and proeed in the same way for the other inputs. We alulate the graph of all exeution paths whih inludes the reahale status. Figure 3 and 4 show suh graphs. Only this stepwise alulation of all reahale status ensures that all possile exeution paths for the given input are alulated. This inludes all non-determinism in the speifiation (modeled and arising from the semanti model of state mahines) and effets from proessing events asynhronously. Example. Let us assume an internal event i. Proessing this event from the queue [a] will produe a new internal event j. Event i will e generated in response of input event a. For the step we want to proess the input sequene a. The prolem is, that while testing we annot oserve the queue of the system under test. So we do not know how event will interleave with the internal a 01

6 [...] [x,y,z]... [...] [...] [x,v,w] [u,v,w] i x u v y v j w z w k pass inonlusive inria , version 2-23 Apr 2008 [...]... [i,j,k] [...] Figure 4: Hull with alulated sequenes of oservations. events. So we first insert event a into the queue and alulate that there are three reahale status with different queues: [i], [j], []. The first queue results from just proessing a. The seond results from proessing a and i. The third results from proessing a, i and j. By inserting input into all the queues we prepare for respeting all possile interleavings. The resulting queues are [i,], [j,], [] and during the step [,j] whih properly respets one possile interleaving. Due to inserting the event to all reahed status, event will also e inserted to the queue [a]. This results in the queue [a,] refleting the situation that the environment triggered oth events efore the system under test proessed the first one. Figure 3 shows the priniple of the stepwise state spae exploration for the input sequene [a,,]. After proessing all events from the input sequene we an identify among the set of all reahed status those status whih are finally reahed y proessing the omplete input sequene. These nodes on the hull of the exeution graph are so-alled quiesent. That means that their event queue is empty and thus they annot proeed without a new input from the environment. Figure 4 shows an exeution graph with the status on the hull. We now extrat from these status the oservations, whih would e emitted when exeuting a partiular path. These oservations are the events whih the state mahine sends to the environment. All oservation sequenes omprise the possile orret oservations we an make when triggering the system under test with the input sequene. Our idea is now to treat all oservations as an alphaet for a language. The alulated oservation sequenes form aepted words of these language ausing the test exeution to pass. All other sequenes ause the test exeution to fail. We now just need to uild an aeptor for the alulated oservation sequene and use them as our test orale to automatially evaluate the test exeution. Before we an do that we need to solve one prolem whih an arise when alulating the oservation sequenes. We argued that we alulate a fix-point for the set of reahale status. Due to the fat that the state mahine an generate (internal) events and produe internal infinite loops the alulation of the fix-point does not terminate in any ase. 5 Figure 4 shows suh a situation in the lower left orner. To solve the prolem we limit the numer of steps to an upper ound. Tehnially, every reahed status has got a ounter 5 Here we susume the prolem that the time to alulate the fixpoint is unaeptale high. Figure 5: Aeptane graph with an inonlusive test verdit. for the numer of steps neessary to reah this status. If a ounter reahes a speified upper ound we mark this status and aort further proessing of this status. As a onsequene we alulate two types of oservation sequenes. One whih ould e alulated within the given ound, and one whih ould not. The latter type ould e interpreted as follows: all oservations made so far are orret, ut not all oservations ould e alulated. Hene, after proessing all oservations, we have no further information to ompare the output of the system under test. We an neither say that further oservations are orret nor an we say that they are not orret. We only an stop testing the system under test with this input sequene and give an inonlusive test verdit. This verdit says that all oservations so far are orret ut that we stopped proessing the urrent exeution path further. It would also e possile to deide for a pass or a fail verdit. But introduing a third verdit allows a finer distintion of differently aused test exeutions results. As a onsequene we have to distinguish the two sets of possile oservation sequenes. The aeptane graph we uild out of these sets omprises two aepting nodes. One for all oservation sequenes whih ould ompletely e generated and one for all oservation sequenes whih were ounded. The aeptor itself is a deterministi finite automaton aepting oth sets of oservation sequenes. A test ase exeution finishing in one of these nodes results in a pass or an inonlusive verdit. All oservations not overed y the aeptane graph result in a fail verdit. Figure 5 shows an aeptane graph for the oservations of Figure 4. Algorithm 1 shows the ontrol struture of the test ase generation algorithm. The loop will e exeuted as often as inputs should e sent to the system under test in the test ase. The inner while-loop ontrols the fix-point alulation of reahale status. While there are newly generated status the simulation step is suessively repeated to alulate all reahale status. If there are no newly generated status the algorithm proeeds with the input event. The results of the loop are a set of all ompletely alulated oservation sequenes and a set of all inompletely alulated oservation sequenes. Out of these sets an aeptane graph will e alulated. Algorithm 2 shows the alulation of the suessive status for the alulated status in the previous step. First, the state mahine is initialized with the onfiguration from the status and the trigger event is seleted from the orresponding event queue. Then, all possile firing transitions sets and all possile transition exeution orders are exeuted to estimate the resulting status and the generated events. This inludes: saving reahed onfiguration, adding internal events to the input queue, and saving generated events whih should e sent to the environment. The latter events are the possile orret oservations whih we use to uild the aeptane graphs.

7 inria , version 2-23 Apr 2008 input : state mahine: sm output: an aeptane graph sm.onfiguration initial onfiguration result initial simulation node inonlusives while trigger < input length do trigger generate a new trigger store forall node result do node.queue trigger store {node} steps 0 while result steps < limit do temp simulationstep(result) steps steps + 1, result forall node temp do if steps = limit then inonlusives {node } else store {node} result {node} result store; generateaeptanegraph(result,inonlusives) Algorithm 1: Test Case Generation: Control Struture. Both: the suessive status and the generated events will e stored in a new simulation node. The set of all new simulation nodes will e returned as the result of the simulationstep. A test ase omprises of the input sequene to stimulate the system under test and an aeptane graph to automatially evaluate the exeution of this test ase. The length of a test ase and the numer of test ases an e influened y the seletion poliy of input sequenes as explained aove. The generated test suite is sound. That means that no orret systems under test will e rejeted due to a test ase. Instead, the test verdit fail will only e assigned if the oservation of the system under test annot e explained y the possile orret oservations of the speifiation (see the onformane relation for state mahines). This is true eause we alulate all possile exeution paths to generate the sets of possile orret oservations. With unlimited omputation power and time the presented algorithm is ale to ompute a omplete test suite, whih is apale to exatly differentiate etween orret and inorret implementations. The presented algorithm has exponential omplexity. The exponential omplexity arises from the ranh fator introdued y the different sets of firing transitions, the different possile exeution orders of transitions, and the neessity to onsider possile interleavings in the event queue. Thus the effort to alulate a test ase grows with the length of the input sequene and indiretly y the numer of internally generated events (f( x)). The ranh fator is ounded y the finite numer of transitions and the finite numer of events (). Thus we an approximate the effort A to generate a test ase for a given input sequene of length x as follows: A(x) e (x+f( x)) (11) input : set of simulation nodes: input output: set of new generated simulation nodes: result result forall node input do if node.queue <> then sm.onfiguration node.onfiguration event node.dequeue forall T : sm.getfts(event) do permutations permute(t ) forall firing_transitions: permutations do effets [] forall t: firing_transitions do effets fire(t) temp node temp.onfiguration sm.onfiguration forall effet: effets do forall ev: effet do if ev / E SM then temp.oservation ev else temp.queue ev; return result result {temp } sm.onfiguration node.onfiguration Algorithm 2: Test Case Generation: Simulation Step. The exponential effort is visualized in Figure 6 y the douly dotted urve. 3.4 Comining Test Sequenes When testing non-terminating emedded systems it is also interesting to exeute longer input sequenes. To redue non-determinism in the speifiation is not possile without any further knowledge aout the system under test. Thus we onentrate on the asynhronous event proessing. The lion s share of the alulation effort results from respeting all interleavings of the input sequene with internal generated events. Now we an argue that it is not neessary to onsider all of these interleavings. For example, in pratie it is the ase that the system under test immediately starts to proess the first reeived input. It usually does not wait until "ten" events are reeived from the environment. With the distane of these events the proaility falls that an internally generated event (as a onsequene of proessing, for example, the first input) interleaves with the tenth generated event. Based on this idea we developed various strategies to redue the alulation effort. To demonstrate the ore idea we implemented a strategy where we introdue so alled oservation points. Oservation points are points in time where we give the system under test enough time to alulate its reation. Compared to our semanti model of state mahines the system under test reahes a status in whih the event queue must e empty. So no more reation an e produed for the given input. This is the same situation as explained aove for the general algorithm to alulate the possile orret oservations all the status on the hull are quiesent. Continuing after suh an oservation point now means: to enqueue the input to all (non-inonlusive) status on the hull of the previously alulated exeution graph (note that for these status the

8 Computation Time shows the general struture of a omined test ase. When reahing a pass node in an aeptane graph for an input sequene we an ontinue to trigger the system under test with the input sequene and hek the newly generated output of the system under test at the oservation point. First experiments with this stati strategy showed that if we an introdue suh oservation points for the system under test this strategy works quite well. But further researh and experiments are needed to investigate more elaorate (dynami) strategies. We espeially think of using knowledge aout the system under test like speifi properties of the used uffer to store events, introdue proailisti strategies to handle possile interleavings or to oserve the memory onsumption of the system. The Test Case Generation and Driver omponent ontains the Test Case Generator and the Test Driver. The Test Case Generator we use to automatially generate test ases out of a state mahine speifiation. For seleted inputs a loaded state mahine speifiation will e exeuted step y step to ompute the possile orret oservation sequenes. Based on them an aeptane graph as the test orale is generated. Input sequenes and aeptane graphs will e stored for eah test ase in separate files for later exeuinria , version 2-23 Apr 2008 Length of the Input Sequene Figure 6: Linearization of the exponential Complexity. event queue is empty). We also reset the olleted possile oservations. We an do so eause at an oservation point we assume the system under test to have ompletely alulated its reations. These reations will e heked y the last proeeding aeptane graph. 6 Now we an proeed to alulate the possile orret oservation sequenes for the omplete input sequene. The redution in the omputation effort results from the fat that we do not onsider possile interleavings resulting from events in the first input sequene with events in the seond input sequene. Figure 6 visualizes this effet. We now repeatedly alulate only the first part of the exponential urve. The overall alulation effort follows from adding the efforts needed to alulate the oservations for the individual input sequenes. The average effort has a linear gradient depited y the dotted line. Compared to the effort for proessing one input sequene with the length of the sum of all su-sequenes this is an enormous redution in the alulation effort. Now the effort for omined test sequenes still grows exponentially with the length n of the partiular input sequenes ut linear with the numer x/n of omined sequenes and onsequently with the length x of the overall input sequene: A om (n,x) e (n+f(ñ)) x (12) n The onsequene of this redution is that now the possile orret ehavior is over-approximated. We do not alulate all possile oservation sequenes for the omposed input sequene. Instead, we approximate them in the way that we treat more ehavior to e orret, i. e. more oservation sequenes to e orret. Therefore, this kind of a test ase is weaker eause it is not ale to detet all errors a test ase with only one input sequene would detet. But the test ase is still sound. We do not rejet orret systems under test with this strategy. The over-approximation follows from the fat that oservation sequenes from different aeptane graphs an e omined in any possile order. This would not e possile for a omplete input sequene. Depending on the used testing strategy we an now parameterize how test ases should e generated and omined. On the one hand y the effort we need to proess the total ount of inputs, and on the other hand y the redution apaility when splitting the input sequene into smaller parts. Figure 7 6 An improvement of this strategy would e to ollet possile orret oservations for more than one oservation point. 3.5 Evaluating the Test Proess When a test suite is generated with the algorithm aove and a system under test is tested with this test suite we would like to know how extensively we tested the system under test. The numer of test ases and the length of the input sequenes in the test ases only onditionally allow to draw onlusions related to that question. Still today the question is hard to answer. The mostly used approah is to measure the overage of different (strutural) elements of the system under test or the speifiation. For general ode this is ommon pratie. The used riteria are usually ased on ontrol flow or data flow information in the ode or on funtional desription in the speifiation. With our test approah we address emedded systems omposed of hardware and software omponents. You an apply well known tehniques to measure overage in the software omponents, ut our impression is that this is not suffiient for suh systems. To measure overage in the hardware omponents is usually not possile. The only way to regard the whole system is to use the speifiation. Thus we need to develop meaningful riteria for state mahines. Our urrent work introdues different riteria ased on strutural elements of state mahines, like states and transitions, and on semanti elements, like onfigurations and sets of firing transitions. First results show that espeially semanti riteria are ale to evaluate the ehavior in a meaningful manner. For the future we partiularly address suh semanti riteria ased on the semanti model of state mahines and their relation to seleted input sequenes. A still open and interesting question is whether it is possile to use those riteria to ontrol the test ase generation proess, viz. to measure overage while generating test ases and to selet the inputs aording to this overage. 4. TOOL SUPPORT To evaluate and to show the pratiaility of our approah we implemented the TEAGER Tool Suite. Figure 8 shows the general arhiteture of the TEAGER Tool Suite. TEAGER onsists of an environment to automatially generate and exeute test ases, and additionally of an environment to exeute state mahine speifiations. The latter we use to analyze the exeution ehavior and the testaility of a state mahine, and to measure overage on a state mahine speifiation to evaluate generated test suites.

9 os2 os1 input1 input2 inputn... os1 os2 Pass os3 os3 os1 Inonlusive Pass Figure 7: General Struture of a omined Test Case. Espeially the modularization of the different task in automatially generating test ases makes the approah interesting for further researh. All disussed strategies are implemented as modules of the tool suite. Thus, different strategies for seleting inputs, for ominria , version 2-23 Apr 2008 tion. The Test Driver in turn loads saved test ases and exeutes them. The exeution inludes oth: stimulating the system under test and omparing the oservation to the omputed possile orret ehavior in the aeptane graphs. The ommuniation with the system under test takes plae over a soket onnetion using pre-implemented adaptors. This onept offers a flexile way to onnet the system under test. It also offers the possiility to use our State Mahine Exeutor as a system under test stu. Thus we an analyze the exeution ehavior of state mahine speifiation or measure the overage of a used speifiation. The omplete test ase generation proess is parameterized to have maximal ontrol over the struture of test ases and the effort needed to alulate them. First you an speify the numer of test ases to e generated, the length of input (su-)sequenes and the numer input sequenes to e omined in a test ase. Then you an speify the way input sequenes are generated. As an example, we implemented different proailisti strategies whih desrie possile environments (f. Setion 3.2). It is also possile to speify the sequene of events as a preamle to generate test ases related to speifi parts of the speifiation. After onfiguring all parameters the test ase generation works ompletely without any user interation. For test ase exeution you an ontrol the frequene at whih inputs are sent to the system under test. To avoid a fixed timing we use a Gaussian distriuted trigger rate with a mean value and deviation to e speified y the tester. The tester also speifies the numer a test ase should e repeated, and the poliy how several exeution results should e omined. Exeuting a test ase several times is espeially neessary when dealing with non-deterministi systems. Every exeution an ause the system to exeute a different path for the same input. However, we need to hek all resulting oservations. How often test ases should e exeuted when dealing with different non-determinism annot e fixed in advane. Thus the given numer is a so-alled test hypothesis. The term hypothesis expresses that we assume the numer high enough to test the system under test adequately. The timeout value whih an e speified is also an test hypothesis. This upper time ound speifies how long the test driver should wait for a desired oservation. Usually, this time ound is higher than the reation of the system. So the system under test has enough time to produe the reation for an input. The timeout value diretly influenes the test ase exeution ehavior sine we use it to implement our oservation points. Up to this ound all reations an e oserved and (as important as the previous fat) no other reations an e oserved. A value too short would ause unneessary false negatives; a value too high would unneessarily slow down the test exeution. For the omination of different exeution results of the same test ase different strategies are imaginale. Atually we use three different strategies: MUST requires every test exeution to pass. STRONG_MAY requires at least one test exeution to pass, whereat the test exeution will e repeated (up to the numer of test repetitions) until the first test ase passes. WEAK_MAY requires that no test exeution fails. The State Mahine Exeutor exeutes a state mahine and thus allows an exploration of misellaneous properties like the testaility or the overage of a speifiation whih we use to evaluate the quality of a generated test suite. Here it is partiularly important to avoid a fixed exeution timing. For UML state mahines the soalled zero time assumption does not hold. Instead, it is assumed that exeuting a transition onsumes time. To respet this and espeially to e ale to investigate effets of the used asynhronous ommuniation we also use a proaility ased sheme for the exeution times of transitions. The tester an speify the mean value and the deviation for a Gaussian distriution whih is used to selet an exeution time of every transition exeution. Thus effets of different timings an e tested. First experiments with the TEAGER showed, that we an meet the state explosion prolems introdued through the semantis of state mahines with our approximation strategy. The generation and exeution proess is parameterized. This allows the appliation of different testing strategies and to have maximal ontrol over the omplete proess. For more information aout the TEAGER Tool Suite, its individual omponents, and the used parameters, we refer the interested reader to our we site [19]. 5. SUMMARY AND OUTLOOK Testing enefits from the fat that the real system is rought to exeution. Thus, the interation of the real hardware and the real software an e evaluated. It aims in falsifiation, i. e. to show inonsistenies etween the speifiation and developed system. Testing is appliale at different levels of astration and at different stages of the development. With our approah UML state mahines an e used in the quality assurane to serve as a speifiation for the desired reative ehavior of the system. It is possile to selet relevant and interesting inputs for a test ase and to alulate the possile orret oservations for given inputs. They allow to automatially evaluate test exeutions whih is in general a diffiult and time onsuming task. Applied approximation makes the generation proess pratial, whereat it is possile to ontrol this proess depending on the time and omputation power to invest.