PCO ASPs IUT. Tester. ASPs PCO. PDUs. Test System TCP. ASPs PCO. PDUs IUT. Service Provider. Lower Tester Control Function TCP

Transcription

1 Accepted for Computer Networks & ISDN Systems: Special Issue on Protocol Testing TTCN: Towards a Formal Semantics and Validation of Test Suites Finn Kristoersen Thomas Walter y Abstract TTCN (Tree and Tabular Combined Notation) is the standardized test notation for the description of OSI conformance tests. Since applicability of TTCN is restricted, work on the denition of concurrent TTCN, an extended version of TTCN for the specication of test cases for multi party testing, has been started a few years ago. In this paper we discuss dierent approaches for the denition of an operational semantics of TTCN and concurrent TTCN and we discuss issues related to the validation of (concurrent) TTCN test cases. Because for the validation of test cases a proper semantics denition is a prerequisite we have developed a semantics denition which utilizes labelled transition systems as its basic model. The applicability of the model is demonstrated: First, we show how identied incompletenesses and ambiguities of TTCN can be solved. Second, we develop a validation framework that denes the necessary machinery for the validation of functional properties of test cases against a formal specication. We illustrate how validation of proper termination and verdict assignments can be performed and how a test case can be validated against an SDL specication. 1 Introduction TTCN and concurrent TTCN [15, 16] are notations for specication of test suites of OSI protocol standards. Standardized test suites are produced to ensure that dierent implementations of the same protocol standard are checked for the same set of requirements [13, 14, 8]. The production of test suites in TTCN and concurrent TTCN has until recently been a manual process 1 and quite often errors has therefore occurred in test suite specications. As errors in test suites have a signicant impact on the quality achieved by conformance testing, validation of test suites is necessary. Performing test suite validation requires the test suite to be given in a well-dened notation, i.e. a notation which has a formally dened syntax and semantics. A formally dened syntax is necessary to utilize tools to validate the syntactical correctness of a test suite. A formal semantics is necessary for the validation of a test suite against functional properties; e.g. proper termination (every test run yields a verdict after a nite number of test steps) or whether valid behaviour as specied in a test case is indeed valid behaviour with respect to the protocol specication. Besides for the validation of test suites, a test notation with a formal syntax and semantics is also of importance in order to implement tools for the generation and the implementation of test suites. The latter issues are, however, out of the scope of this paper. In this paper we discuss two approaches for a semantics denition of TTCN. We give a survey of the ocial semantics denition standardized in Annex B of [15, 16]. This semantics denes an abstract machine that executes test cases. As it has certain weaknesses and does not support formal validation of functional properties we look for an alternative approach to dene a semantics for TTCN and concurrent TTCN. This alternative semantics has been inuenced by a semantics denition for SDL [9]. The dened semantics is an operational semantics that allow reasoning about functional properties of test cases at dierent levels of abstraction. The semantics is not meant to replace the standardized semantics of TTCN, but rather Tele Danmark Research, Lyngs Alle 2, 2970 Hrsholm, Denmark, Tel: , Fax: , nn@tdr.dk y Computer Engineering and Networks Laboratory, ETH Zurich, 8092 Zurich, Switzerland, Tel: , Fax: , walter@tik.ethz.ch 1 The authors' own experiences in ETSI and EWOS projects are that test suite production is still mainly performed manually. Tool support is provided for the editing of test suites and for checking syntax and static semantics. Tools supporting test case generation has been applied only very recently. 1

2 as an alternative model that supports validation of functional properties of test cases against requirements specied in a formal protocol specication. For the latter kind of validation we describe a \validation framework". The essential components are a semantical model for SDL [33] and TTCN and concurrent TTCN, and an appropriate behaviour relation. The model, called common semantics representation (CSR) [7, 30, 31] denes an operational semantics for both languages, providing an abstract execution model which allows to `execute' a test case and a specication and to derive their respective input-output behaviour. Using the dened behaviour relation we demonstrate how the CSR may be used to validate the input-output behaviour of a test case with respect to the behaviour of an SDL specication. The outline of the paper is as follows. In the next section we present some related work which has inuenced the denition of the validation framework. In Section 3 an introduction to TTCN and concurrent TTCN is given. We present dierent approaches for the denition of a semantical model of TTCN (Section 4). The one selected for the validation framework is described in more detail. Dierent aspects of test case validation are identied and in Section 6 we present the validation framework for TTCN test case behaviour against behaviour of an SDL specication. Finally the conclusions are presented. A list of abbreviations used is provided at the end of the paper. Throughout the paper we illustrate concepts with concrete examples. For the introduction to TTCN and concurrent TTCN we use as an example a test case from a test suite for the TP protocol. 2 Related Work It is common understanding that dening semantical models for TTCN and concurrent TTCN is necessary in order to enable test generation, test validation and test implementation. In this section we discuss some approaches. 2.1 TTCN Semantics A denition of a TTCN semantics can, in general, be given using two dierent approaches: direct mapping approach [24, 5, 6] and semantical model approach [3, 11, 26, 15, 16] Direct Mapping Approach The basic idea of this approach is to dene a mapping of TTCN and concurrent TTCN constructs to constructs of a chosen target language that is \semantics preserving". Since the target language is assumed to have a formally dened semantics, the semantics of the target language becomes the semantics of TTCN and concurrent TTCN, too. In [24, 5] a direct mapping from TTCN to LOTOS is dened and in [6, 27] a mapping from TTCN to SDL. Both mappings are based on the identication of conceptual equivalences between the languages. For instance, a TTCN test suite is mapped to a LOTOS specication or SDL specication, a TTCN behaviour tree is mapped to a LOTOS process or SDL system, TTCN PCOs are mapped to SDL channels in the TTCN to SDL mapping, or constraints in TTCN and signal instances in SDL are identied as conceptually equivalent. Although a direct mapping seems to be intuitive there do exist some problems. First, in the TTCN to LOTOS direct mapping PCOs are mapped to gates. But communication at PCOs is asynchronous whereas communication in LOTOS [12] is synchronous, i.e. processes synchronize over gates. So, the mapping of PCOs to gates is not semantics preserving. Secondly, in SDL the following temporal relation between sending a signal instance, receiving a signal instance in an input queue and processing a signal instance by the receiving process instance can be derived: t sending t receiving t processing Since in TTCN before a set of alternatives is evaluated a snapshot is taken, i.e. all PCO queues and timer lists are updated, the following temporal relations hold: t sending t receiving ^ t receiving t processing 2

3 but t sending < t processing The above timing relations of events are interpreted as follows: In SDL, sending, receiving and processing of a signal instance are three events which may be performed simultaneously. In TTCN, on the other hand, only two events may be performed simultaneously: sending and receiving or receiving and processing. Thus, whenever PCOs and SDL channels are considered conceptually equivalent the described subtle dierence in temporal ordering of events has to be taken into account. The above discussion illustrates that even if the direct mapping approach seems intuitively easy to perform the requirement to make it semantics preserving can be dicult or even impossible to fulll. As this requirement is basic to validation we believe that another approach shall be taken if the mapping is to serve also for validation purposes Semantical Model Approach Over the past years a number of semantical models for concurrent programming languages have been developed. In the following we consider only a few that we have found suitable for the denition of an operational semantics of TTCN and concurrent TTCN. The selection of semantical models has been performed with the following requirements in mind: First, the semantical model should improve the understanding of problems related to computer aided test generation and test suite validation; secondly, the chosen semantical model should not imply any restrictions on TTCN and concurrent TTCN; and thirdly, the semantical model should support the analysis of semantical properties (e.g. functional properties) of test cases at dierent levels of abstraction (i.e. with respect to the observable behaviour of entities within a system). Pseudo Code [15, 16]: The intention is to dene an operational semantics in terms of an abstract notation. The pseudo code describes the anticipated behaviour of an abstract machine which executes a test case (see also Section 4.2.1). Streams and stream process function [3]: This semantical model has been used to dene an semantics for SDL. The model is based on functional descriptions of systems using the concept of streams and stream processing function. Streams can be understood as representations of the history of communications between components of a system. Stream processing functions are state machines with input and output. Since the approach does not support the reasoning about timed behaviour, e.g. the specication of timers, this approach is not appropriate for the denition of a semantical model of TTCN and concurrent TTCN. Asynchronous communication tress (ACT) [11]: ACTs can be regarded as labelled transition systems [21]. States of an ACT are comprised of local states of each process and all pending communications sent out by a process but not yet received by another process. The following events are distinguished: reception of a pending communication by the environment, reception of a communication from the environment, and reception of a pending communication by a process within the system. Dependent on the level of abstraction, not all events mentioned are observable. Particularly, the class of events determined by the reception of a communication by a process within the system is regarded unobservable. As we want a semantical model which allows reasoning on system behaviour at dierent levels of event abstraction, the ACT model is less appropriate for this purpose. Common semantics [26, 23]: The SPECS approach is most closely related to our work (Section 4.2.2) although the SPECS objectives are dierent from those we have in mind. Within the SPECS project a common semantics for LOTOS and SDL have been developed. The common semantics consists of a mathematical representation and a common representation language. The mathematical representation is a specication language with the expressive power equal to LOTOS and SDL. Its semantics is given in terms of labelled transition systems. Therefore, it is quite similar to our approach. The essential dierence is that we interpret TTCN and concurrent TTCN directly in terms of labelled transition systems whereas if we have had followed the SPECS approach, TTCN and concurrent TTCN where mapped to the common representation language whose semantics is dened by its mapping to the mathematical representation. 3

4 2.2 Validation of TTCN test cases Several aspects are to be covered by TTCN test case validation. A rst one is checking a TTCN or concurrent TTCN test case for syntactic and static semantics correctness. This kind of validation is well understood. Tools like ITEX TM [17], support this kind of validation. Syntax and static semantics checking is only an initial step in TTCN test case validation. In [1], the use of TTCN is validated against a set of heuristic rules. The rules aim at assessing the correctness of a TTCN test case with respect to the protocol specication in a formal description technique. A simulator is used for both, TTCN and the formal description technique in order to relate the behaviours possessed by a TTCN test case and a protocol specication. In Section 6 we propose an alternative approach of doing TTCN behaviour validation against SDL specications. In [4], validation of verdict assignment in TTCN test cases is discussed. The emphasis is on the validation of the inconclusive test verdict, i.e., with respect to the dened test purpose, the test case execution outcome is such that neither Pass verdict nor Fail verdict can be given. The authors argument that assignment of the inconclusive test verdict is mainly due to the fact that the criteria which dene the assignments of the Pass and Fail test verdicts are not mutually exclusive. To solve this problem they propose procedures for resolving TTCN test cases with inconclusive test verdicts. We extend the work of [4] in the sense that we do a validation of Pass and Fail verdict assignments with respect to the behaviour of a protocol specication (see Section 6). According to TTCN semantics a test case execution should always (after a nite number of steps) result in the assignment of a test verdict [15]. This property cannot be checked statically. A feasible approach validating this kind of property is to \execute" a TTCN test case and to identify unguarded innite loops or unguarded recursion. In case we are able to \execute" a test case then we can also try to validate other properties, e.g., that the test case is correct with respect to the protocol specication. Our approach is to dene a common semantical model for TTCN and concurrent TTCN and the formal description technique. This approach is followed in Section 6 (see also [19]). 3 TTCN and Concurrent TTCN For the purpose of this paper we restrict our attention to TTCN and concurrent TTCN concepts related to the description of test case dynamic behaviour. For further details on other aspects of TTCN and concurrent TTCN we refer the reader to [15, 16] or some tutorial papers [18, 25, 22]. We start with a brief description of testing architectures and testing functions as standardized by ISO and ITU-T [13, 14]. This discussion is essential for the following sections, particularly because it partly motivates our approach to dene an operational semantics for TTCN and concurrent TTCN (Section 4.2.2). 3.1 Testing Architectures and Testing Functions Test cases are dened in terms of input events received from an implementation under test (IUT) and output events sent to the IUT. Since an IUT is generally part of a real system, a variety of possibilities exists on how the behaviour of an IUT can be controlled and observed during execution of a test case. The OSI conformance testing methodology and framework [13, 14], therefore, has prescribed four dierent testing architectures orabstract test methods: local, distributed, coordinated and remote test methods. A conceptual testing architecture is depicted in Figure 1. It can be seen that an IUT is controlled and observed in terms of protocol data units (PDU) and abstract service primitives (ASP) that are exchanged with an IUT. The interfaces between IUT and Tester are pairs of opposite directed FIFO-queues, called points of control and observation (PCO). Since it cannot be assumed that the interfaces below and above an IUT are directly accessible for testing the conceptual testing architecture may be rened as shown in Figure 2 which is the distributed test method. Access to the lower interface of an IUT is provided via a service provider. If the interface above the IUT is not possible to observe and control then this denes the remote test method. The testing architecture shown in Figure 2 is appropriate if an IUT communicates with only one peer entity. The testing architecture is referred to as single party testing architecture. Performing control and observation of an IUT is distributed over components, called lower (LT) and upper tester (UT). The LT interacts with an underlying service provider and is said to control and to observe the IUT from below. On 4

5 PCO ASPs Tester IUT PCO ASPs PDUs Figure 1: Conceptual Testing Architecture Test System SUT LT PCO ASPs TCP PDUs UT PCO ASPs IUT Service Provider Figure 2: Distributed Test Method the other hand, the UT controls and observes the IUT from above. Conceptually, LT and UT are assumed to run on dierent systems. The UT forms part of the system under test (SUT) and the LT is in the test system. In case an IUT has to communicate with multiple open systems, the shown testing architecture has to be extended with several concurrently running LTs and UTs which exchange PDUs or ASPs with the IUT (Figure 3). This testing architecture is called multi party testing architecture. Lower Tester Control Function Test Coordination Procedures LT3 LT2 LT1 TCP TCP TCP PDUs TCP TCP UT1 UT2 UT3 PCO ASPs PCO ASPs PCO ASPs PCO ASPs PCO ASPs PCO ASPs PDUs PDUs IUT Service Provider Figure 3: Multi-party Testing Model [14] From Figures 2 and 3, we can identify four (abstract) testing functions [13]: Lower Tester (LT): An entity that controls and observes an IUT from below via an underlying service provider Lower Tester Control Function (LTCF): An entity that coordinates LTs and UTs, and that is responsible for the assignment of a nal test verdict (see also Section 3.3). Upper Tester (UT): An entity that controls and observes an IUT from above. 5

6 Test Coordination Procedure (TCP): A procedure that denes, either explicitly or implicitly, how LTs, UTs and LTCF cooperate during testing. In the single party testing architecture the following testing functions are required: an LT which behaves as a peer entity to the IUT and which assigns a test verdict, optionally a UT which behaves as a \user" of the IUT, and optionally a TCP between LT and UT. Note that the LT assigns the test verdict and thus also performs tasks that are, in other testing architectures, done by the LTCF. In the single party testing architecture LT and LTCF functions are merged into a single function. In the multi party testing architecture the following testing functions are required: a number of LTs which all behave as peer entities to the IUT, optionally a number of UTs which behave as \users" of the IUT, a LTCF which coordinates the behaviour of LTs and UTs (if any), and optionally a TCP between LTs, UTs and LTCF. 3.2 TTCN and concurrent TTCN Testing Architectures In [14] it is recommended to use a standardized test notation, i.e. either TTCN [15] or concurrent TTCN [16], to specify test cases. TTCN is applicable in the single party testing architecture and concurrent TTCN is applicable in the multi party testing architecture. Generally speaking, concurrent TTCN is a superset of TTCN. Whereas in TTCN a test case is executed by a single test component which implements the LT and optionally UT functions, concurrent TTCN supports execution of a test case by several test components (TC) running in parallel. LTs, UTs and LTCF in the multi party testing architectures are realized as TCs. Communication between TCs is asynchronous. TCs are interconnected by coordination points (CPs). A CP is a concept similar to a PCO, i.e. a CP is realized by two FIFO-queues one for each direction of communication, however a CP does not connect an LT or a UT with an IUT. An additional restriction applies [16]: CPs shall not be used between pairs of LTs and UTs (e.g. LT1 and UT1 in Figure 3). In order to coordinate their joint behaviour TCs exchange information, called coordination messages (CM). The exchange of CMs between TCs and the processing of received CMs is dened in a TCP. Example 1 Concurrent TTCN denes a means to set up complex test congurations. This is done in several steps. First, all TCs used in a test conguration are declared (Figure 4 2 ). Note that a distinction is made between main test component (MTC) and parallel test components (PTC). Generally, the MTC plays the role of the LTCF (therefore the name LTCF is assigned to the MTC in Figure 4) and LTs are simply PTCs. Second, the CPs used are declared (Figure 5). Third, a test components conguration is given (Figure 6) providing the information which test components, PCOs and CPs are used. Note that MTC LTCF does not use PCOs, that LTCF and LT T communicate through CP CP T, and that LTCF and LT0 communicate through CP CP0. Note further, that no UT is used in this conguration. Thus, test conguration CONFIG 1 (Figure 6) is an instance of the multi party testing architecture shown in Figure 3. Figure 4: Test Component Declaration 2 A test component conguration can be used in several test cases. For test suites that are dened using concurrent TTCN every test case should make a reference to a test component conguration. 2 Throughout the paper we use the graphical (or tabular) form (TTCN.GR) of the TTCN syntax. TTCN.GR is the form \suitable for human readability" [15]. All TTCN.GR tables shown have been produced using ITEX TM Version 2.2 [17]. ITEX TM is a TTCN development environment that supports editing and analyzing of TTCN and concurrent TTCN test suites. 6

7 Figure 5: Coordination Point Declaration Figure 6: Test Conguration 3.3 Test Case Dynamic Behaviour Descriptions A TTCN test case describes the dynamic behaviour of TCs during test execution. A dynamic behaviour description consists of statements and verdicts. A verdict is a statement concerning the conformance of an IUT with respect to the sequence of test case events that was performed. Possible verdicts are: Pass, Fail and inconclusive. Every execution of a test case results in a verdict assignment. Statements can be grouped into statement sequences and sets of alternatives. In the graphical form of TTCN, sequences of statements are identied by dierent levels of indentation, e.g. lines 1 { 7 of Figure 7. Statements on the same level of indentation are alternatives, as line 8 and 9 of Figure 7. Figure 7: Test case behaviour description 7

8 TTCN distinguishes statements that are test events, constructs and so-called pseudo events. Test events dene basic input and output actions. An input action or RECEIVE event is the reception of an abstract service primitive (ASP) or a protocol data unit (PDU) from a PCO. In case of concurrent TTCN, the input action includes also reception of a coordination message (CM) from a coordination point (CP). An output action or SEND event is sending an ASP or PDU to a PCO or sending a CM to a CP. Test events may be qualied by Boolean expressions and may be followed by a combination of assignments and timer operations. Example 2 Examples of RECEIVE and SEND events are given on lines 4 and 5 of Figure 7. The RECEIVE event in line 4 is to be interpreted as follows: The TC accepts from CP CP T a CM of type Start TS CM with value as specied by constraint reference cm start. If the RECEIVE event is executed, test case execution continues with the SEND event that transmits a CM of type Comp TSCM with value cm start (see column \Constraints Ref") along CP CP0 to TC LT0. 2 DONE is a test event that can be used by the MTC to check whether a specic PTC or all PTCs have terminated. The DONE event is successful when all of the indicated PTCs have terminated. E.g. in Figure 7 line 8 and 12 it is checked for PTC LT T and the PTCs LT T and LT0 respectively. Constructs are used to guide the ow of control in test cases and to give test cases a modular structure. For the latter TTCN provides the ATTACH mechanism that allows combinations of statements, called test steps, to be \attached" to a test case. For determining the ow of control, GOTO and REPEAT constructs, which are in one or another form known from many other programming languages, are introduced into TTCN. Additionally, CREATE is a construct for controlling the concurrent behaviour of TCs. It is the responsibility of the MTC to set up all PTCs of a test conguration. The CREATE operation associates a PTC with a behaviour tree. If the behaviour tree is parameterized the CREATE operation passes also actual parameters for formal parameters. For instance, the CREATE statement on line 3 in Figure 7 instantiates TC LT0 with behaviour description BDL (Figure 8). The newly created PTC runs in parallel with the MTC. If the PTC to be created is in the course of execution of a behaviour tree then execution of a CREATE by the MTC results in a test case error indication. Figure 8: Test component behaviour description Pseudo-events are qualiers (i.e. Boolean expressions), assignments and timer operations. 8

9 3.4 Test Case Execution A behaviour tree [15, 16] denes the relative ordering of test events that a TC 3 (either the MTC or a PTC) executes. Execution of a test case starts with the execution of the MTC which is responsible for setting up the test conguration dened for the test case. As stated above, MTC and PTCs run in parallel. The statements given in a behaviour tree are executed in sequence only if a statement is successful. A statement is evaluated successfully under the following conditions: A REPEAT and a GOTO is always successful. A SEND event, an assignment and a timer operation are always successful provided that the optional accompanying qualier holds. A RECEIVE event is successful if an ASP, a PDU or a CM can be received from the specied PCO or CP and the ASP, PDU or CM received matches the constraint dened in column headed \Constraints Ref" and provided that the optional qualier holds. A CREATE operation and a DONE event are successful under the above mentioned conditions. A qualier evaluates successfully if it evaluates to true. A TIMEOUT event (Figure 7 line 9) is successful if the specied timer (i.e. timer CASE TIME) has expired. In TTCN and concurrent TTCN a discrete time model is assumed where time is counted in time units of arbitrary granularity (from picoseconds to minutes). Alternatives of a set of alternatives are evaluated one after another starting with the rst alternative until one alternative is found successful. If all alternatives have been evaluated unsuccessfully, the evaluation is repeated starting with the rst alternative. Note that before a set of alternatives is evaluated the current state of the test system is frozen or, as stated in [15], a snapshot is taken. That is all PCO and CP queues and expired timer lists are updated and during the evaluation no update takes place. In that sense evaluation of a set of alternatives is an atomic action. Test case execution terminates when a nal verdict is assigned or a leaf in the behaviour tree is reached. With respect to concurrent TTCN, termination of a test case is quite more complex. As a general rule, whenever the MTC terminates either by assigning a nal verdict or reaching a leaf in the behaviour tree, test case execution terminates independently of any PTC still running. If a PTC terminates either by assigning a nal verdict or reaching a leaf in the behaviour tree then this does not imply that test case execution terminates. If a test verdict is assigned in a PTC then this is a local matter only. The test verdict assigned by a PTC may only be communicated to the MTC by exchanging an appropriate CM. Every TC manages its own set of variables and timers. In the context of concurrent TTCN, variables local to a TC are denoted test component variables. Access to globally dened variables, e.g. test suite variables, is restricted to the MTC. Test suite parameters and constants are shared between TCs. 4 TTCN and Concurrent TTCN semantics In the OSI conformance testing standard [15] it is stated: \...this part of ISO/IEC 9646 denes an informal test notation, called the Tree and Tabular Combined Notation (TTCN), for OSI conformance testing,..." However, the test case development process requires a precisely dened semantics as certain tasks, e.g. test case validation, test case compilation, test case execution, and test result analysis, can only be supported by tool based methods if a formal semantics is available. The main focus of this section is to present an approach to formally dene a semantics of TTCN that enables test case validation. 4.1 Preliminaries The operational semantics presented in the following two sections is dened for TTCN test cases that are in a specic form which we call canonical form. Whenever during execution of a TTCN test case a set of alternatives is reached, then the set of alternatives is transformed into its respective canonical form as follows (see Annex B of [15, 16]): First, all REPEAT constructs are substituted by a combination of a Boolean expression and a GOTO construct. Second, all default behaviour trees 4 are \attached". Third, all attached trees are expanded, i.e. the referenced test step behaviour description is substituted for the tree attachment construct. 3 We assume that LT and UT of a single party testing architecture are modeled by one TC. Thus, the following statements are also valid for TTCN. 4 A default behaviour description contains all statements which may occur at any level of a behaviour tree. 9

10 Example 3 The initial set of alternatives in the behaviour tree shown in Figure 8 is in canonical form since it does not make use of the REPEAT construct and no tree attachment is used. After the initial set of alternatives has been evaluated, tree attachment +PR S3nInp0 has to be expanded into canonical form before evaluation of this set of alternatives can be done Operational Semantics of TTCN and concurrent TTCN Although the standard itself [15, 16] does not claim to be a formal denition of an operational semantics, it is a starting point for the denition of a formal semantics of TTCN and concurrent TTCN. In Section we present an alternative approach and subsequently we compare the two approaches The ISO Operational Semantics The ocial ISO TTCN semantics [15, 16] is an operational semantics dened in terms of a TTCN machine. A TTCN machine executes a programme that simulates the execution of a test case in canonical form. The programme is dened in terms of a set of functions that includes a function for every test event, construct and pseudo-event not eliminated during transformation of a test case into its canonical form. The programme is dened using a pseudo-code notation. In the following we elaborate on some of the functions. Receive event: A RECEIVE event checks whether a specic ASP, PDU or CM (specic with respect to types and values) has been received; i.e. whether the rst element of the PCO satises the requirements of the \Constraints Ref" (Figure 8). Recall that a test event can be qualied by a Boolean expression and associated with assignments and timer operations. The function that denes the semantics of receiving an ASP or PDU and execution of assignments and timer operations is as follows: function RECEIVE(PCOidentifier or CPidentifier, ASPidentifier or PDUidentifier or CMidentifier, Qualifier, Assignment, TimerOperation, ConstraintsReference, Verdict, Level) : BOOLEAN begin if RECEIVE_EVENT(PCOidentifier or CPidentifier) then begin if (RECEIVED_OBJECT(ASPidentifier or PDUidentifier or CMidentifier, ConstraintsReference) AND EVALUATE_BOOLEAN(Qualifier)) begin EXECUTE_ASSIGNMENT(Assignment); TIMER_OP(TimerOperation); REMOVE_OBJECT(PCOidentifier or CPidentifier); VERDICT(Verdict); Level := NEXT_LEVEL; LOG(PCOidentifier or CPidentifier, ReceivedObject); RETURN(TRUE); end else RETURN(FALSE) else RETURN(FALSE) end where function RECEIVE EVENT() returns true if the incoming queue of PCO PCOidentifier or CP CPidentifier is not empty. In this case a copy of the rst element is created. In the next step the value 10

11 of the received ASP, PDU or CM is compared with the value dened in constraint ConstraintsReference. If function RECEIVED OBJECT returns true and if qualier Qualifier is true, i.e. function EVAL- UATE BOOLEAN() returns true, then the assignments are executed (function EXECUTE ASSIGNMENT()), the timer operations are performed (function TIMER OP())and the received object is removed from the incoming queue (function REMOVE OBJECT()). The current value of the verdict is calculated by function VERDICT. If the verdict is a nal verdict then test execution is stopped; otherwise, the next level of alternatives is determined (function NEXT LEVEL()). As part of the execution of the RECEIVE event, a log le is updated before processing of the next set of alternatives is started. Since function RECEIVE() returns only after all assignments and all timer operations accompanying a RECEIVE event, we conclude that execution of a RECEIVE event is an atomic action. This holds true for all other TTCN statements as well (see below and [15, 16]). CREATE: A PTC should only be created by the MTC if the PTC is not already executing; otherwise a Test Case Error should be generated. If creation of a PTC is successful then the PTC runs in parallel with the MTC and all other PTCs previously created. The pseudo-code for the CREATE construct is as follows: function CREATE(TCIdentifier, TreeReference(ActualParList), Qualifier,... ) : BOOLEAN begin if EVALUATE_BOOLEAN(Qualifier) then begin... START_EVALUATION(TCIdentifier, TreeReference(ActualParList));... RETURN(TRUE); end else RETURN(FALSE) end Function START EVALUATION() instantiates the PTC TCIdentifier with behaviour description Tree- Reference and actual parameters ActualParList. PTC TCIdentifier starts to execute which means that a TTCN machine is created simulating the execution of behaviour tree TreeReference. Function START EVALUATION() then returns immediately. DONE: The MTC can use the DONE construct to check the status of PTCs specied as actual parameters. function DONE(TCompList, Qualifier,... ) : BOOLEAN begin if EVALUATE_BOOLEAN(Qualifier) AND ALL_TERMINATED(TCompList) then begin... RETURN(TRUE); end else RETURN(FALSE) end 11

12 The status of PTCs TCompList is checked by calling function ALL TERMINATED() which returns true if all PTCs have terminated, or which returns false otherwise. Note that an optional qualier is evaluated rst. EVALUATE TEST CASE: Up to now we have discussed execution of test events, constructs and pseudo-events. We continue with a brief discussion of the execution of a behaviour tree: procedure EVALUATE_TEST_CASE(TestCaseId, BehaviourTree) begin Level := FIRST_LEVEL(BehaviourTree); EVALUATE_LEVEL(TestCaseId, BehaviourTree, Level, TestCaseId, DefRefList) end procedure EVALUATE_LEVEL(TestCaseOrStep, Level, Defaults) begin EXPAND_LEVEL(TestCaseOrStep, Tree, Level, Defaults); repeat TAKE_SNAPSHOT; if EVALUATE_EVENT_LINE(A1, Level, Defaults) THEN EVALUATE_LEVEL(TestCaseOrStep, Tree, Level, Defaults);... if EVALUATE_EVENT_LINE(Am, Level, Defaults) THEN EVALUATE_LEVEL(TestCaseOrStep, Tree, Level, Defaults); UNTIL SNAPSHOT_FIXED(Level); STOP end Execution of a behaviour tree starts with the rst level of indentation, i.e. the rst set of alternatives. If necessary a set of alternatives is expanded (EXPAND LEVEL) in preparation for evaluation of alternatives. The evaluation (EVALUATE EVENT LINE) is done one by one starting with the rst alternative and ending with the last one. But before evaluation of alternatives is performed a snapshot (TAKE SNAPSHOT) is taken which means that incoming PCO and CP queues and timer list are updated and the termination status of any other test component is determined. The rst alternative which is evaluated successfully is taken and processed. Afterwards, i.e. upon return from EVALUATE EVENT LINE the set of alternatives at the next level of indentation that follows the event taken is evaluated. This process continuous until test case execution terminates (by assigning a nal verdict) or behaviour tree execution deadlocks, i.e. all PCO and CP queues have some messages pending and all relevant timers have expired but these are not the messages or timers expected to be received or to be expired. At rst sight the denition of an operational semantics in a procedural form seems to be adequate. One can think of implementing a programme that, as described by the dierent functions, executes a test case step by step. However, some details of the informal (but normative) denition of TTCN and concurrent TTCN are not properly dened: [15] Section c) states that \in cases where a qualier is specied on the event line, the qualier shall evaluate to TRUE; the qualier may contain references to ASP parameter and/or PDU elds". Given function RECEIVE() (see above), evaluation of a qualier is done by calling function EVALUATE BOOLEAN() with parameter Qualifier. However, parameter Qualifier is the same as the formal parameter Qualifier of function RECEIVE(). Thus, possible references to ASP parameters, PDU elds or, in the case of concurrent TTCN ([16] Sections and ), CM elds are not updated by the referenced values of an ASP, a PDU or a CM received. Evaluation of Qualifier may yield an incorrect or even undened result. We conclude that the denition of function RECEIVE() is incomplete and ambiguous. 12

13 A PTC should be created only if the PTC is not already running [16]: \The execution of a CREATE construct on a Test Component which has already been created shall be a Test Case Error". In function CREATE() no check is performed whether PTC TCidentifier is running or not. It is neither obvious when to generate the Test Case Error nor is it foreseen in the denition of function CREATE() that creation of a PTC may fail. As indicated, some aws can still be found even in the denition of TTCN, although TTCN is already an international standard and concurrent TTCN is to become international standard A Common Semantics Representation Approach The common semantics representation (CSR) has originally been developed as a semantical model for both SDL and TTCN [7]. The CSR is a model for the representation of the dynamic behaviour of syntactically well-formed and static semantically correct TTCN and concurrent TTCN test cases. It is dened as a hierarchical model with a structure as shown in Figure 9. Test System PCOs Test Module CPs Test Components Basic Process Input Process Timer Process Figure 9: Common semantics representation for concurrent TTCN The structure of the CSR has been motivated by the following observations: The test system denes the top level in the semantical model. Communication between a test system and an IUT takes place via PCOs. All communication along a PCO can be seen by an observer in the environment of a test system. In the CSR each PCO is modeled by two unidirectional channels. PCOs introduce a nite delay on the messages they convey between source and destination. The test module is the model for test components and CPs of a test conguration. Since communication between test components through CPs is inside a test system, this communication is not observable outside the test system. A test module encapsulates the internal structure of a test system and abstracts also from internal communication. Lower and upper testers (Figures 2 and 3) are mapped to test components in the CSR. In case of concurrent TTCN, TCs may be connected by CPs (see the restrictions discussed in Section 3.2). Like PCOs, each CP is modeled by a pair of unidirectional channels in the CSR. Every test component manages its own set of variables that includes test suite constants, test suite parameters and test component variables, and its own set of timers. If the test component is the MTC it manages also test suite variables and test case variables. For this purpose a test component in the CSR has associated an environment and storage for the mapping of variables to (memory) locations and the assignment of locations to values, and a timer process that maintains a list of running timers. A basic process is the entity that controls the actions that may be performed next by a test component. To be more precise, the basic process is a representation of the behaviour description that has been assigned to the test component. As PCOs and CPs introduce a nite delay on messages, it is necessary to control the reception of ASPs, PDUs and CMs by a test component. An input process stores all messages that a test component has received but not yet processed. 13

14 Note that in the model no distinction is made between lower and upper tester nor does the model distinguish the lower test control function; all are test components. We dene an operational semantics for TTCN and concurrent TTCN in terms of labelled transition systems (LTS) [21]. Particularly, we associate with each CSR entity an LTS that describe the behaviour of that entity. Denition 1 A labelled transition system (LTS) is a triple (S; ;!) which consists of a set S of states, a set of atomic events and a transition relation! S S. From a formal point of view the operational semantics of an entity is given by a family of LTSs, i.e. an LTS for every initial state. An element in! is termed transition and is written s?! e s 0 for s; s 0 2 S, e 2, and (s; e; s 0 ) 2!. 2 In this paper we do not give the complete denition of the LTSs in the CSR. More complete descriptions of the CSR can be found in [31, 7, 19]. In the following paragraphs we give a brief description of the principles of the approach Basic Processes The basic process is the control part of a test component. A behaviour description of a basic process is constructed from a set of operators and a set of atomic events. The set of operators include event prexing, denoted \;", and priority choice, denoted. The set of atomic events includes events for input and output of messages, control of timers, manipulation of variables and data terms, managing of test components, etc. We refer to the set of atomic actions and the set of operators as basic process algebra (BPA). Example 4 Given the TTCN Test Case Dynamic Behaviour of the MTC shown in Figure 7. In the BPA, this behaviour description is represented as shown in Table 1. A TTCN statement is transformed into a sequence of atomic events. Since in our approach parallelism is modeled by arbitrary interleaving of events (see Figure 10 below), whenever an atomic event has been executed the same or another CSR entity may be scheduled for execution. With respect to the evaluation of a set of alternatives this does not cause problems as taking a snapshot and checking which alternative is successful coincide. Furthermore, all test components have their own copies of variables and timers so that the update of a variable or expiration of a timer by one process does not interfere with events performed by another process. Timeouts are modeled as messages, i.e. whenever a timer expires, a timeout message is created and is put into the input process. The timeout message can be retrieved using a simple input event. For more details on the transformation of TTCN into BPA expressions and the evaluation of set of alternatives the reader is referred to [7]. 2 Given a basic process all information that determines the future behaviour of that process is included in the process denition itself. A basic process evolves by performing atomic events thus performing statetransitions. The start state and the next state of a state-transition of a basic process are given by a term in the BPA. The set of states S of an LTS consists, thus, of all terms over the BPA. The transition set! of an LTS is not given explicitly but it can be determined by applying a set of inference rules [21, 10]: Given a state s, an event e and a next state s 0 e then s?! s 0 is a transition if this transition can be proven from the set of inference rules. Note that indices are used to distinguish between inference rules for dierent entities. Example 5 Let e be an event of a basic process, and let P; P 0 ; Q; Q 0 ; e; P and P Q be basic processes. The following inference rules are dened: 1) 2) e; P e?! B P e P?! B P 0 P Q?! e B P 0 and e Q?! B Q 0 P Q?! e B Q 0 i.e. a basic process e; P can always perform any event e and evolve into P (inference rule 1) and a basic process P Q can perform an event e if one of its constituent basic processes can perform event e (inference rule 2). 2 14

15 P ::= start(case T IME; now + T IMEOUT V ALUE); createtc(lt T; T BDL ); createtc(lt 0; BDL ); input(cm start) from CP T [true]; output(cm start) via CP 0; input(cm comp) from CP 0[true]; output(cm comp) via CP T ; (done(lt T ) ; stop; nil) (input(case time) [true]; output(cm timeout) via CP T ; output(cm timeout) via CP 0; done(lt T; LT 0); v := Fail; stop; nil) Table 1: Basic process of Test Case Dynamic Behaviour description (Figure 7) Test Components Similarly an operational semantics is dened for all other basic entities in terms of an LTS. Basic entities of the CSR are input processes, timer processes, CPs and PCOs. For the other entities of the CSR (test components, test modules and test system), the operational semantics is dened relative to the lower level entities. For each such entity the operational semantics is deduced from the operational semantics of its constituent parts. Example 6 In order to dene an LTS of a test component we have to consider its lower level constituents (Figure 9): basic process, input process and timer process. A state of a test component is given by the state of the basic process, the state of the input process and the state of the timer process. Additionally, the local storage of the test component, i.e., the mapping of variables to values forms part of a state for a test component. The compositional structure of the CSR is not only reected in the denition of states but also in the denition of inference rules. Let P; P 0 be states of a basic process, let ; 0 be states of an input process, that is sequences of messages, and let T C; T C 0 be states of a test component. The inference rule that denes the behaviour of a test component processing a received message is dened by: P input(st) via l [bt]?! B P 0 deliver(sig; l)?! I 0 T C?! C TC T C 0 A test component in state T C can perform an internal event evolving into state T C 0 provided the basic process in state P can perform event input(st) via l [bt] evolving into state P 0 and provided the input process in state can perform event deliver(sig; l) evolving into state 0. These two transitions form part of the premise of the inference rule. To verify the premises of the inference rule it suces to check whether both transitions can be proven from the inference rules for basic processes and for input processes (see [19]). Recall that a test component performs a RECEIVE event only if received values and expected values (as given by a constraint) match. In our approach this requirement is expressed as part of a side-condition to the inference rule; in the above example C denotes the following side-condition: C = 8 < : sig =k st k "; k bt 0 k "; = true k now k "; = time The constraint is specied by data term st that is evaluated in environment " and storage, this means that the evaluation is performed with a specic mapping of variables to values. For the condition to be satised the received value sig must match the expected value k st k ";. Since processing of the messages should be performed only if Boolean expression bt holds, this requirement is expressed in side-condition C as k bt 0 k "; = true where bt 0 is derived from bt by substituting (received) values for references in bt. The 15

16 third line of the side-condition is to express that before the transition can be performed the local clock must hold the current global time. This condition is specied for every inference rule of a test component. This is in order to deal with timing constraints as imposed by timers. In the CSR an discrete model of time is assumed. Time is counted in ticks but no specic duration is associated with a tick. An inference rule is applicable only if the side-condition is true. 2 For the CREATE operation and the DONE event the inference rules are as follows: Example 7 In order to determine the status of PTCs we introduce a Boolean array b[0 : : : n] where n is the number of test components. Before test case execution starts, all elements in array b[] are set to false. After test case execution has started, element b[0] becomes true indicating that the MTC has been created and is running. Whenever a PTC is created by the MTC and starts execution, the corresponding element in array b[] becomes true. If a PTC stops execution then the corresponding element in array b[] is again set to false. Assuming the MTC creates a PTC X (the test component X must be a valid test component of the currently used test conguration) and associates to X behaviour description (Q; (l 1 ; : : : ; l p ); (y 1 ; : : : ; y m ); ((z 1 ; t 1 ); : : : ; (z n ; t n ))) where Q is a basic process, l 1 ; : : : ; l p are formal PCO and CP identiers, y 1 ; : : : ; y m are formal value parameters and (z 1 ; t 1 ); : : : ; (z n ; t n ))) are local variables and their initial value, then the following inference rule applies: P createtc(x; Q(l0 1 ; : : : ; l0 p )(t1; : : : ; tm))?! B P 0 C T C createtc(x; Q(l0 1 ; : : : ; l0 p )(a1; : : : ; am))?! TC T C 0 where l 0 ; : : : ; 1 l0 p are actual PCO and CP identiers and a i =k t i k "; for i = 1; : : : ; m are actual value parameters. Side condition C has the following form: b[x] = false C = k now k "; = time i.e. a test component should only be created if it is not already being used, condition b[x] = false. For the DONE event the inference rule applies: where C = P done(x1; : : : ; Xn)?! B P 0 C done(x1; : : : ; Xn) T C?! TC T C 0 8i = 1 : : : n : b[xi ] = false k now k "; = time i.e. a done event is successful only if all PTCs X 1 ; : : : ; X n have terminated (condition 8i = 1 : : : n : b[x i ] = false). 2 The following inference rule models the evaluation of a set of alternatives. Let P = Q 1 : : : Q m be a priority choice expression. Let ; 0 be states of an input process and let ; 0 be states of a timer process. For the following inference rule let T C = hp; ; i be a state of a test component, i.e. a tuple consisting of states of a basic process, an input process and a timer process. The inference rule in the CSR for evaluation of a set of alternatives is as follows: f:hq i ; ; i 6?! TC j i < jg ej hq j ; ; i?! TC hq 0 j ; 0 ; 0 i ei ej hp;;i?! TC hq 0 j ;0 ; 0 i C where C =k now k "; = time 16