A GENERIC TOOL FOR AUTOMATIC PROTOCOL CONFORMANCE TESTING WITH APPLICATION TO ATM EQUIPMENT *

Transcription

1 A GENERIC TOOL FOR AUTOMATIC PROTOCOL CONFORMANCE TESTING WITH APPLICATION TO ATM EQUIPMENT * M. Alvarez-Campana, E. Vázquez, J. Vinyes Dept. Ingeniería de Sistemas Telemáticos Universidad Politécnica de Madrid Spain ABSTRACT Protocol conformance testing provides an extremely valuable mechanism in order to reduce incompatibility problems when interconnecting equipment from different manufacturers. This is achieved by defining a common set of test cases that an implementation must pass to be considered compliant with a particular standard. The benefits of the conformance testing methodology have led major standardization organizations to its application to an increasing number of communication protocols (for example Asynchronous Transfer Mode, Integrated Services Digital Network, Signaling System No. 7, etc.) Thus, nowadays it is common practice to find the definition of Test Suites along with the technical specifications of a protocol. Unfortunately, things are not that easy when it turns to apply a test suite to a real system. On the one hand, the test suite may consist of a considerable number of test cases. On the other hand, simple they may seem, their preparation and execution may require a long time for the test operator. Typical tasks involved in passing a test case include configuring the test equipment, building and sending the adequate protocol data units, monitoring the responses from the system under test, interpreting the results, etc. Furthermore, the possibility of introducing human errors in this process must not be neglected. This paper describes a generic conformance testing tool aimed to solve the above problems. The tool provides a general platform that allows the operator to select a test suite and execute all or part of the test cases without requiring manual intervention. The tool has been designed so that new test suites can be easily added as needed. As an application, several test suites for ATM systems specified by the ATM Forum have been implemented within the tool. KEYWORDS: INTEROPERABILITY, CONFORMANCE TESTING, TEST SUITE, AUTOMATION, ATM. This work has been partially supported by the Spanish CICYT project CEPAT (Certificación de Protocolos ATM).

2 1. INTRODUCTION Standardization in the Telecommunication industry has play a major role in fostering the interoperability between equipment from different manufacturers. In practice, however, it is often the case that implementations drawn from the same standard do not perform well when connected together. Incompatibilities may be due to different causes: programming bugs, misinterpretations of the standard, systems supporting only a subset of the features described in the standards, etc. From the above discussion, it is clear how valuable would be a methodology allowing users, providers and manufactures to know in advance whether two products will interoperate. The ideal solution would be to perform interoperability testing between the particular systems to interconnect. These tasks could be done by a certification laboratory so that efforts would not be duplicated. In practice, however, interoperability testing becomes unfeasible as the number of implementations grows. Obviously, performing interoperability testing for every possible pair of existing products may be exhausting. For that reason, another methodology, known as protocol conformance testing, is used in practice. Protocol conformance testing helps to minimize the chances of incompatibility by defining the test cases that an implementation must pass to be considered compliant with a particular standard. Although less rigorous than interoperability testing, conformance testing has the key advantage of only requiring each product to be tested once for each specification it supports. The benefits of protocol conformance testing have been widely recognized to the point that a standardized methodology, IS 9646 [1], has been internationally agreed. The ISO conformance methodology is being increasingly adopted by the major standardization bodies (e.g. ITU-T [2], ETSI [3]) and other relevant organizations in the standardization field (e.g. the ATM Forum). Thus, nowadays it is common practice to find the definition of the conformance test suites along with the technical specifications of communication protocols. Examples of fields where the IS 9646 methodology has been applied are the Integrated Services Digital Network, the Signaling System No. 7, and the Asynchronous Transfer Mode, to cite just a few of them. The ISO conformance testing methodology has an additional attractive for the people who actually have to perform the testing. This is the possibility of considering the automation of the testing process. The IS 9646 specification contains some characteristics specially adequate for that purpose. To date, however, known works have not exploited the idea to its full extent. The tool described in this paper represents an attempt to complete the automation of the whole conformance testing process. The rest of the paper is organized as follows. In section 2, we review the ISO conformance testing methodology. The possible tasks subject to automation are discussed in section 3, focusing in those aspects not covered by the known tools. Section 4 describes the generic tool for automatic conformance testing that has been developed by the authors. As an application case, section 5 describes how the tool has been used to test ATM equipment. Finally, in section 6 we summarize the main contributions of our work and outline some possible future research directions. 2. ISO CONFORMANCE TESTING This section provides an overview of the ISO Conformance Testing methodology. The objective is not to provide a detailed description of the standard, but to introduce the terminology that will be used through the rest of the paper as well as a general background for beginners. For a tutorial introduction to Conformance Testing see [4,5]. The IS 9646 provides a consistent framework for the specification and execution of conformance testing for implementations of protocol standards. For that purpose, it defines the conceptual architecture shown in figure 1. The particular part of the system that is to be tested is called the Implementation Under Test (IUT). The Upper (UT) and Lower Testers (LT) can be viewed as the Means of Testing (MOT) that are connected to the IUT at the upper and lower layer service interfaces, respectively. The connection is done trough the corresponding Points of Control and Observation (PCO), where the exchange of Abstract Services Primitives (ASP) takes place. Both testers are depicted together as surrounding the IUT to reflect the fact that a test coordination procedure between them is necessary.

3 Test Coordination Procedure Layer N+1 [N] PDUs Layer N-1 PCO PCO Upper Tester (UT) [N] ASPs Implementation Under Test (IUT) [N-1] ASPs Lower Tester (LT) Figure 1. Conformance Testing Model The abstract conformance testing architecture may adopt in practice different realizations. For example, it often happens that the IUT cannot be isolated from the System Under Test (SUT). This may be due either to physical limitations (the SUT inners cannot be accessed) or simply to the fact that there are not real service layer boundaries in the SUT. As a result, the tester will not be able to use PCOs adjacent to the layer under test. In these cases, the problem is usually solved by connecting a separate piece of test equipment to the SUT, either directly or remotely via a network, as depicted in figure 2. Several abstract testing methods based on this approach are described in IS Tester Lower Tester (LT) PCO Test Coordination Procedures [N] PDUs Service Provider System Under Test (SUT) Implementation Under Test (IUT) Figure 2. Conformance Testing using a Separate Piece of Test Equipment Assuming the test method has been decided, the next step is to determine which test cases must be applied to the IUT. According to IS 9646, the test cases are given by the corresponding Abstract Test Suite (ATS) defined for the particular protocol standard that is being considered. An ATS is a standard itself that specifies the set of test cases that a protocol must pass to be considered conformant. As pointed out in the introduction, many ATSs have already been or are being defined by standardization organizations. ATSs are described using an informal language called Tree and Tabular Combined Notation (TTCN), described in part 3 of the IS 9646, which allows test cases to be modeled using an event tree approach. The main part of a test case specified in TTCN, called the dynamic behavior of the test, is an automaton that controls the sequence of service primitives exchanged during the test. The test cases make use of declarations (e.g. constants, parameters, variables) and Protocol Data Unit (PDU) constraints, which are conditions evaluated during the test cases (e.g. to check if a PDU received from the IUT is valid). Although it is not a particularly elegant language, TTCN has the advantages of being easy to learn and supporting a machine processable form. This last feature can be exploited for deriving an Executable Test Suite (ETS) from an ATS. See section 3.1. The IS 9646 methodology also defines some aspects related with the test realization process. Figure 3 illustrates the three basic stages involved in the process, those being: Preparation for testing. In this stage, the adequate abstract testing method and ATS are selected taking into account the capabilities of the IUT. These capabilities are reflected in a standardized document called the Protocol Implementation Conformance Statement (PICS). The information contained in the PICS is also used to complete the Protocol Implementation Extra Information for Testing (PIXIT). This is a document that includes information not specified by the base standard, but indispensable for testing purposes (e.g. particular values for configuring the IUT and the tester). Once the required information have been completed, the next step is the preparation of the IUT and the tester. Test operations. This stage is where the conformity of the IUT to the protocol standard is actually investigated. In a first phase, the PICS is analyzed in order to determine whether the static conformance requirements are satisfied. In case

4 the IUT does not meet these requirements, it is declared as non conformant and the testing process moves to the report production stage. If the static conformance review is satisfactory, a selection of test cases from the ATS is made. The resulting set of test cases will depend on the information contained in the PICS and the PIXIT. This information will also determine the particular parameter values (e.g. timers, addresses) to use when executing the test cases. At that point of the process, is when everything is ready to begin the execution phase or test campaign. PICS PIXIT Static Conformance Results Begin Preparation for Testing - Selection of ATM and ATS - Preparation of IUT and tester Test operations - Static conformance review - Test selection/parameterization - Test campaign Test Report Production End IUT ATM + ATS MOT (ETS) Test Results SCTR PCTR Figure 3. Conformance Assessment Process The test campaign consists in running the selected test cases one after the other, registering the resulting verdicts (pass, fail or inconclusive). As a general rule, if an error happens during a test campaign, it should be started again from the beginning. However, there may be cases where the interruption does not invalidate previously executed test cases. Under that assumption, it is feasible to resume the test campaign at the point where the error was found. Test report production. The goal of the last stage in the conformance assessment process is the production of two documents. These are the System Conformance Test Report (SCTR) and the Protocol Conformance Test Report (PCTR). The format of both documents is defined in IS The SCTR provides an overall summary of the test results for the protocols for which the SUT has been tested. It includes also general information such as the name and version of the SUT or the dates of testing. The PCTR reflects a more detailed information of the testing performed for a particular protocol. This information includes a listing of the test cases that have been run together with their verdicts. 3. AUTOMATING CONFORMANCE TESTS As mentioned in the introduction, one of the advantages of conformance testing is that it reduces the number of test sessions when compared to interoperability testing. That is because conformance testing only requires the IUT to be tested with one other system, namely the test system. There is, however, an additional advantage that may be considered even more attractive, specially to the people actually involved in the testing process. That is the possibility of automating the testing process. It is easy to understand why automation is desirable by analyzing the costs associated to the main tasks involved in the conformance testing process. Let us assume that a test laboratory is asked to test a given protocol implementation according to the corresponding standardized ATS. If no automatic facilities are available, the operator would typically proceed as follows. The first step would be to visually inspect the PICS and the PIXIT to perform the static conformance review. In case the static requirements were satisfied, the operator would proceed to select the applicable test cases from the ATS standard document. Once arranged the testing scenario, which would include the preparation of the IUT and the (manually operated) test equipment, the execution of the test cases starts. Typically, the execution of each test case would involve the following steps: configuring the test equipment, building and sending the adequate PDUs, monitoring and decoding the responses from the system under test, interpreting and writing down the results, etc. Obviously, the above procedure may be quite tedious if manually performed, specially considering that an ATS may consists of a considerable number of test cases. Furthermore, the possibility of introducing human errors during the testing process should not be neglected. From the above discussion, it seems clear how advantageous would be to automate, at least partially, the conformance testing procedure. The

5 benefits would be obvious: shorter testing times, lower costs, higher confidence in the results, repeatability of the tests, etc. Fortunately, the IS 9646 specification contains some characteristics that make the automation idea feasible. Automation may be considered in two main areas within the conformance testing methodology. They are discussed in the following sections. 3.1 Implementation of Abstract Test Cases Test cases defined by an ATS are abstractly specified, so they cannot be directly executed by a real test equipment. As mentioned before, trying to manually run test cases is an exhausting and errorprone task. Consequently, it is important to automate the execution of test cases as much as possible. Fortunately, conformance test cases are specified in TTCN, a language that supports a machine processable form (TTCN-MP). This fact has led to the development of compilers from TTCN to conventional programming languages (e.g. to C language) [6,7]. This kind of tools can be used to obtain the skeleton of the program that models the dynamic behavior of a test case. The generated code may also include calls to functions from a library supporting general tasks such as handling of timers, tracing of messages (e.g. to display a verdict), etc. Sometimes this can be complemented with the use of tools for handling of PDUs (e.g. an ASN.1 compiler for encoding and decoding application layer PDUs). By using the appropriate combination of tools, the process of deriving an Executable Test Suite from the Abstract Test Suite can be highly automated. 3.2 Management of Conformance Test Sessions To date, most of the known efforts to automate the conformance testing methodology have focused in the derivation of ETSs from ATSs. There is, however, another area where automation is helpful as well. It has to do with several aspects related to the management of conformance test sessions. A conformance test session can be defined as a realization of the ISO conformance assessment process for a particular system with regard to a particular protocol standard. Therefore, a given system may be involved in as many testing sessions as protocols it supports. Similarly, there may exist as many testing sessions for a protocol as potential implementations. Typically, a test laboratory will be required to test the same protocol for several systems, or several protocols for the same system. The number of test sessions can be even higher due to the possibility of system and protocol upgrades. As a result, the laboratory may be confronted to the problem of managing a considerable number of test sessions. Several aspects within the management of test sessions may be considered for automation. One of them is the use of databases to store information about individual systems (e.g. name and version number), protocol implementations (e.g. the PICS), abstract test suites (e.g. the TTCN specification of test cases), test sessions (e.g. the test verdicts), etc. The information can be used to automatically perform tasks such as the review of static conformance requirements and the generation of standard reports (SCTR and PCTR). A further automation degree can be obtained by integrating the databases and the test equipment. The advantages of this integrated approach are analyzed in the following section, where a generic tool for automatic conformance testing developed by the authors is described. 4. THE CONFORMANCE TESTING TOOL This section presents the main features of a generic protocol conformance testing platform developed in accordance to the ideas just discussed. The platform incorporates a number of facilities for assisting the test operator through the conformance assessment process. More specifically, it provides an automated testing environment both for the execution of test cases and for the management of test sessions. The tool has been designed following an open approach so that new protocol test suites can be easily added. This allows the operator to select an executable test suite off the shelf and perform a test session on a given protocol implementation. This fact contrasts with other known tools, which are generally protocol-specific. The general architecture of the conformance testing platform is shown in figure 4. The tool has three main parts: the user interface, the test management module and the test execution modules. The functions performed by each module are described in the following paragraphs.

6 User User Interface Module (UIM) Test Management Module (TMM) Conformance Testing Tool Commands Responses Test Session Database Figure 4. Tool Architecture Test Execution Modules (TEMs) ATS c ATS b ATS a ATM_UNI_ES Test Equipment (e.g. ATM Analyzer) User Interface Module (UIM). The UIM allows the operator to interact with the conformance testing platform through a window-based graphical interface. The user interface provides the test operator with a number of controls (menus, text windows, etc.) for performing test sessions. The meaning of the controls and how they are used during a typical test session, will be presented later. By using the UIM controls, the user can select an ATS, load an IUT, request the execution of a test case, save and print test results, etc. These actions are actually carried out by the underlying module, the TMM, described below. Test Management Module (TMM). This module represents the heart of the conformance testing platform, being responsible of managing all aspects related to the test sessions. The TMM functions can be classified into two major categories: database handling functions and test driving functions. The TMM makes use of the database approach discussed in section 3.2. The database handling functions are used to store/retrieve information related to an ATSs, an IUTs, or a test session. These functions are automatically invoked as a consequence of the stimuli received from the UIM or the TEMs (see below). The test driving functions are used to control the execution of test cases during a test session. When the user requests a test case to be executed, the TMM invokes the corresponding TEM. During the test execution, the TMM monitors the responses sent back by the TEM, which may be trace messages (e.g. to keep track of the TTCN code) or the test verdict (pass, fail, inconclusive). The TMM processes these responses in order to notify the UIM and to update the database. The repertory of messages exchanged between the TMM and the TEMs is shown in table 1. TMM to TEM TEM to TMM Table 1. TMM/TEM Interface Messages Type Control Control Verdicts Traces Message EXEC STOP STOP PASS FAIL INCONCLUSIVE NOTAPPLY TTCN PDU INFO ERROR Meaning request the test case execution to be started request the test case execution to be aborted notifies that the test case execution has ended verdict pass verdict fail verdict inconclusive undefined verdict, selection reference failed ttcn code pdu contents debug info error message Test Execution Modules (TEMs). Using the IS 9646 terminology, a TEM is equivalent to an Executable Test Suite. Each TEM concentrates all aspects that are specific to the ATS it implements. Therefore, there will be a separate TEM for each ATS supported by the tool. As mentioned above, the execution of the test cases is driven by the TMM. The execution itself is performed by the TEMs, which implement the dynamic behavior of the test cases and interact with the test equipment. Therefore, each TEM depends both on the ATS and on the particular piece of test equipment that is being used. Section 5 describes an example of TEM developed for the conformance testing tool. Figure 5 shows the general aspect of the UIM. Typically, right after starting the user interface, the user will load an ATS and an IUT. This is done through the pull-down menus Test Suite and Implementation, which open the appropriate temporary file selection windows. Once the ATS and the IUT have been selected, the tool checks for the existence of a previous test session. If that is the case, the tool recovers the status of the test session from disk, displaying the corresponding verdicts, log traces, etc. If no test session exist for the selected ATS-IUT combination, a new one is created. The top window of the interface shows a summary of the test cases together with the verdicts (either pass, fail, or inconclusive) for the cases that have been already executed. Otherwise, the entry in the

7 verdict column for test cases not executed will be either empty or showing a Do not apply message. The latter means that the test case cannot be executed because the IUT (i.e. its PICS or PIXIT) does not meet the selection reference clauses described in the ATS. The user can select individual test cases by clicking on the corresponding line. The line is then highlighted and a textual description of the selected case is displayed in the middle window. Test Suite Implementation Session Help Test suite: ATM_UNI_ES Sample_IUT Implementation: Test Name Verdict Date 1 Ver_Gen_GFC0_VP Inconclusive 13/12/97 09:31:06 2 Ver_Gen_GFC0_VC Inconclusive 13/12/97 09:31:10 3 Ver_Gen_Unass_Cell Pass 16/12/97 12:56:59 4 Ver_Disc_OAM_Seg_VP_LB Don t apply 13/12/97 09:31:14 Test case name: Ver_Gen_Unass_Cell Purpose: Verify that if the IUT has a physical layer that has synchronous cell time slot, it generates and adds unassigned cell to the assigned cell stream to be transmitted, transforming a non-continuous cell stream of assigned cells into a continuoues stream of assigned and unassigned cells. Test case Ver_Gen_Unass_Cell started at 16/12/97 12:56: Label Behaviour Description Constraints Ref V START T_Test LT_PCO?CELL CELL_UNASSIGNED P Test case Ver_Gen_Unass_Cell ended at 16/12/97 12:56:59 Exec Cancel Log On Batch Off 00:00:03 Elapsed time: Figure 5. Graphical User Interface The execution of individual test cases can be initiated by clicking twice in the corresponding line or by pushing down the Exec button. During the test execution, the bottom window shows traces of the actions performed by the TEM. The user can individually activate or deactivate three types of tracing information: TTCN code, PDU contents and execution progress messages. The execution of a test case can be aborted at any time by pushing the cancel button. The operator can also request the execution of several test cases in batch mode. This operation mode requires the Batch toggle button to be set to on. The set of test cases to execute can be selected either manually or by specifying one ore several matching criteria from a list. Examples of predefined categories are tests not executed, test requiring/not requiring operator intervention, test with verdict FAIL,.etc. One the selection has been done, the execution can be started by pressing the Exec button. This action causes the selected test cases to be sequentially executed by the tool. The tool includes a number of additional features such as the test report generation (SCTR and PCTR), handling of PICS and PIXIT, etc. This section has tried only to summarize some of the main features of the tool. The purpose is not to provide a detailed description of the tool (which can be found in [8]) but to emphasize the huge potential that can be obtained by using an automated conformance testing environment. The benefits are better understood by considering the application of the tool to a real case. This is precisely the subject of the following section, which describes the implementation of an ATS for testing ATM equipment. 5. TESTING OF ATM END SYSTEMS The number of protocols for which standardized ATSs have been or are being defined is continually increasing. This trend has reached, as it could be expected, to the family of protocols related to the Asynchronous Transfer Mode (ATM). So, one of the major organizations in the standardization of ATM protocols, the ATM Forum, has specified to date several ATSs [9]. This section describes the implementation of one of the ATSs defined by the ATM Forum, namely the Conformance ATS for the User Network Interface 3.1 ATM Layer of End Systems [10]. This ATS, from now on ATM-UNI-ES, defines a total of thirty seven test cases intended for testing the ATM Layer for end systems at the UNI 3.1 [11]. (ATSs for the ATM UNI 4.0 have not been approved yet.) The test cases are grouped into three categories as follows: General. This test group deals with the ATM cell structure and encoding at the UNI. The objective is to verify general connectivity issues as, for example, that the IUT encodes all GFC bits to zero in a VPC. Cell Discrimination. The purpose of this test group is to verify the ability of the IUT to perform cell discrimination based on VCI and PTI values. Management Plane. This test group tries to determine whether the IUT performs management plane functions according to the specification. The test cases verify things such as the OAM cell formats, fault management flows, etc. The preparation of a typical test case requires building the ATM cells to be sent with the header

8 and the payload contents as specified in the ATS. The execution requires sending the adequate cells and monitoring whether the cells received in response are the expected ones and they arrive in time. Although most ATM analyzers include facilities to build, send, capture, and examine ATM cells, it is evident that a manual testing procedure may result overwhelming. As an alternative, an executable version of the ATM- UNI-ES ATS has been implemented and integrated as a TEM within the generic conformance testing tool described in the previous section. The TEM allows test cases to be automatically executed, reducing manual interventions to a minimum. In particular, by selecting the batch mode, a test session can be performed in less than five minutes. Figure 6 shows the test configuration considered during the development of the TEM. According to the ATM-UNI-ES specification, the configuration is based on the remote test method discussed in section 2 (see figure 2). The PCO is placed immediately below the protocol layer under test, that is, between the ATM layer and the Physical layer. Consequently, test cases are based in the exchange of ATM cells between the test equipment and the IUT. Recall that the remote test method considers a separate piece of test equipment connected to the SUT. In our case, the test equipment was an ATM analyzer where the conformance testing modules (UIM, TMM, and TEM) were implemented. Tester (ATM Test Equipment) Tester ATM PHY PCO PDUs ATM Cells System Under Test (ATM End System) IUT ATM PHY Figure 6. Test Configuration for the ATM-UNI-ES ATS Note that, although not shown in the figure, the test equipment can be also indirectly connected to the ATM end system, via one or several intermediate systems. The software architecture of the ATM-UNI-ES TEM consists of four modules. See figure. Test Session Database IUT parameters ATM Board Configuration Commands Test Management Module (TMM) Test Interface Module (TIM) Test Behavior Module (TBM) ATM Board API ATM Board Responses Test Case Selection Reference Test Declarations and Constraints Module (TDM) ATM_UNI_ES Test Execution Module Figure 7. Architecture of the ATM-UNI-ES Test Execution Module The Test Interface Module (TIM) contains all the aspects that are not specific of the ATM-UNI-ES. These modules perform the communication with the TMM (with the messages presented in section 4) and the access to the test session database in order to read the values of the IUT parameters required for the execution of the test cases. The execution of the test cases is driven by the TIM, which uses the TBM (Test Behavior Module) and the TDM (Test Declarations and Constraints Module). These two modules contain the ATM-UNI-ES specific functions resulting from the translation of the TTCN specification into C language. The translation was manually performed because of the small number of cases included in the ATS. In the future, a semiautomatic translation procedure based on existing TTCN tools will be considered for adding new ATSs. The TBM implements the dynamic part of the ATM-UNI-ES specification, that is, the state machines that model the behavior of the test cases. During the test case execution, the TBM makes use of the TDM, which contains the declaration and the constraint parts. The selected ATS has 37 cases only. Some test suites, e.g. the ones defined for signaling protocols, typically have several hundreds of cases.

9 The execution of a test case usually involves the exchange of two or three ATM cells between the ATM board and the SUT. The Application Programming Interface (API) allows the TBM to interact with the ATM board for sending and receiving ATM cells. The steps that take place during the execution of a test case can be described as follows. The process is initiated by the TMM, which invokes the ATM- UNI-ES TEM with the IUT name and the Test Case name as arguments. The former is used by the TIM to retrieve the parameter values of the IUT from the test session database. Before proceeding with the test case execution, the selection reference clause is evaluated by the TDM for the current combination of parameter values. If the selection reference fails, the TIM sends the messages NOTAPPLY and STOP to the TMM, and the test case execution ends. In case the selection reference clause is matched, the TIM uses the API to configure the ATM board for capturing cells on the appropriated VPI/VCI pairs. The values of the VPI/VCI pairs are determined on real-time from the IUT parameter values. Once the ATM board has been configured, the TIM invokes the TBM to perform the test case dynamic behavior. This means successive calls to the C function that implements the TTCN specification of the test case, until the verdict is obtained. 6. SUMMARY AND CONCLUSIONS Nowadays, we are assisting to a spectacular growth in the number of communication networks based on the interconnection of multivendor equipment. In practice, however, real systems may not interoperate when connected together, despite the claims made by the manufacturers about their products being in conformance to the standards. The ISO Conformance Testing methodology was developed to provide a consistent framework for determining whether a given equipment complies with the standards, and to what extent. This is achieved by specifying the test cases that implementations must pass to be considered in conformance with a particular standard. The success in passing conformance test cases reduces the probability of interoperability problems when different implementations are interconnected. An additional advantage of the ISO Conformance Testing is the possibility of automating the conformance assessment process. Two main areas of automation can be identified in this process. One has to do with the implementation of Executable Test Suites, so that test cases can be automatically run by the operator. The other automation area, which is equally important but which has received less attention, has to do with the management of test sessions. It covers the many aspects involved when considering the conformance assessment process for systems supporting several protocols or for different implementations of a same standard protocol. This paper has shown the benefits of generalizing the automation of the conformance testing methodology rather that individually considering its application to specific protocols. A generic conformance testing tool following this approach has been implemented. The tool provides a graphical user interface which allows the test operator to perform test sessions for different systems and standard protocols, reducing manual interventions to a minimum. During the test session, the user can request the execution of one or several test cases by simply selecting them with the mouse. The traces of the execution and the resulting verdicts can be visualized on real-time and saved for posterior retrieval. In particular, they can be used to automatically generate standard test reports. One of the key features of the tool is its flexibility for supporting new protocol test suites. As an example, several test suites for ATM systems defined by the ATM Forum have been implemented and integrated within the tool. Although the implementations have been done by hand, the use of a TTCN compiler is being considered for the future. Current work includes the improvement of the tool in order to support interoperability test suites. For this purpose, the Interoperability Abstract Test Suite for the ATM Layer [12] has been chosen as a case study. In the future, new conformance and interoperability ATSs will be implemented, possibly using a TTCN compiler. Note that, although the ATSs considered up to now were defined for ATM, the platform is prepared to handle ATSs for any protocol, for example Signaling System No. 7, GSM, ISDN, etc.

10 REFERENCES [1] ISO, OSI - Conformance Testing Methodology and Framework, ISO/IEC 9646, [2] ITU-T, OSI conformance testing methodology and framework for protocol Recommendations for ITU-T applications, Recommendations X.290 to X.296, [3] ETSI, Conformance Testing Specifications: Standardization Methodology, ETS , [4] K.G. Knightson, OSI Protocol Conformance Testing - IS 9646 Explained, McGraw-Hill Series on Computer Communications, [5] B. Baumgarten, A. Giessler, OSI Conformance Testing Methodology and TTCN, North Holland, [6] Telelogic AB, ITEX - Interactive TTCN Editor and Executor, Version 3.1, [7] Expert Telecoms, TTCN Expert Compiler, Expert Telecoms GbmH, Stuttgart. [8] M. Alvarez-Campana, E. Vazquez. Protocol Conformance Testing Tool for ATM Systems, DIT-UPM, Mayo [9] List of Approved ATM Forum Technical Specifications, ATM Forum WWW Server, December [10] Conformance Abstract Test Suite for the UNI 3.1 ATM Layer of End Systems, ATM Forum Technical Committee, Doc. af-test , June [11] ATM User-Network Interface Specification, Version 3.1, Doc. af-uni , ATM Forum, [12] Interoperability Abstract Test Suite for the ATM Layer, Doc. af-test , ATM Forum, April 1995.