Ch7 Conformance Testing Methodology

Transcription

1 Outline VII. Conformance Testing Methodology General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process Dept. Electrical & Information Engineering, Computer Engineering Lab 7-1 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-2 Dr. Junzhao Sun Outline General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process What is Conformance Testing? The process of testing the extent to which implementations of protocol entities adhere to the requirements stated in the relevant standard or specification Concerned with external behavior (black box) Prerequisites for interoperability The primary objective of conformance testing is to increase the probability that different product implementations actually interoperate. Testing of implementations But NOT with performance, reliability, fault tolerance, efficiency, etc. ISO 9646: A conforming implementation is one which satisfies both static and dynamic conformance requirements, consistent with the capabilities stated in the Protocol Implementation Conformance Statement (PICS) Dept. Electrical & Information Engineering, Computer Engineering Lab 7-3 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-4 Dr. Junzhao Sun

2 Standard ISO/IEC 9646 (ITU-T X.290) Conformance Testing Methodology and Framework (CTMF) for the Implementations of OSI and ITU Protocols 7 part standard ISO General Concepts ISO Abstract Test Suite Specification ISO Tree and Tabular Combined Notation (TTCN) ISO Test Realization ISO Conformance Assessment Process ISO Protocol Profile Test Specification ISO Implementation Conformance Statement Conformance Testing Limitations Aspects of protocol testing not considered in ISO 9646 Interoperability testing, performance testing and robustness testing The realization of service primitives iti Internal aspects of the system implementation The system environment and testing tools Procurement procedures Test certificates, and accreditation of test labs Conformance testing will always be limited, for technical and theoretical as well as for economic reasons No guarantee can be given that an implementation absolutely conforms to the relevant protocol standards Conformance testing with positive test results will increase the degree of confidence in a protocol implementation Dept. Electrical & Information Engineering, Computer Engineering Lab 7-5 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-6 Dr. Junzhao Sun Basic Standard vs. Profile Base standard is a standard relative to a single protocol in a given OSI layer A profile is a set of one or more base standards necessary for accomplishing a particular function a multilayer stack of several adjacent base standards Function might be a type of application, or a transport service based on specific telecommunications techniques A protocol can be a base standard by itself, or can belong to a profile The purpose of defining a profile is to facilitate interoperability Profile is twofold Multiplicity: a profile may retain options in a set of base specifications Refinement: a profile restricts the choices of options in the base specifications. It may also specify additional conformance requirements. We will mainly concentrate on the conformance testing of protocol Conformance Testing Process (1/2) Test generation To generate the abstract test suites (ATSs) manually or automatically Development of proformas for implementation conformance statements (ICS) and implementation extra information for testing (IXIT) Test realization The realization of test tools and means of testing, capable of executing particular test suites To generate the executable test suites (ETSs) with ATSs Including both selection and parameterization process Test execution Perform test to an implementation with executable test case selected Conformance assessment process Performed at a test laboratory for a client, with a protocol conformance test report (PCTR) as the ultimate result Dept. Electrical & Information Engineering, Computer Engineering Lab 7-7 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-8 Dr. Junzhao Sun

3 Conformance Testing Process (2/2) Standard protocol specification Protocol implementation System Under Test Test generation Abstract test suite Executable test suite Tester TTCN Test realization Test execution Verdict: (result) accept failure inconclusive error (in test HW) Dept. Electrical & Information Engineering, Computer Engineering Lab 7-9 Dr. Junzhao Sun Test Types Test types in testing process Basic interconnection tests (BIT) To determine whether interconnection is possible at all as an initial stage To establish main features of the protocol are implemented Basic interconnection tests are standardized but may be omitted Capability tests To validate that all the the observable external static capabilities of the are valid with respect to the static conformance requirements expressed in the PICS Capabilities tests are standardized and mandatory Behavior tests To check dynamic conformance of an with respect to the dynamic conformance requirements specified in the protocol standard Behaviour tests are standardized and mandatory Conformance resolution tests Special tests for purposes beyond ordinary behavior testing e.g. exception Conformance resolution tests are not standardized and not mandatory Dept. Electrical & Information Engineering, Computer Engineering Lab 7-10 Dr. Junzhao Sun Test Types Test Results Behavior tests Capability tests Basic interactioi n tests Conformance resolution tests Test results Pass Observed test outcome shows that the test purpose has been fulfilled and the has displayed only valid behaviors as specified in protocol standards Fail Test outcome proves that some conformance requirement has been violated Inconclusive Neither pass nor fail can be assigned to the test outcome Happens when underlying provider signals problems e.g. disconnect or reset Documentation of test results SCTR = system conformance test report PCTR = protocol conformance test reports Requirements on test results Repeatable, comparable and auditable to a reasonable degree Dept. Electrical & Information Engineering, Computer Engineering Lab 7-11 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-12 Dr. Junzhao Sun

4 Outline Test Generation General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process Standard Standardized Test Purposes (in English) TPs Standardized (Protocol) Implementation Conformance Statement (tables) Standardized Abstract Test Suite (in TTCN) ATS ICS Standardized (Protocol) Implementation extra Information for Testing (tables) IXIT Dept. Electrical & Information Engineering, Computer Engineering Lab 7-13 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-14 Dr. Junzhao Sun Test Specification to Be Produced Conformance Requirements Systems are called conforming if their externally visible behavior fulfils conformance requirements stated in protocol specification Mandatory requirements, always have to be observed Conditional requirements, have to be observed under conditions defined Dependence on previous choices of parameter values and options Optional requirements, can be freely selected Static conformance requirements Specify permitted combinations of capabilities implemented in real system E.g. value ranges for size parameters or timers, as in ICS Dynamic conformance requirements Specify the observable behavior of implementation permitted by standard Define actual protocol e.g. PDU exchanges, timing, format of PDUs. Profile is based on base specifications with additional constraints Dept. Electrical & Information Engineering, Computer Engineering Lab 7-15 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-16 Dr. Junzhao Sun

5 ICS (1/2) ICS = Implementation Conformance Statement A checklist of the capabilities supported by the A questionnaires i completed by client to present conformance requirements Produced by filling out an ICS proforma with static requirements Provides an overview of the features and options that are implemented Used to select and parameterise test cases and as an indicator for basic interoperability between different products PICS = Protocol Implementation Conformance Statement realizes a single protocol Profile ICS contains a set of interrelated protocols, e.g. a stack of protocols ICS is not just for testing Used by specification writer to clarify the intent of standards Used by product implementer for selection of features ICS (2/2) Standardization organizations Base specification Base specification Base Base specification specification Profile specification PICSproforma PICS- PICSproforma PICSproformproforma Profile RL Profile specific ICS proforma RL = Requirements List Client Profile ICS PICS PICS PICS PICS Profile RL Profile specific ICS Dept. Electrical & Information Engineering, Computer Engineering Lab 7-17 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-18 Dr. Junzhao Sun ICS Proforma Protocol Capability Status of capability in protocol specification Mandatory, optional, conditional prohibited, excluded d out-of scope, not applicable Capabilities can be grouped into categories Major capabilities Supported PDUs and fields within PDUs Supported timers Encoding variations Negotiation mechanisms Error procedures Dept. Electrical & Information Engineering, Computer Engineering Lab 7-19 Dr. Junzhao Sun ICS Proforma Structure ICS proforma corrigenda (version information) ICS proforma instructions, abbreviations and terms Identification of the implementation, supplier and client Identification of the protocol, information object or profile Protocol, information object or profile corrigenda Major capabilities table Supported PDUs Supported Supported Table PDUs Table PDUs Table Supported parameters Supported parameters Supported table in PDU parameters table in PDU table in PDU Supported timer tables Supported protocol error handling table PICS proforma tables Supported negotiation capabilities table Multi-specification dependencies table Dept. Electrical & Information Engineering, Computer Engineering Lab 7-20 Dr. Junzhao Sun

6 ICS Proforma ISDN Example IXIT, SCS Index Protocol capability Does the implementation support Conditions for status Status Reference Support M1 3.1kHz audio calls O [] Yes [ ] No M2 Data calls O [ ] Yes [ ] No M3 Packet mode calls O 6.3 [ ] Yes [ ] No M4 Interworking with non-isdn networks M 8.4 [ ] Yes [ ] No M41 Calling line identification presentation M 8.4 [ ] Yes [ ] No M421 Calling line identification restriction M1 not M1 M422 Advice of charge M1 not M1 M N/A M N/A 8.4 [ ] Yes [ ] No 8.4 [ ] Yes [ ] No Dept. Electrical & Information Engineering, Computer Engineering Lab 7-21 Dr. Junzhao Sun IXIT = Implementation extra Information for Testing Contains additional information beyond specification as ICS An IXIT proforma filled out by testt laboratory and client With both protocol IXIT and profile IXIT, as in ICS types Specify additional testing information related to an specification Physical setup, connection of test, and testing environment SUT hardware, socket, telephone numbers, and tester SAP addresses Test-specific timer or parameter values SCS = System Conformance Statement List references to specifications of the, related ICSs, related testt reports, together with configuration information of SUT, I.e. an index Conformance of an : An is said to be conforming if it satisfied the static and dynamic requirements of the relevant standards referenced in SCS and capabilities in ICSs. Dept. Electrical & Information Engineering, Computer Engineering Lab 7-22 Dr. Junzhao Sun IXIT Proforma ISDN Example TSS & TP #PC_BASIC #PC_POINT_TO_POINTPOINT TO POINT #PC_POINT_TO_MULTIPOINT #PC_EN_BLOC_RECEIVING #PC_OVERLAP_RECEIVING #PX_CDPN_OCTET3 #PX_CPN_PTC1 #PX_LCPN_PTC2 #PX_CPN_PTC2 TRUE FALSE TRUE TRUE TRUE 81 O O 07 O O TSS = Test Suite Structure TP = Test Purposes TSS & TP are derived from the relevant base standards. Provide an informal, easy-to-read description of each test case Concentrate on the meaning of the test rather than detailing how it may be achieved TP is a textual description of a single requirement or a set of related requirements which should be tested Test Purposes are grouped into a logical Test Suite Structure according to suitable criteria, e.g., basic interconnection, error handling, functionality etc. Will be introduced later in ATS section Dept. Electrical & Information Engineering, Computer Engineering Lab 7-23 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-24 Dr. Junzhao Sun

7 ATS & ETS Test Suites & Specification ATS = Abstract Test Suite The entire collection of Test Cases (ABSTRACT) Each Test Case specifies the detailed d coding of the Test Purposes, written in the standardized test specification language TTCN Can be generated manually from protocol and service standard Can be generated automatically from formal specification of protocol CATG = Computer-Aided Test Case Generation Some tools are commercially available to generate test suite from e.g. SDL Will be studied later in more detail ETS = Executable Test Suite Test cases for truly test execution Can be quickly and easily implemented from the ATS using the TTCN compilers available on most modern test tool platforms Dept. Electrical & Information Engineering, Computer Engineering Lab 7-25 Dr. Junzhao Sun Tests apply to a single system under test (SUT), and one protocol standards Exhaustive test of standards specifications Must cover all features mandatory & optional Must cover valid, invalid, & inopportune protocol exchanges Use of standardized test specification may Improve transparency and increase of the objectiveness for the test process Provide comparability of test results As the criteria for accreditation bodies to assess test laboratories Common test specifications are the basis for harmonized test services Provide a common starting point to suppliers, users, test laboratories and certification bodies Test specification language TTCN Dept. Electrical & Information Engineering, Computer Engineering Lab 7-26 Dr. Junzhao Sun Test Specification Development in ETSI Typical Test Suites GSM, GPRS, 3G (UMTS in 3GPP) Broadband Radio Access Networks (HiperLAN) DECT(Digital Enhanced Cordless Telecommunications) Bluetooth Local Loop, Digital Subscriber Line (DSL) IN, INAP (Intelligent Network Application Protocol) ISDN, B-ISDN SMS in PSTN/ISDN SIP (Session Initiation Protocol), IPv6 IP Cablecom XML (Extensible Markup Language) Dept. Electrical & Information Engineering, Computer Engineering Lab 7-27 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-28 Dr. Junzhao Sun

8 Outline ATM General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process ATM = Abstract Test Methods NOT Automatic Teller Machine NOT Asynchronous Transfer Mode A set of ATMs are defined by OSI testing methodology, describing Abstract test configuration Possible types of observation and control Providing for test coordination ATMs are determined by Configuration of with respect to SUT Entire testing configuration, I.e. test system + SUT + service provider Where tester can control and observe the Difference degrees of precision of the test coordination procedures Common test architecture and 4 test methods Dept. Electrical & Information Engineering, Computer Engineering Lab 7-29 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-30 Dr. Junzhao Sun OSI System & OSI Terminology Complete system Application Presentation Session Transport Network Partial system Application Presentation Session Transport Network Application relay system Application Presentation Session Transport Network Network relay system Application Presentation Session Transport Network Layer S Layer T Service provided PDU Service used Layer S Layer T Session layer SP SAP SP Transport layer Data link Physical Data link Physical Data link Physical Data link Physical Layer N Layer N Network layer implement one or a combination of OSI protocols from one layer or several adjacent layers Dept. Electrical & Information Engineering, Computer Engineering Lab 7-31 Dr. Junzhao Sun SP : Service Primitive SAP : Service Access Point PDU : Protocol Data Unit Dept. Electrical & Information Engineering, Computer Engineering Lab 7-32 Dr. Junzhao Sun

9 Conformance Testing Terminology Conceptual Test Architecture (1/3) Upper Tester Test operator interface Test System (TS) Product operator interface System Under Test (SUT) PDU ASP ASP ASP ASP PCO PCO ASP: Abstract Service Primitive PCO: Point of Control and Observation : Implementation Under Test PDU: Protocol Data Unit Lower Tester Control Function TCP Lower Tester Lower Tester Lower Tester Test Co-ordination Procedures (TCP) PDU Upper Tester Upper Tester Upper Tester PCO Lower Tester The PCO has two FIFO queues: Send (from tester to ) Receive (by tester from ) A PCO maps to a SAP (Service Access Point) in the OSI reference model PCO Lower level service provider There can be several LTs and UTs being simultaneously used Dept. Electrical & Information Engineering, Computer Engineering Lab 7-33 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-34 Dr. Junzhao Sun Conceptual Test Architecture (2/3) Testing contexts Single-party testing is for testing an which communicates with exactly one real open system, represented by a single lower tester Multi-party testing is for testing an communicating with multiple real open systems, represented by more than one lower tester Configuration for components can be homogeneous or heterogeneous Lower tester (LT) controls and observes the s lower service boundary, indirectly, via the underlying service provider In single-party testing, behaves as the peer entity to In multi-party testing, act as peer entities working in parallel Lower tester control function (LTCF) coordinating all LTs Assign the test case verdicts Mandatory in multi-party context, inapplicable in single-party context Conceptual Test Architecture (3/3) Upper tester (UT) controls and observes s upper service boundary, by operator access, API, or hardware interface In single-party context, t UT behaves as a user of In multi-party context, UTs working in parallel act as users of Test coordination procedures (TCPs) are used to ensure cooperation between the UTs and LTs How tester shall respond Passing (preliminary) results Synchronisation TCP is NOT Transport Control Protocol, as in TCP/IP Dept. Electrical & Information Engineering, Computer Engineering Lab 7-35 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-36 Dr. Junzhao Sun

10 ATM Classification Test Case ATMs for multi-party testing Several parallel upper and lower testers In complex situation ti a upper tester t control function (UTCF) is needed d Special cases include only one upper tester, or even no upper tester at all ATMs for single-party testing Local Test Method (L) Upper Tester and Lower Tester in Test System Distributed Test Method (D) Upper Tester in SUT, Lower Tester in Test System Co-ordinated Test Method (C) As above, uses Test Management Protocol Remote Test Method (R) Lower Tester in Test System, no Upper Tester Lower Tester ConnectRequest CR ConnectConfirm CC Upper Tester ConnectIndication CONind ConnectResponse CONresp Recall service primitives Request Indication Response Confirm Dept. Electrical & Information Engineering, Computer Engineering Lab 7-37 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-38 Dr. Junzhao Sun Local Test Method Local Test Method TCP coordination UT N-layer entity under test LT System Under Test (SUT) PCO 2) CONind 3) CONresp Test System (TS) Upper Tester TCPs Lower Tester Upper Tester is located in Test System Requires an upper interface on is built in the tester No ATSs for this method Good for the testing of a hardware component Example: Ethernet driver 1) CR 4) CC PCO Dept. Electrical & Information Engineering, Computer Engineering Lab 7-39 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-40 Dr. Junzhao Sun

11 Distributed Test Method Distributed Test Method LT TCP UT PDUs Service provider System Under Test (SUT) Upper Tester 2) CONind 3) CONresp PCO Test Co-ordination Procedures (TCP) Test System (TS) Lower Tester UT in SUT, LT remote Requires synchronisation Suitable for upper layer protocols / protocols offering API Example: socket communication 1) CR 4) CC PCO Dept. Electrical & Information Engineering, Computer Engineering Lab 7-41 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-42 Dr. Junzhao Sun Co-ordinated Test Method Co-ordinated Test Method LT TM-PDUs PDUs UT System Under Test (SUT) 2) CONind 3) CONresp Test Management Upper Tester Protocol Test System (TS) Lower Tester Service provider UT in SUT but no access, LT remote No assumption of upper interface to the Use only one PCO below the LT Uses Test Management Protocol (TMP) embedded in Suitable for mid-layer protocols 1) CR 4) CC PCO Dept. Electrical & Information Engineering, Computer Engineering Lab 7-43 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-44 Dr. Junzhao Sun

12 Remote Test Method Remote Test Method LT TCP PDUs UT System Under Test (SUT) Test System (TS) Lower Tester Service provider No Upper Tester Upper Tester can be native application or a user accessible interface Manual co-ordination Limited, but easy to use 1) CR 2) CC PCO Dept. Electrical & Information Engineering, Computer Engineering Lab 7-45 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-46 Dr. Junzhao Sun ATMs Put Together Testing Configuration Examples MHS MHS = Message Handling Systems Multi-user in single-party context RTS = Reliable Transfer System LT TCP UT-2 UT-1 MHS- RTS- MHS- PDUs RT service provider Dept. Electrical & Information Engineering, Computer Engineering Lab 7-47 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-48 Dr. Junzhao Sun

13 Testing Configuration Examples Multiplexing Multiplexing happens in transport or network layer protocols where more than one connection is multiplexed over a single lower connection, e.g. transport connections over network one Testing Configuration Examples Relay Relay systems testing should be in multi-party context, without using UTs Network- LT TCP UT-2 UT-1 NW- PDUs Service provider Transport- LT 1 PDUs Service provider TCP LT 2 PDUs Service provider Dept. Electrical & Information Engineering, Computer Engineering Lab 7-49 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-50 Dr. Junzhao Sun A Example Test System Architecture Outline TC Editor TC Translator Protocol Tester Interface PICS,PIXIT Editor Test Select Result Analysis&report SUT General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process TC Library Transmit Test Execute Receive Service Provider Dept. Electrical & Information Engineering, Computer Engineering Lab 7-51 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-52 Dr. Junzhao Sun

14 ATS Development Process ATS = Abstract Test Suites 1. Determine the testable conformance requirements from the relevant specifications, ICS and related documents 2. Determine the test groups & objectives that cover these requirements 3. Develop test purposes covering requirements and test group objectives 4. Choose testing context (single-/multi-party) and an ATM (configuration) A test suite is always dedicated to a single test method 5. Specify ATS, including test steps, using standardized notation like TTCN 6. Specify the selection and parameterisation criteria with respect to the ICS and IXIT proformas ATS should be used to derive ETS by compilation and adaptation to the means of testing (MOT) MOT is the combination of test devices, equipments and procedures Recall the Test Generation Process Standard ICS Standardized Test Purposes (in English) TPs Standardized (Protocol) Implementation Conformance Statement (tables) Standardized Abstract Test Suite (in TTCN) Standardized (Protocol) Implementation extra Information for Testing (tables) ATS IXIT Dept. Electrical & Information Engineering, Computer Engineering Lab 7-53 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-54 Dr. Junzhao Sun ATS Structure (1/2) ATS Structure (2/2) Test suite Test group Test group Test Test case Test case Test case case Test step Test Test case case Test case event Test group Test case Test case Test case Test step Test case Test case Test event Test step Test group Test Test case Test case Test case case Test step Test Test case case Test case event Test Test case Test case Test case case Test Test case case Test case event A test suite consists of test cases with no execution order Test cases are logically grouped into test group according to test purposes Each test t case has an individual id test t purpose to demonstrate t a certain behavior A test group objective may be provided for each test group The nested hierarchy should be created in a top-down fashion Ensure systematic derivation of test purposes Achieve good coverage of all protocol aspects and conformance requirements A test case may be modularized into nesting test steps Modularization facilitates test case development and maintenance Group commonly used test steps together into test step libraries Economical that they need not be specified repeatedly Test event is the elementary unit within a test case Usually the exchange of a PDU or an ASP at a PCO Dept. Electrical & Information Engineering, Computer Engineering Lab 7-55 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-56 Dr. Junzhao Sun

15 TSS = Test Suite Structure TSS FTAM Example which tests are performed on which aspects of the reference specification Adopt a top down approach deriving TSS from base specification when no technical constraint forbids it, structure TSS as a tree 1. The name of base specification should be placed at the top of the TSS 2. Test groups related to the "major functions", or "roles" of the base specification 3. Pre-defined groups of tests according to the "nature" of the tests: Basic interconnection tests; capability tests; Valid behaviour tests; t invalid behaviour tests; t inopportune behaviour tests; t 4. Pre-defined groups of tests according to the "functional aspect" tested: State event transitions; parameter variations; parameter combinations; timers 5. Groupings relevant to the base specification E.g. name of functional units, then of PDUs FTAM = File Transfer Access and Management Dept. Electrical & Information Engineering, Computer Engineering Lab 7-57 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-58 Dr. Junzhao Sun Test Purposes (1/2) Test Purposes (2/2) Each test case with one test purpose for meaning description Test purposes focus on one or more related conformance requirements, derived from OSI specification Test purposes and their hierarchical grouping are specified in a TSS & TP (Test suite structure and test purposes) standard Structure chosen depends largely on personal experience and style Test purposes for a test suite have to cover aspects of protocol Capabilities, in particular mandatory and optional ones Behavior tests, with the tester displaying valid/invalid behavior PDUs sent to/received from, and relationship between the PDUs Protocol phases and protocol machine states Timing, encoding, individual parameters and parameter combinations The expression of a TP shall not be limited to checking a state or a state transition of a protocol state machine Example TP: "Check that t the sends a COMMON CONTROL ACK after receipt of a COMMON CONTROL message containing..., as a result of state transition S1 S2.". A clear explanation of the relation between TPs and the conformance requirements in the reference specification shall be provided The TPs shall be designated by the same numbering and referencing scheme as the TSS Dept. Electrical & Information Engineering, Computer Engineering Lab 7-59 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-60 Dr. Junzhao Sun

16 Combined TPs (CTPs) TP Combination Principle The final cost of a test campaign directly depends on the size of the ATS The objective of CTP is to reduce the number of test cases, but to keep a good coverage Dept. Electrical & Information Engineering, Computer Engineering Lab 7-61 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-62 Dr. Junzhao Sun TSS for CTP Test Purpose Component Dept. Electrical & Information Engineering, Computer Engineering Lab 7-63 Dr. Junzhao Sun Initial condition should define precisely the expected state of the at the beginning of the test Example: Ensure that the is in state connection established (reception of a connect confirm on the LT on response to a connect request). Check(s) to be performed Example: Ensure that the, on reception of a valid SETUP message with the Sending complete information element sends any of a CALL PROCEEDING, ALERTING or CONNECT message. Verdict criteria should explicitly describe the observable events which will be used to assess the result of the test. Example: Check that the in state S1, on receipt of a valid connect request, enters state S2 (sends a valid connect confirm). Dept. Electrical & Information Engineering, Computer Engineering Lab 7-64 Dr. Junzhao Sun

17 Testing States Test Case Scheme Testing state is a combination of the states of SUT (), test system, and underlying provider which can occur during testing Stable state t can be maintained i between test t case without t tester t activities iti Idle state is that no connection is established, independent of previous tests Test state is the one that test case is truly performed to check test purpose Initial state the start point of the execution of a test case End state is the stop state of a test One test case may stop at several end states A test case is conceptually composed of 3 parts Test body, the combination of test events for achieving test purpose Starts t in a initial iti testing ti state t Test preamble, takes the test from stable states to initial state Test postamble, takes the test from end states to one stable state If end state is not unique, it has to be checked by a verification step Postamble Stable State Preamble Test Test Body End State Verification State (Test Body) End State (Verification) Dept. Electrical & Information Engineering, Computer Engineering Lab 7-65 Dr. Junzhao Sun The outcome for each test case should be assigned a verdict Calculated from verdicts assigned to test events in the outcome Dept. Electrical & Information Engineering, Computer Engineering Lab 7-66 Dr. Junzhao Sun Outline Test Realization General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process Test realization is the process of producing a means of testing (MOT) for conformance to protocol standards, by reference to an ATS standard. MOT is a real test system used to execute a test suite on s Combination of equipment and procedures that derive, select, parameterize and execute the test cases of a given ATS, together with produce conformance log Does not mean the actual execution of test cases ETS derivation is to convert the ATS into final ETS Selection of applicable test cases (abstract or executable) Performed mainly on the basis of ICS (Imple. Conformance Statement) Parameterization of selected test cases (abstract or executable) Performed mainly on the basis of IXIT (Imple. extra Info for Testing) Derivation process, see next Dept. Electrical & Information Engineering, Computer Engineering Lab 7-67 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-68 Dr. Junzhao Sun

18 Derivation Process MOT Functions One-to-one correspondence between selected abstract & executable test cases Correspondence in all respects e.g. behavior, test purpose, verdicts, relation to ICS/IXIT entries, selectability, independency, etc. May include some MOT specific selection Independent from and compatible with the selection & parameterization processes ATS Derivation ETS Selection SATS Parameterization PATS Derivation & execution Derivation Derivation Selection SETS Parameterization PETS Execution Realization of configured ATM (abstract test method) Realize the LTs (lower testers) and LTCF (LT control function) LT s encoder/decoder d realization: simple but not efficient i LT s reference protocol implementation: efficient but difficult Specify or realize UTs (upper testers) UT can be dedicated software, human operator, existing protocol layers, etc. Specify or realize TCP (test coordination procedures) ATS to ETS Contain derived executable test cases Provide means for selection and parameterization Provide means for executing PETS Conformance log generation Must be human readable and permit verification of test verdicts assigned Dept. Electrical & Information Engineering, Computer Engineering Lab 7-69 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-70 Dr. Junzhao Sun Outline General concepts Testing documents Abstract test methods Abstract test suites Test realization Conformance assessment process Dept. Electrical & Information Engineering, Computer Engineering Lab 7-71 Dr. Junzhao Sun Conformance Assessment Process (1/2) Conformance assessment process is the entire activities that are necessary to validate and document the protocol conformance Test campaign is the actual execution of the test t suite 3 phases for conformance assessment process 1. Preparation for testing Production of SCS, ICS & IXIT Choice of ATM & ATS Preparation of SUT & MOT 2. Test operations Static conformance review of ICS Test realization (selection & parameterization) Test campaigns 3. Test report production Provides clients with documentation Dept. Electrical & Information Engineering, Computer Engineering Lab 7-72 Dr. Junzhao Sun

19 Conformance Assessment Process (2/2) Test Laboratories and Test Clients OSI specifications, proformas, test specifications i SCS, IXIT, ICS, PTS, ATS static conformance review results, test results, test log PCTR SCTR Preparation for testing administrative activities completion of fdocuments choice of test method and ATS choice of test specifications SUT and MOT preparation Test Operation static conformance review test selection & parameterization test tcampaigns Test Report Production Test Laboratory is responsible for conducting the conformance assessment of s and can be classified according to their role First-party t is operated by the supplier of, I.e. implementer, vendor Second-party is affiliated to the user of who is willing to test it Third-party is independent from either of the two whose business is the testing of products Clients will frequently be implementers or vendors of protocol implementation who want to have their own products tested and certified, but can also be important user of Dept. Electrical & Information Engineering, Computer Engineering Lab 7-73 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-74 Dr. Junzhao Sun Preparations for Testing (1/2) Preparations for Testing (2/2) Production of SCS, ICS By client filling proformas Production of IXIT By lab & client filling proformas Choice of ATM & ATS Agreement between client & lab Preparation of SUT By client Preparation of MOT By lab ATS ICS SUT decs ICS pro. IXIT Extra decs IXIT pro. SUT config. MOT config. Dept. Electrical & Information Engineering, Computer Engineering Lab 7-75 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-76 Dr. Junzhao Sun

20 Testing Operations (1/3) Testing Operations (2/3) Static conformance review Checks ICS for consistency with static conformance requirements Test selection Select a suitable subset of test cases from ATS resulting SATS Test parameterization Parameters are assigned values as specified in ICS & IXITs resulting PATS Transform PATS into PETS (parameterized executable test suite) Verification of TCPs To verity MOT & SUT are able to use TCP (Test Coordination Procedure) Test campaign, see next slide Execution of PETS and production of conformance log Test campaign Assignment of verdicts = [pass, fail, inconclusive] Test lab has to assign exactly one test t verdict per executed test t case Abnormal test termination or error leads to not run in report Inconclusive verdict have to be repeated at least once Conformance log Outcome of all executed test cases to document for client Negotiated exit Time when client and test lab jointly decide to stop the test campaign No formal test report will be produced Dept. Electrical & Information Engineering, Computer Engineering Lab 7-77 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-78 Dr. Junzhao Sun Testing Operations (3/3) Test Report Production (1/2) ETS ICS Derivation IXIT Selection SETS SUT config. MOT config. ETS ICS Derivation IXIT Selection SETS SUT config. MOT config. SUT decs SUT decs ATS ICS pro. Extra decs Parameterization IXIT pro. PETS Execution ATS ICS pro. Extra decs Parameterization IXIT pro. PETS Execution Comp. Report Dept. Electrical & Information Engineering, Computer Engineering Lab 7-79 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-80 Dr. Junzhao Sun

21 Test Report Production (2/2) SCTR Proforma SCRT = System Conformance Test Report Gives an overall summary of the conformance status to the SUT Contains a summary of achieved test t results & administrative i ti information PCTR = Protocol Conformance Test Report One for each protocol under test Administrative and technical information A summary of all achieved test results Individual detailed presentation of each test case 1. Identification summary Administrative information for identification of SCTR Information about test t laboratory, test t client, SUT, profile, etc. 2. System report summary One section for each tested protocol is required Document reference definition, protocol specification, ICSs, IXITs, PCTR, ATS, ATM, etc. Conformance status Whether static and/or dynamic conformance error Test cases run Number of passed, failed, and inconclusive test cases Dept. Electrical & Information Engineering, Computer Engineering Lab 7-81 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-82 Dr. Junzhao Sun PCTR Proforma Test Conducting 1. Identification summary Information about PCTR,, and testing environment 2. Protocol conformance summary conformance status This has/has not been shown by conformance assessment to be nonconforming to the referenced base specification(s) Static conformance summary The ICSs for this is/is not consistent with the static conformance requirements in the referenced base specification Dynamic conformance summary The test campaign did/did not reveal errors in the 3. Protocol conformance report Static conformance review report Test campaign report, for each test case in ATS ATS reference, selected or not, run or not, verdict, and observations Dept. Electrical & Information Engineering, Computer Engineering Lab 7-83 Dr. Junzhao Sun Formal tests Done by test laboratories that will issue official test certificate Informal tests In-house or private lab product test Tests at trade shows, bake-off meets Remote test access to tester through Internet (e.g. on Web server) Quality of Tests Consistency and repeatability Standard ATSs & approved ETSs Standard of test laboratory operations Test Report for procurement and regulatory bodies National and International recognition of test results Dept. Electrical & Information Engineering, Computer Engineering Lab 7-84 Dr. Junzhao Sun

22 Detailed Testing Process Whole Information Flow ETS Selection Derivation ICS IXIT SETS SUT config. MOT config. SUT decs Extra decs Parameterization ATS ICS pro. IXIT pro. PETS Execution Comp. Report Dept. Electrical & Information Engineering, Computer Engineering Lab 7-85 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-86 Dr. Junzhao Sun Summary References of This Chapter Conformance testing Black-box approach of testing protocol implementations against protocol specification Three phases for conformance assessment process 1. Preparation for testing Test generation: from ICS+IXIT & TPs, get ATS Document proforma Configuration of ATM architecture 4 methods 2. Test operations Test realization: from ATS, through selection & parameterization, get PETS Test execution 3. Test report production SCTR & PCTRs Bernd Baumgarten and Alfred Giessler, OSI Conformance Testing Methodology and TTCN, Elsevier Science, 1994 Dept. Electrical & Information Engineering, Computer Engineering Lab 7-87 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 7-88 Dr. Junzhao Sun