Test System Validation Guideline

Transcription

1 Test System Validation Guideline This document is the guideline for the engineering process of demonstrating that a particular Test System meets the specified requirements according to the Bluetooth Specification. WARNING: This document is under development and is distributed for information purposes only. Any material contained herein should not be considered final and is subject to change. Document No. 1.Q.198/0.8 June 9th 2000

2 Test System Validation Guideline page 2 of 19 Special Interest Group (SIG) The following companies are represented in the Bluetooth Special Interest Group: 3Com ERICSSON Mobile Communications AB IBM Corp. Intel Corp. Lucent Technologies Microsoft Motorola NOKIA Mobile Phones Toshiba Corp. Contributors Carsten Kuhfuss Magnus Sommansson Georges Seuron Lawrence Jones Waldemar Hontscha Wayne Park John Shi Brad Tipler Brian Redding 7 layers ERICSSON IBM ComBit NOKIA 3Com Toshiba Intel Motorola Revision History Revision Date Comments 0.8 May 12, 2000 First Member release Disclaimer and copyright notice THIS ADDENDUM DOCUMENT IS PROVIDED "AS IS" WITH NO WARRANTIES WHATSO- EVER, INCLUDING ANY WARRANTY OF MERCHANTABILITY, NONINFRINGEMENT, FIT- NESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION OR SAMPLE. All liability, including liability for infringement of any proprietary rights, relating to use of information in this document is disclaimed. No license, express or implied, by estoppel or otherwise, to any intellectual property rights are granted herein. This document is an intermediate draft for comment only and is subject to change without notice. Readers should not design products based on this document. Copyright Telefonaktiebolaget LM Ericsson *Third-party brands and names are the property of their respective owners. 2 8 June 2000

3 Test System Validation Guideline page 3 of 19 CONTENTS 1 Scope and Use Definitions References Process introduction Test System Validation Specification Test System Validation Report Criteria for Validation Report Acceptance Parts of the Test System Validation Required Documentation and Equipment Test System Conformance Statement Test Platform Documentation Test Case Documentation Equipment for Validation RF Test System Validation RF Test Platform Validation Goals of the RF Test Platform Validation Methods for the RF Test Platform Validation Mapping of Methods to Goals for RF Test Platform Validation RF Test Case Implementation Validation Goals of the RF Test Case Implementation Validation Methods of RF Test Case Implementation Validation Mapping of Methods to Goals for RF Test Case Implementation Validation Protocol Test System Validation Protocol Test Platform Validation Protocol Test Case Implementation Validation June

4 Test System Validation Guideline page 4 of 19 1 SCOPE AND USE This annex describes requirements and necessary validation activities for Bluetooth conformance test systems on a test system implementation independent level. It is the Bluetooth specific application of the generic references [4], [5], [6], [7], [8]. This guideline assumes that the Test Specifications [3] have been verified against the Bluetooth Specification [1]. However, it is essential that both the Bluetooth Specification [1] and Test Specifications [3] must be considered. Additional to the Test Specification which defines the requirements to be tested the requirements of the core specification must be implicit fulfilled by the test system, e.g. TCI/ HCI interface, test mode usage or physical parameters like hopping sequence, timing, modulation or packet types. The Bluetooth testing concept for Bluetooth Qualification [2] is open for multiple implementations from different test equipment manufacturers. Test system validation is therefore an essential process, which has to be done for all conformance test systems to make test results comparable and reliable. Validation is an engineering process to demonstrate that test results from conformance test systems are evident to show compliance with the specifications. It is furthermore a prerequisite for activation of test cases as category A. After successfully performed validation and recognition of the test system by the BQRB, delivered test results will be accepted for qualification of category A test cases. 4 8 June 2000 Scope and Use

5 Test System Validation Guideline page 5 of DEFINITIONS Bluetooth implementation sample Test system Test Case Implementations Test system validation guideline The Bluetooth implementation samples are Bluetooth devices that provide all the capabilities necessary to cover the scope of test cases of the test system, being the sample IUT for the purpose of validation. If required, additional Bluetooth implementation samples have to be used. Each Bluetooth sample shall be well documented regarding its conformance and revision status. The test system is the complete hardware and software needed to perform compliance test cases. It consists of a Test Platform and Test Case Implementations. The software using the capabilities of the Test Platform to perform test cases as described in the test specifications. Specifies the requirements for validation on a test system independent level. 1.2 REFERENCES [1] The Bluetooth Specification, as authored and published by the Bluetooth Promoters, and in the version applicable from time to time, the first version to be the specification dated July 26, 1999, document number 1.C.47/1.0 B, Volume 1 (Core) and Volume 2 (Profiles). [2] Qualification Program Reference Document, as authored and published by the Bluetooth Promoters, and in the version applicable from time to time, the first version to be the document dated 16 th of December, 2000, version 0.8. [3] Bluetooth Test Specifications. [4] ISO/ IEC 9646 Part 1-7 Conformance Testing Methodology and Framework. [5] Validation of Test Methods, EAL-P11, February [6] Interpretation of Accreditation Requirements in ISO/IEC Guide 25 and EN 45001, October [7] DIN ISO , August [8] EG V1.1.1 ( ) Methods for Testing and Specification (MTS); Planning for validation and testing in the standards-making process. Scope and Use 8 June

6 Test System Validation Guideline page 6 of 19 2 PROCESS INTRODUCTION Validation is an essential process, which has to be done for all conformance test systems to achieve: Comparable results. Reliable and traceable results. Test system independent results. Conformance to the Bluetooth core and test specifications. The Test System Validation Applicant shall send the validation specification and the validation report for the particular test system to the Bluetooth test and interoperability working group (BTI) for review. The BTI checks if all requirements for Bluetooth validation are fulfilled and send a notification to the BQRB. The BQRB has then the authority to announce new validated test system. Figure 2.1 gives a short overview of the relation of the different documents and activities. It is suggested that the validation specification is supplied a head of time. 35' Policies for validation 9DOLGDWLRQÃ*XLGHOLQH Test system independent definition of the technical requirements 9DOLGDWLRQÃ6SHFLILFDWLRQ Test system specific realisation of the validation 9DOLGDWLRQ 5HSRUW Validation Results %7, &KHFNLQJÃ9DOLGDWLRQÃ'RFXPHQWDWLRQ Review of the validation report %45% Accredit test system version Figure 2.1: Overview of the test system validation activities. 2.1 TEST SYSTEM VALIDATION SPECIFICATION The test system validation specification is the implementation specific description of all validation activities. It shall cover all aspects of this validation guideline and their adoption to the test system. It shall define the validation activities in detail. 6 8 June 2000 Process introduction

7 Test System Validation Guideline page 7 of TEST SYSTEM VALIDATION REPORT The test system validation report summarise the results after the validation is finalised, it contains: Date of validation. Identification of the test system (HW, SW). Reference Identification to the Bluetooth implementation samples. Who performed the validation. Validation specification used. Measurements instruments used. Detailed results of the validation activities. 2.3 CRITERIA FOR VALIDATION REPORT ACCEPTANCE The criteria that must be fulfilled for validation of a conformance test system are expressed differently for radio frequency, protocol and profile characteristics. The specific criteria shall be stated in the validation specification. As a general rule, best practice engineering shall be applied when defining these criteria. 2.4 PARTS OF THE TEST SYSTEM VALIDATION In general this annex subdivides the test system validation in RF Test System Validation and Protocol Test System validation. Each of them is further split in a Test Platform validation and a Test Case Implementation validation. The Test Platform is the hardware and software providing the capabilities to perform a scope of test cases in accordance to the Bluetooth core specifications. The Test Platform consists of a specific hardware and software, which will be used for implementing test cases. The Test Case Implementation is the software, which use the capabilities of the Test Platform to perform test cases as described in the test specifications. See Fig. 2.2 on page 8 for the structure of this guideline. Process introduction 8 June

8 Test System Validation Guideline page 8 of 19 7HVWÃ6\VWHPÃ9DOLGDWLRQ 5) Test System validation 3URWRFROÃ3URILOHÃ Test System validation Test Platform validation Test Case Implementation validation Test Platform validation Test Case Implementation validation Figure 2.2: Parts of the test system validation. The separation between Test Platform and Test Case Implementation is done based on the fact that the Test Platform is a fixed framecommon for all test cases, these capabilities need therefore not to be validated in association with each test case. The test cases need however to be validated individually since they are unique and tied to a small part of the behaviour of the Test Platform. 2.5 REQUIRED DOCUMENTATION AND EQUIPMENT This type of documentation is required as a basis for validation: Test system conformance statement. Test Platform documentation. Test case documentation. Equipment for validation Test System Conformance Statement The test system conformance statement is the official document delivered by the test equipment manufacturer stating the implemented capabilities of the test system. This statement contains: Scope of test cases, which are to be performed on the test system. This may be a test case list or a list of test specifications. Declaration of implemented Bluetooth capabilities, including: Used versions of Bluetooth test specifications. Limitations and deviations of the test system in respect to the Bluetooth test and core specifications. 8 8 June 2000 Process introduction

9 Test System Validation Guideline page 9 of Test Platform Documentation The Test Platform documentation is required to verify if the specified capabilities are according to the specifications. This typically contains: Technical documentation. Statement of used versions of the Bluetooth core and test specifications including the used errata. Hardware description (e.g. instruments and purposes). Software description and sources (e.g. path compensation routine, protocol stack). Configuration management description (procedure and status). User manuals Test Case Documentation This typically contains: Software description. Statement of used versions of the Bluetooth test specifications including the used errata. Configuration management description (status). Used measurement equipment and configuration. User manuals. Measurement uncertainty budgets Equipment for Validation This typically contains: Test Platform. Test case implementations. Analysing and monitoring tools. Bluetooth implementation samples. Process introduction 8 June

10 Test System Validation Guideline page 10 of 19 3 RF TEST SYSTEM VALIDATION 3.1 RF TEST PLATFORM VALIDATION The Test Platform must be in compliance with the necessary parts of the specifications. Validation shall demonstrate that the Test Platform is capable to perform the specified scope of test cases. Deviations from the Bluetooth core specifications are allowed if they do not reduce the designated scope of test cases and are properly documented. The Test Platform validation shall furthermore demonstrate that the Test Platform can achieve the required measurement accuracy Goals of the RF Test Platform Validation The RF Test Platform must provide Bluetooth radio and baseband physical characteristics in compliance with the Bluetooth core specification. The scope of the RF Test Platform validation is therefore divided into the following 4 goals: 1. Physical characteristics, which typically contains: Carrier frequencies Hopping schemes Modulation Packet-types Packet coding Timing 2. Signalling capabilities, which typically contains: Τest mode activation Connection establishment Packet type selection Data transmissons 3. Measurement accuracy, which typically contains: Power levels Timing of packets Frequency errors 10 8 June 2000 RF Test System Validation

11 Test System Validation Guideline page 11 of 19 The Test Platform provides capabilities for testing, which must meet the accuracy requirements in the Bluetooth test specifications. 4. Manipulation capabilties, typically contains: Dirty transmitter. Interfering signals. Corrupted data packets. These capabilities are necessary to verify that the reaction of the device during test on an invalid behaviour simulated by the test system. These manipulation features has to be implemented according to the test specifications Methods for the RF Test Platform Validation The validation process is based on a set of defined methods, which are used to verify a set of goals. The same set of methods are used in order to validate both the Test Platform and the Test Case Implementations, these are: 1. Documentation review. 2. Code review. 3. Verification with Bluetooth implementation samples; Execution of test cases on the test system with connected Bluetooth implementation samples. 4. Perform verification scenarios; Verification with additional measurement equipment and/ or manipulation of the test set-up configuration. The goals are however unique for the Test Platform and the Test Case Implementations respectively. RF Test System Validation 8 June

12 Test System Validation Guideline page 12 of Mapping of Methods to Goals for RF Test Platform Validation The following table shows methods, which have to be applied to achieve the RF Test Platform goals: Goals Methods Documentation Code Verification Perform review review with Bluetooth verification implementation scenarios samples 1. Physical characteristics X X X 2. Signalling capabilities X X X 3. Measurement accuracy X X X 4. Manipulation capabilities X X Table 3.1: Mapping of methods to goals for RF Test Platform validation Physical Characteristics (goal1) 1. Documentation review. The review is based on the described documents of chapter 2.5. Typically tasks shall be as followed: Verify that the referenced test specifications for the Test Platform and the test cases are consistent. These references shall also be consistent with the declared scope of test cases. The revision history shall be possible to follow. Required equipment shall be available or possible to obtain. Is the Bluetooth physical part (RF, BB physical) implemented in the correct version? Are all deviations and limitations described? 2. Code review Not applicable to this goal. 3. Verification with Bluetooth implementation samples Perform tests with Bluetooth implementation samples and check that the results comply with the Bluetooth specification June 2000 RF Test System Validation

13 Test System Validation Guideline page 13 of Perform verification scenario Perform verification scenario with Bluetooth implementation samples. Verify with additional measurement equipment the correct physical characteristics (e.g. frequency, timing, level measurements) Signalling Capabilities (goal2) 1. Documentation review The review is based on the described documents of chapter 2.5. Typical questions that shall be answered are: Is the Bluetooth protocol implemented in the correct version? Are all deviations and limitations described? Check that they do not affect the defined scope of test cases. 2. Code review Assure that the software structure is according to the Bluetooth specification. 3. Verification with Bluetooth implementation samples Execution of test cases with Bluetooth implementation samples. Verify with the results that the protocol is in conformance with the Bluetooth specification. 4. Perform verification scenario Not applicable to this goal Measurement Accuracy (goal3) 1. Documentation review The review is based on the described documents of chapter 2.5. Typical questions that shall be answered are: Declared uncertainty budgets shall be reasonable with respect to the parameters they are related to. Is the achievable accuracy described? Which internal calibration is needed for this? Are the internal calibration routines sufficiently described and feasible to achieve the accuracy? RF Test System Validation 8 June

14 Test System Validation Guideline page 14 of Code review Check that the implementation of the internal calibration routine is correct. 3. Verification with Bluetooth implementation samples Not applicable to this goal. 4. Perform verification scenario Perform a verification scenario to demonstrate the effectiveness of the path compensation Manipulation Capabilities (goal4) 1. Documentation review The review is based on the described documents of chapter 2.5. Typical questions that shall be answered are: Check the Test Platform documentation whether the declared scope of test cases can be performed on the described Test Platform. 2. Code review Not applicable to this goal. 3. Verification with Bluetooth implementation samples Not applicable to this goal. 4. Perform verification scenario Perform a verification scenario to demonstrate the manipulation capabilities (e.g. dirty transmitter, corrupted data packets) June 2000 RF Test System Validation

15 Test System Validation Guideline page 15 of RF TEST CASE IMPLEMENTATION VALIDATION The usage of the Test Platform capabilities in the Test Case Implementation must be in compliance with the Bluetooth test specifications. Validation shall demonstrate that the Test Case Implementation correctly uses the adequate set of Bluetooth capabilities when performing test cases as defined in the Bluetooth test specifications. The Test Case Implementation validation shall furthermore demonstrate that the measurement and manipulation capabilities are applied as defined in the Bluetooth test specifications Goals of the RF Test Case Implementation Validation The scope of the Test Case Implementation validation is divided into the following 5 goals: 1. Test Platform configuration. 2. Reference to the Bluetooth specifications. 3. Initial conditions. 4. Test procedure. 5. Verdict Methods of RF Test Case Implementation Validation The validation process is based on a set of defined methods, which are used to verify a set of goals. The same set of methods are used in order to validate both the Test Platform and the Test Case Implementations, these are: 1. Documentation review. 2. Code review. 3. Verification with Bluetooth implementation samples; Execution of test cases on the test system with connected Bluetooth implementation sample. 4. Perform verification scenarios; Verification with additional measurement equipment and/ or manipulation of the test set-up configuration. The goals are however unique for the Test Platform and the Test Case Implementations respectively. RF Test System Validation 8 June

16 Test System Validation Guideline page 16 of Mapping of Methods to Goals for RF Test Case Implementation Validation The following table shows methods, which have to be applied to achieve the RF Test Case Implementation goals: Goals Methods Documentation Code Verification Perform review review with Bluetooth implementation verification scenarios samples 1. Test platform configuration X X X 2. Reference to the Bluetooth specification X 3. Initial conditions X X X X 4. Test procedure X X X X 5. Verdict X X X X Table 3.2: Mapping of methods to goals for RF Test Case Implementation validation Test Platform Configuration (goal1) 1. Documentation review Is the Test Platform configuration described in the documents? Is this configuration applicable for the test case? Are the measurement accuracy requirements fulfilled? 2. Code review Are the Test Platform configuration settings in line with the documentation? Are the correct path compensation methods activated? 3. Verification with Bluetooth implementation samples Not applicable to this goal. 4. Perform verification scenario Generate verification scenarios to demonstrate the activation of internal path compensation procedures. Check the configuration of the Test Platform June 2000 RF Test System Validation

17 Test System Validation Guideline page 17 of Reference to Bluetooth specification (goal2) 1. Documentation review Are the current versions of the Bluetooth test specifications (including all errata) used? Further methods not applicable to this goal Initial Conditions (goal3) 1. Documentation review Is the described initialisation in line with the test specification? 2. Code review Check all initialisation steps: - Are all steps present? - Are all values implemented correctly? 3. Verification with Bluetooth implementation samples While performing the test case, does a Bluetooth implementation sample reach the initial conditions? Check trace files and implementation sample to verify. 4. Perform verification scenario Perform a verification scenario to check the behaviour if for a Bluetooth implementation sample the initial conditions cannot be set completely(e.g. manipulation of the signalling to the IUT) Test Procedure (goal 4) 1. Documentation review Is the described test procedure in line with the test specification? 2. Code review Check all test steps: - Are all steps present? - Are all values implemented correctly? 3. Verification with Bluetooth implementation samples While performing the test case, does a Bluetooth implementation sample perform the test procedure? Check traces to verify. RF Test System Validation 8 June

18 Test System Validation Guideline page 18 of Perform verification scenario Measure stimulus (e.g. Carrier level, frequency) and compare to correct values. Measure reaction of the IUT with external equipment to get independent results. (e.g. for level measurement insert additional attenuator, or change carrier frequency ). Perform the verification scenario, but change the test system to get different results Verdict (goal5) 1. Documentation review Are the definitions of the pass and fail verdicts in line with the correct version of the test specification? 2. Code review Check in the code the verdict assessment: - The definition of the pass verdict - The definition of the fail verdict - The definition of the inconclusive verdict - Check logic of assessment 3. Verification with Bluetooth implementation samples Verify the verdict assessment for the Bluetooth implementation sample. 4. Perform verification scenario Manipulate the test system to force a fail verdict June 2000 RF Test System Validation

19 Test System Validation Guideline page 19 of 19 4 PROTOCOL TEST SYSTEM VALIDATION TBD 4.1 PROTOCOL TEST PLATFORM VALIDATION TBD Protocol Test Case Implementation Validation TBD Protocol Test System Validation 8 June