Test Process Management (TPM) Software Tool Suite for Verification & Validation of Electronic Equipment. Product Overview

Transcription

1 Process Management (TPM) Software Tool Suite for Verification & Validation of Electronic Equipment Product Overview 1

2 Table of Contents Introduction 3 Life Cycle Overview 4 TPM Concept 5 TPM Modules 6 TPM Creation 7 TPM Editor 8 TPM Definition Elements 9 TPM SIgnal Browser 12 Execution 13 TPM Executor 14 Documentation 15 TPM Reporter 16 Scenario Example 16 Summary 18 2

3 Introduction Introduction TPM Overview TechSAT s Process Management (TPM) application is a comprehensive software tool suite for managing the entire life-cycle involved in the testing of electronic equipment. TPM supports you in all phases of the test process, from importing test requirements and system ICDs, creating test cases and test plans, executing tests, to generating test reports and performing impact analyses. As an Automatic Equipment (ATE) for testing, validating, and certifying electronic equipment, TPM can perform both interactive and fully automated tests. Therefore, TPM is ideally suited for production, integration, and maintenance testing. TPM Benefits TPM offers the following benefits: >> Reduced costs through the use of a single tool suite for the entire test life-cycle >> Reduced test implementation effort through the use of intuitive form-based test definitions >> Complete traceability from system requirement to test result >> Increased productivity through a high degree of automation during all stages of the test process >> Improved test quality through consistent use of the same test process TPM Process Management 3

4 Life-Cycle Overview The figure below shows the test life-cycle using DOORS together with TechSAT s TPM and Avionics Development System ADS2. DOORS is used for managing system requirements, test requirements, test results, and all interrelations between them, thereby providing traceability. By importing test requirements from DOORS into TPM, it is possible to define test cases, based on which test plans can be established. The test plans are then compiled and built into test executables. These can be executed on ADS2, which serves as the test execution engine. During test execution, the test results are fed back to TPM to generate a test result report. Subsequently, the test results can be exported to DOORS to create result and traceability matrices. DOORS Functional/ Functional/ System System Requirements Requirements Requirements Requirements Traceability Traceability Matrix Matrix Result Result Matrix Matrix TechSAT TPM Cases Cases & Plan Plan Executable Executable Result Result Report Report Results Results TechSAT ADS2 Execution Execution 4

5 TPM Life-Cycle Overview and Concept TPM Concept The test category covered by TPM is»black box testing«of any equipment that is connected to its environment through I/O connections. The idea is to stimulate the inputs of the unit under test (UUT) and verify the outputs against the expected values. TPM test programs do not access the I/O lines directly, but use the concept of signals as an interface. Signals are named data objects of different data types carrying a value and a timestamp. TPM manipulates and verifies only such signals. The transfer of signals to/ from physical I/O and thus to/from the UUT is performed by the underlying test execution engine based on ADS2. The following diagram shows the data flow between TPM and a unit under test: ADS2 Execution Engine TPM Program created with TPM Editor, then compiled to executable test program Signals I/O Hardware I/O Connections unit under test 5

6 TPM Modules The TPM software suite is comprised of the following modules: TPM Editor used to create and modify test cases based on the built-in test definition elements. These can be linked to test requirements and access signals as defined in the system ICD. cases can be combined to test plans. TPM Definition Elements - used to create test cases. TPM provides different types of test definition elements, e.g. for signal stimulation and verification, flow control, and event handling. TPM Compiler generates C code from the defined test plan which is then built to a test executable that can be executed on the test rig. The compiler is an integral part of the TPM Editor. TPM Signal Browser used to select signals and insert them in test definition elements. The browser allows importing system ICDs and provides powerful search and filter options. TPM Executor manages the test execution and offers monitoring, debugging, and tracing capabilities. TPM Reporter creates customizable test result reports based on the test case definitions and the test results. Additionally, the TPM Reporter can generate test description reports before the tests are executed. The following diagram shows the TPM modules and the data flow between them. Requirements ICD TPM Editor Plan TPM Compiler TPM Reporter Executable TPM Executor Description Report Result TPM Reporter Result Report 6

7 TPM Modules and Creation Creation Creation Requirements (e.g. DOORS) Case Concept Editor Plan Compiler Executable System ICD Signal Browser Job The flowchart above shows the test creation process with TPM. The TPM Editor can import test requirements from requirement engineering tools like DOORS through XML or CSV files. By importing a requirement document (e.g. a DOORS module), a test definition document is created, which has the same structure as the imported requirement document. For each requirement an empty test case is created, which is linked to the corresponding requirement. The test definition document can be navigated through a tree browser, which reflects the structure of the document. According to the associated requirement each test case has to perform certain operations on the system under test to determine if the requirement is fulfilled. These operations are carried out on process variables or signals, which are defined in the system Interface Control Document (ICD). With the TPM Signal Browser, the ICD can be imported in XML format and stored in a signal database, which can be accessed by the test cases. The signals can therefore be referenced directly in the test cases or inserted from the signal browser via drag and drop. When requirements are imported, TPM creates merely the test case structure. It is therefore up to the test engineer to implement each test case based on the test concepts. The test cases are implemented using the test definition forms in the TPM Editor. Once the test cases are implemented, it is possible to package subsets into a test plan. After invoking the editor s integrated compiler, it first converts the selected test plan into C-code. Subsequently a C-compiler compiles and builds the test plan into a test executable which will execute on the test engine, perform the specified system operations, and deliver the test results. Currently, the TPM Compiler can generate test executables for Windows, Linux, and VxWorks platforms. An additional output of the TPM Compiler is the test job, which is a complete description of all test plan components, including the name of the author, revision number, etc. This information is used to create test description reports as well as test result reports. 7

8 TPM Editor The TPM Editor GUI and its main components are depicted below Project View 2 Definition Element Property View 4 3 Online Help View 4 Console Output View The test project view (see above) displays the components of a test project: the»test case«list and the»test plan«list. All test cases are organized in the test case list, and all test plans are organized in the test plan list. By clicking the expand/collapse icons, you can expand or collapse the content of the test definition element, thereby changing the level of detail displayed in the view. When you select a particular definition element in the test project view, its properties are displayed in the test definition element property view for editing. The online help view provides detailed information on how the definition element works and how it can be configured. The console output view lists diagnostic information of the TPM Editor and compiler. 8

9 TPM Editor and Definition Elements TPM Definition Elements definition elements are used to implement test cases. They are represented in corresponding forms provided by the TPM Editor. The main test definition elements are depicted below. 9

10 Case The Case is a container for the test procedure and the corresponding test requirement. It is usually created automatically when you import test requirements into the TPM Editor. However, you can create additional test cases if desired. cases can be referenced in test plans to create specific test scenarios. Requirement A Requirement element is created for every imported test requirement. Requirements contain a general description of the requirement as well as a description of the required system conditions and the expected results. Procedure The Procedure is a container for the executable part of the test case. It typically contains an initialization component called a Prologue, one or more Steps, and an Epilogue. Step The Step implements a sequence of system operations executed in real time. Typically, it first stimulates the system, then waits for an event to occur, and finally verifies that the system reacts as expected, i.e. that it fulfills the system requirement associated with the Step. Prologue The Prologue is a non-realtime element used to initialize and prepare the system for the subsequent Steps. The Prologue is typically used to preset signals and load resources, such as simulations and panels. Epilogue The Epilogue is a non-realtime element which concludes the Procedure. It is typically used to reset the system to its initial state and to unload previously loaded resources. Time Slot The Step contains one or more Time Slot definitions. Time Slots are defined by absolute or relative time, by events, or by conditions. Within a Time Slot, multiple signal stimulation or verification operations are performed concurrently. Time Slots may overlap, thus allowing a very natural way of defining test conditions. Examples: >> 2 sec after PowerOn until PowerOff: verify voltage > 10V >> on WOW == False until WOW == True verify doors all closed Time Slots are event controlled, i.e. the execution of the contained statements depends on the occurrence of the event that triggers the time slot. Typical events are, for instance, the elapse of a time delay or a signal reaching a certain value. Per default, a Time Slot is triggered when the previous Time Slot is finished. To trigger the execution of Time Slots, you can either use system events or define your own events. 10

11 TPM Definition Elements Statements Statements are actions such as signal stimulation, signal verification, and dialog pop-ups. All Statements grouped in the same Time Slot are executed in parallel. By default, a Time Slot finishes when the last active statement in the Time Slot has finished. However, if you specify a timeout, the Time Slot is forced to finish upon the elapse of the timeout period, regardless whether there are still running statements. TPM supports the following types of statements: Stimulation Statements Stimulation statements are used to set signals to constant values or to values computed by a function generator. Examples: - set A to 5 - hold B at 7 - ramp A from 0 to 100 The function generator outputs are continously assigned to the signal as long as the statement is running. Verify Statements Verify Statements are used to compare actual signal values with their expected values for deciding whether a requirement is fulfilled or not. TPM offers several types of Verify Statements, including: >> VerifyLimits verifies whether the signal value is between the given minimum and maximum value. >> VerifyTolerance verifies whether the signal value is within the tolerance of the given nominal value. >> VerifyExact verifies whether the signal value exactly matches the nominal value. >> VerifyExpression verifies whether the given expression is true. >> VerifyNoChange verifies that the signal value remains unchanged during the time slot period. Trigger Statements Trigger statements are used to generate events on certain conditions. When an trigger expression evaluates to true, its associated event is raised possibly starting or terminating a time slot. BranchCondition Statement This statement is used to define a list of trigger statements. The BranchCondition Statement can raise one event only. Dialog Statements These statements are used to create message boxes and input dialogs. UserCode Statement The available TPM statements are sufficient to implement most test cases. However, for special complex behavior, the TPM statements may not be sufficient to define a test. In this case TPM provides the framework to integrate customized C-code through the UserCode Statement. Using this statement, TPM will automatically generate templates for the include file, the source file, and the make file. This makes it simple to add the necessary code and integrate it into the Case. Libraries Often test cases have certain repeating elements. For this purpose, TPM provides the Library Definition Element, which makes it possible to create libraries of Procedures, Steps, Prologues, and Epilogues. These can be referenced (called) in different test cases. 11

12 TPM Signal Browser The Signal Browser is a support utility for the Editor. It is used to select and insert signals into test definition elements. For this purpose, the TPM Signal Browser can import the ICD of the system under test. The Signal Browser GUI is shown below. In addition to its ICD import functionality, the Signal Browser can be used to search for required signals in the database. Found signals can be selected with the mouse and inserted into the test cases via drag and drop. You can search the database for particular signals by applying custom match patterns on signal names or on information which is specific to the signal type. For example, you can search for AFDX signals with a certain VL identifier and for MIL-STD-1553 signals with a certain RT address. The Signal Browser has four built-in filter types: 1) CVT 2) AFDX 3) A429 4) MIL-STD-1553 The CVT filter is a search filter based on signal names only. All other filters provide search criteria that are specific to a particular I/O type. 12

13 TPM Signal Browser and Execution Execution Creation Execution Requirements (e.g. DOORS) Concept Signal Browser Case HW & SW Resources Editor Plan Compiler Executor Executable Job Log Result The flowchart above illustrates the Execution phase, which follows immediately after the Creation. The Execution is managed by the TPM Executor, which uses, as described earlier, the test executable and the test job as inputs. To perform the execution, the Executor additionally needs hardware and software resources, such as the test rig with the test execution engine (ADS2), simulations, etc. The Executor does not only allow loading and executing different test executables, but also provides means to control the execution using step execution with user-defined step time intervals. The TPM Executor generates a test log file, which contains the value history of all signals referenced in the executable. The test log file is very useful for tracing and post-analyzing signal values as they were recorded during the test run. The test log file can be customized to contain only information on a selected number of signals. Besides in ASCII format, the test log file is also stored in the SQLite database format (sld) for better browse, filter, and search capabilities. Finally, the Executor generates a test result file in XML format. The test result file is based on the test job file and the output of the test executable. The following information is contained in the test result file: Exectuted test components (test components that were not executed are discarded) and their associated results (passed/ failed) Information on potential user interference during execution, for example, aborting, pausing to step through, etc. Absolute timestamp information regarding the start and end of individual test components as well as the entire test Miscellaneous information such as the name of the test person, the TPM Executor revision, etc. From this pool of information, the TPM Reporter builds customizable reports. 13

14 TPM Executor The TPM Executor GUI is divided into six areas as shown below Job View 2 Component Properties View 3 Execution Console View 1 4 Execution Control View 5 5 Execution Engine (ADS2) Log View 6 Signal View The Job View on the left-hand side displays the tree structure of the loaded test job. By clicking a test component in the tree structure, you can display the corresponding properties in the Component Properties View. You start the test execution by clicking the START button in the Execution Control View, and you terminate it by clicking the STOP button. When you stop the execution, the Executor will finish the current cycle and then execute the Epilogue component before terminating the test. Furthermore, you may pause the automatic execution and continue the test manually step by step in time intervals specified either in seconds or number of cycle times (or time frames) of the underlying execution engine. You can resume the normal execution at any time. During execution the execution details are automatically displayed in the Execution Console View. This view does not only list the test component currently executed, but also the absolute timestamp of its execution and the associated events. For better visualization, the Execution Console View highlights passed verify statements in green and failed verify statements in red. During test execution the tree structure in the Job View is automatically expanded, thereby indicating the execution progress. Analog to the Execution Console View, executed test procedures are highlighted either in green or red. Finally, the Execution Engine Log View provides information on the status of the ADS2 execution engine. You can set the level of information detail by selecting the message categories to be logged: errors, warnings, and information. 14

15 Execution and Documentation Documentation Creation Execution Requirements (e.g. DOORS) Case HW & SW Resources Log Concept Editor Plan Compiler Executor Executable Result System ICD Signal Browser Job Documentation Reporter Reporter TRG Description Report Result Report The last step of the test process involves the test documentation, i.e. the generation of a test report in PDF format, and the transfer of the test results back to the requirements tool, if such a tool is used. As depicted in the flowchart above, test reports can be generated at two points in the test process. First, you can create a test description report based on the test job file generated by the TPM Compiler. Usually, you create the test description report before you start the test. This way you can print an overview of the test you intend to execute. Second, the TPM Reporter allows you to generate a test result report based on the test result file. The test description report and the test result report are identical in structure. However, only the latter contains actual results. The test result report serves to evaluate the test run. Only if each individual verify statement evaluates to passed, the test as a whole will do so, too. If an error occurs, the report indicates its location and, if possible, also its cause. Besides generating the test result report, you can also use the test result file to transfer the results back to a requirements tool for conducting statistical analyses. With a configurable script included in TPM, you can extract result information from the test result file and store it to a separate file in comma-separated values (CSV) format. Then you can import the CSV file to a requirements tools such as DOORS. 15

16 TPM Reporter The TPM Reporter GUI is divided into five sections as depicted in the screenshot below Icon Bar Section 4 2 Report Selection Section 3 Result Overview Section 5 4 Report Configuration Section 5 Log and Control Section The Icon Bar Section contains two icons to choose between the two modes of report generation: test description report ( Description mode ) or test result report ( Result mode ). Additionally, it contains an icon to start the report generation. The Report Selection Section lets you select particular test jobs from the available test projects. In Description mode a test description report can be generated. In Result mode, however, you can select a particular test run of a test job and generate the corresponding test result report. When you select a test run, the Result Overview Section provides a result overview without the need to generate the report beforehand. In the Report Configuration Section you can tailor the report content and layout according to your wishes. On the one hand, you can select from numerous check boxes to decide what type of information you want to include in the report. On the other hand, you can define XML style sheets (XSL) to customize every aspect of the report page layout, including logos, headers, and footers. TPM comes with various default style sheets suitable for different purposes, such as»debug level«reporting,»management level«reporting, etc. During report generation, the generation progress is displayed in the Log and Control Section of the GUI. In this section, you can also stop the ongoing report generation. 16

17 TPM Reporter and Scenario Scenario Example The following is an example of how to formulate a test requirement with TPM. Power Switch Verification Requirements >> (1) Verify that the power_ok condition becomes true within one second after turning power on. >> (2) Verify that the voltage is 10 V +/- 0.1 V and the current is > the 2.5 A after power_ok for at least 10 seconds. >> (3) Verify that 0.4 sec after switching power off, the voltage is 0 V +/- 0.1 V and the current is < 0.1 A. Create Case Now we define the test case corresponding to the above test requirements: We divide the whole requirement into two test steps, one is power_on and the other is power_off. Step power_on: Start: >> trigger PowerOnFail on timeout==1.0sec >> set power = 1 >> Trigger PowerReady on power_ok==1 Ver: OnEvent PowerReady, for 10.0sec >> Verify, ALWAYS, Voltage == 10.0V, Tolerance=0.1V >> Verify, ALWAYS, Current > 2.5A OnEvent PowerOnFail: >> Popup Message FAILED Step power_off: Start: Delay = 1.0, >> Set power = 0 Ver: After 0.4sec for 10.0sec: >> Verify, ALWAYS, Voltage == 0.0, Tolerance = 0.1 >> Verify, ALWAYS, Current <

18 Within the Step power_on, there are three Time Slots: Start, Ver, and Fail. Within the Time Slot Start, the signal power is set to 1 (which means power is turned on). Furthermore, an event called PowerReady will be triggered when the condition power_ok == 1 becomes true. The two Statements Set and Trigger should be finished within one second; else, the event PowerOnFail will be triggered. Within the Time Slot Ver voltage and current values are verified against the expected values for 10 seconds. The Time Slot Fail is only used to display an error message in case of a timeout of Time Slot Start, that means that the power_ok signal is not raised for 1.0 second after setting the power to 1. The power_off test step is similar. The following screenshot shows the result of the test definition of this example in the TPM Editor. After defining the test case, you can click the Build icon in the menu bar to call the TPM Compiler to build the executable C file from the test case. When the TPM Compiler has successfully compiled the test case, you can use the TPM Executor to select the required test job. As soon as you click the Start button, the test job is executed automatically in the ADS2 environment. The test results will be saved automatically to the specified path and according to the layout defined before. 18

19 Summery Summary The following lists some of the highlights of TPM: All-in-one software tool suite covering the entire test lifecycle of electronic equipment High degree of automation in the entire test process Easy and intuitive test implementation based on input forms for standard test components Full traceability between test requirements, test case(s)/plan(s), and test results C-Code interface for complex test cases Support for libraries for code-reusability SQLite database with GUI for easy import and handling of the system ICD execution control with debugging capabilities Tailored test report generation Integrated version control (Clear Case, PVCS, and CVS) Support for Windows, Linux, and VxWorks platforms 19 19

20 techsat GmbH Technical Systems for Avionics and Gruber Strasse 46c Poing Germany Tel +49 (8121) Fax +49 (8121) techsat Hamburg GmbH Hein-Saß-Weg Hamburg Germany Tel +49 (40) Fax +49 (40) techsat North America Lake Washington Air Harbor Hangar Three 3849 N.E. 98th Street Seattle WA USA Tel +1 (206) This document describes TechSAT's Process Management (TPM) tool suite. The information in this document is subject to change without further notice. TechSAT assumes no responsibility for any error that may appear in this document. All product names are (registered) trademarks of the referenced companies. Copyright 2010 TechSAT GmbH / Rev All rights reserved.