VxWorks BSP Validation Test Suite User's Guide, 6.7. VxWorks BSP Validation Test Suite USER'S GUIDE 6.7

Transcription

1 VxWorks BSP Validation Test Suite User's Guide, 6.7 VxWorks BSP Validation Test Suite USER'S GUIDE 6.7

2 Copyright 2008 Wind River Systems, Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means without the prior written permission of Wind River Systems, Inc. Wind River, Tornado, and VxWorks are registered trademarks of Wind River Systems, Inc. The Wind River logo is a trademark of Wind River Systems, Inc. Any third-party trademarks referenced are the property of their respective owners. For further information regarding Wind River trademarks, please see: This product may include software licensed to Wind River by third parties. Relevant notices (if any) are provided in your product installation at the following location: installdir/product_name/3rd_party_licensor_notice.pdf. Wind River may refer to third-party documentation by listing publications or providing links to third-party Web sites for informational purposes. Wind River accepts no responsibility for the information provided in such third-party documentation. Corporate Headquarters Wind River Systems, Inc. 500 Wind River Way Alameda, CA U.S.A. toll free (U.S.): (800) 545-WIND telephone: (510) facsimile: (510) For additional contact information, please visit the Wind River URL: For information on how to contact Customer Support, please visit the following URL: VxWorks BSP Validation Test Suite User's Guide, Nov 08 Part #: DOC ND-00

3 Contents 1 Overview Introduction BSP VTS Overview Host and Target Requirements Installation Test Configuration Files and Scripts... 3 Test Suite Configuration... 3 Board Configuration... 4 Host Configuration... 5 Test Suite Build Script... 6 Test Suite Run Script... 7 Results Summary Script... 7 Log Files Additional Documentation... 7 Wind River Documentation Setting Up the Test Environment Introduction... 9 iii

4 VxWorks BSP Validation Test Suite User's Guide, Setting Your Host Environment Solaris and Linux Hosts Windows Hosts Configuring Run-Time Support in Workbench Working with the BSP VTS Introduction BSP Validation Test Workflow Workbench Command Line Running the BSP VTS Configuring the Target and Host Workbench Command Line Building the Tests Workbench Command Line Running the Tests Workbench Command Line Reviewing the Test Results Displaying the Test Results Workbench Command Line Analyzing Your Test Results A Troubleshooting the BSP VTS A.1 Introduction A.2 Harness Exceptions iv

5 Contents A.3 Test Case Aborts and Failures A.4 Re-running Failed Test Cases Re-running Selected Test Modules A.5 Debugging Test Cases B Glossary Index v

6 VxWorks BSP Validation Test Suite User's Guide, 6.7 vi

7 1 Overview 1.1 Introduction BSP VTS Overview Additional Documentation Introduction The VxWorks board support package validation test suite (BSP VTS) is provided to help you test and validate your VxWorks BSP. This document provides instructions for using the BSP validation test suite, including: an overview of the BSP VTS step-by-step instructions for building, configuring, and running the test suite and viewing results information on reading and interpreting the test suite results information on troubleshooting the test harness and test suite This document does not provide extensive information on the overall development or validation process for VxWorks BSPs. Complete information on developing and validating your BSP is available in the VxWorks BSP Developer s Guide. 1

8 VxWorks BSP Validation Test Suite User's Guide, 6.7 NOTE: If you are not familiar with the content of the VxWorks BSP Developer s Guide (including BSP validation), Wind River recommends that you first review that guide before attempting to test and validate your BSP using the BSP VTS. 1.2 BSP VTS Overview The VxWorks BSP VTS is a collection of host-initiated test scripts that exercise a target board running VxWorks. The purpose of this test suite is to provide an objective (although not exhaustive) report that you can use to judge the basic functionality of your VxWorks board support package (BSP). The BSP VTS uses a test framework that allows you to test a new BSP on your custom hardware. The test framework provides a set of command line scripts that allow you to: Configure a set of tests, hosts, and boards. Build test images for your target hardware. Execute the tests on your hardware. (On Windows hosts, loading and executing the test image must be performed manually from the command line.) Analyze the results of your test to assess the functionality of your BSP. These scripts can be executed from the command line using a VxWorks Development Shell, or from within Workbench. These tests allow you to get a quick understanding of the state of your BSP. The tests exercise the basic functionality of a BSP. For more information on the scripts and files that make up the BSP VTS, see Test Configuration Files and Scripts, p.3. For more information on specific test modules, see the VxWorks BSP VTS API Reference Host and Target Requirements In general, the BSP validation test suite has the same host and target system requirements as your Platform. (For specific information on Platform host and target requirements, see your release notes.) However, you should also note the following issues related to host and target requirements: 2

9 1 Overview 1.2 BSP VTS Overview All BSP VTS test scripts are supported on Linux and Solaris hosts. These scripts provide support for BSP VTS configuration, build, execution, and results reporting. On Windows hosts, all test scripts are supported with the exception of the test execution script (see Test Suite Run Script, p.7). The scripts available on Windows hosts support BSP VTS configuration, build, and results reporting. However, test execution must be started manually from the command line (see Running the Tests, p.19). You must have a full installation of VxWorks in order to run the BSP VTS. The target hardware you wish to test must boot with a VxWorks boot loader. Your target hardware must have a serial port connection and Ethernet connection to the target console port. You must have FTP or RSH access to the host from the target. In order to download the VxWorks image, the host must either authorize FTP read transfers from the user target with a password or use RSH for the transfers, no password required. The default username is target and the default FTP password is vxtarget. However, these values are configurable (see Host Configuration, p.5) Installation The BSP VTS is installed as part of the VxWorks operating system product installation; no additional installation steps are required Test Configuration Files and Scripts This section briefly describes the files and scripts that make up the BSP VTS test framework. For detailed usage information for each script, see the VxWorks BSP VTS API Reference. Test Suite Configuration Test suite configuration is performed using the suite.prm file and the suiteconfig test script. 3

10 VxWorks BSP Validation Test Suite User's Guide, 6.7 suite.prm suiteconfig The suite configuration file, suite.prm (where suite is the name of the test suite), is located in installdir/vxworks-6.x/vxtest/src/config/. The file is used by the suiteconfig script to define the set of test modules that are included in the test suite for a given test. suiteconfig scans a set of test module source files (.c files) which are specified by a suite.prm file, or on the command line using the -test option. The script then extracts XML tags embedded in those files and produces an image test configuration (ITC) file. The ITC file is an intermediate file used to pass the test configuration information found in the test modules to the suitebuild script for inclusion in the VxWorks test image. Board Configuration boardconfig board.prm Board configuration is performed using the boardconfig test script. The boardconfig script is used to generate a board configuration file (board.prm). The board configuration file specifies information about a target board (such as IP address, boot device, and so on) which is then used by the suiterun script in order to program the target s boot shell. This information is also used to issue commands to boot and load test images. The board.prm file (where board is the name of the target board) is generated by the boardconfig test script. When executing boardconfig from the command line, board.prm is located in $HOME/vxtestParamFiles/boards/ (where $HOME is derived from the host environment variables; $HOME on Solaris and Linux, $HOMEDRIVE\$HOMEPATH on Windows). When executing boardconfig from Workbench, the board configuration file is placed in your workspace directory under /vxtest/vxtestparamfiles/boards (for example, installdir/workspace/ vxtest/vxtestparamfiles/boards). This file is used to manage target parameters such as IP address, netmask, boot device, and so forth for a given target board. 4

11 1 Overview 1.2 BSP VTS Overview Host Configuration hostconfig host.prm Host configuration is performed using the hostconfig test script. The hostconfig script is used to create a host configuration file (host.prm). The host configuration file specifies information about a host (such as the name, IP address, FTP password, and so on). The suiterun test script uses this information to program the target's boot shell, and to issue the necessary commands to boot and load test images. You only need to run the hostconfig script if you wish to download your VxWorks image from a host other than the host used to execute the test scripts, or if you wish to override the default values for certain host parameters such as IP address, user name, or password. The host configuration file, host.prm (where host is the name of your host machine), is used to manage host configuration parameters such as host name, IP address, and the user name and password for the host. This file can be created in the following ways: If your VxWorks image will be downloaded from the same host you are running the test suite on, an appropriate host.prm file is generated automatically by the suiterun script (if one does not already exist). If you wish to override certain host parameters such as user name or password, you can use the hostconfig script to generate an appropriate host.prm file. You can manually update any existing host.prm file to suit your desired host. When executing hostconfig from the command line, the host.prm is placed in $HOME/vxtestParamFiles/hosts (where $HOME is derived from the host environment variables; $HOME on Solaris and Linux, $HOMEDRIVE\ $HOMEPATH on Windows). When executing hostconfig from Workbench, the host configuration file is placed in your workspace directory (for example, installdir/workspace/vxtest/vxtestparamfiles/hosts). 5

12 VxWorks BSP Validation Test Suite User's Guide, 6.7 Test Suite Build Script The suitebuild test script builds a set of VxWorks Image Projects (VIPs) in installdir/vxworks-6.x/vxtest/$user/proj in a directory with the following naming convention: bsp.tool.board.profile.vsb.mpmode.suite The script requires that you specify one or more of each of the following: BSP(s) toolchain(s) target board(s) You can optionally specify the following: a suite a VxWorks Source Build (VSB) directory a multiprocessing mode a set of build profiles When executing the test build script in Workbench, the projects are created in your project directory (for example, installdir/workspace/projectdir/vxtest/proj). suitebuild adds all of the run-time components required to run all of the test modules in the test suite. The test modules and test harness driver are linked directly to the VxWorks image so that no downloading is required during the test. If a target board CDF file corresponding to the given board (installdir/vxworks-6.x/ vxtest/src/config/boardcdf/board.cdf) exists, then it is used in the project build of the test image. If the ITC file (testroot/itc/suite.itc) for the named test suite is missing, it is automatically generated by invoking suiteconfig with the given -suite and -test options (if any). In this case, testroot is installdir/vxworks-6.x/vxtest/$user/ when using command-line mode, and installdir/workspace/projectdir/vxtest when using Workbench. If the test suite contains any test modules that are expected to test features that are not supported by the BSP (as reported by VxWorks), then those test modules are not included in the test image. These excluded test modules are reported in the build log and in the detailed (verbose) reports generated by suiteresults. 6

13 1 Overview 1.3 Additional Documentation Test Suite Run Script suiterun runs a given test suite on one or more boards in parallel. With the exception of configuring location-specific board parameters (boardconfig), and possibly the host parameters (hostconfig), this script can perform all of the steps necessary for a complete test run. This includes configuring the suite (suiteconfig), configuring the host (hostconfig), building the test module libraries and test image (suitebuild), running the test suite, and creating a report of the results (suiteresults). suiterun accepts all of the input parameters that are accepted by suitebuild and suiteconfig, and most of the parameters that are accepted by suiteresults. Results Summary Script The suiteresults script scans the test suite run log files and produces either a high level summary, or a more detailed account of the test results. You can also choose to create an HTML report in any directory, if desired. Log Files When executing from the command line, build and test logs are created in installdir/vxworks-6.x/vxtest/$user/logs/logfolder/, where the directory name logfolder can be set as an optional parameter. When running under Workbench, the logs are created in your project directory (for example, installdir/workspace/projectdir/vxtest/logs/default). 1.3 Additional Documentation This section lists additional documentation related to the technologies described in this book. 7

14 VxWorks BSP Validation Test Suite User's Guide, 6.7 Wind River Documentation The following Wind River documents present information associated with the VxWorks BSP VTS or information that can help you work with the BSP VTS: VxWorks BSP VTS API Reference This document contains specific API documentation for the BSP VTS. VxWorks BSP Developer s Guide This document includes a full discussion and guidelines for the VxWorks BSP development process. Wind River Workbench User s Guide This document describes the general Wind River Workbench user interface. Wind River Workbench By Example This document covers some common user scenarios related to working with VxWorks and Workbench. 8

15 2 Setting Up the Test Environment 2.1 Introduction Setting Your Host Environment Configuring Run-Time Support in Workbench Introduction This section discusses how to set up a proper test environment for the BSP VTS. This includes properly setting your host environment and how to configure the support for BSP VTS into your Workbench-based BSP VTS project build. When running the BSP VTS from the command-line, depending on your host and development set up, you can choose to run the test suite from the VxWorks Development Shell or from a shell on your development host. When running the BSP VTS from within Workbench, you must build a project that includes support for the BSP VTS. This information is provided in 2.3 Configuring Run-Time Support in Workbench, p.11. 9

16 VxWorks BSP Validation Test Suite User's Guide, Setting Your Host Environment To execute the tests from a shell on your development host, you must first set the environment properly using the wrenv utility provide by Wind River. (For more information on wrenv, see the VxWorks Command-Line Tools User s Guide.) NOTE: If running from a VxWorks Development Shell, or from within Workbench, wrenv is run for you automatically. Solaris and Linux Hosts To set up the proper environment on Solaris or Linux hosts, do the following: 1. If one is not already running, start an FTP server on your host. (Solaris hosts typically have an FTP server running by default.) Otherwise, you must be sure to allow RSH connections from your target board. 2. Make sure that the DISPLAY variable is set so that any spawned xterm sessions appear on the local X server. 3. On Linux hosts, be sure that the /usr/libexec/pt_chown script has the proper permissions. Execute the following command on your host machine (this must be executed as a superuser): # chmod 4755 /usr/libexec/pt_chown If this is done correctly, running /usr/libexec/pt_chown as a user does not return any errors. 4. Initialize the Wind River environment by running the wrenv.sh script (located in installdir). NOTE: If you are using Workbench or the VxWorks Development Shell, you do not need to execute wrenv separately. 5. Verify that each of the following exists on the host search path using the which command: sh xterm (may be under /usr/bin or /usr/x/bin) ifconfig (may be under /sbin or /usr/sbin) 10

17 2 Setting Up the Test Environment 2.3 Configuring Run-Time Support in Workbench expect (should return $WIND_FOUNDATION_PATH/$WIND_HOST_TYPE/ bin/, where $WIND_FOUNDATION_PATH and $WIND_HOST_TYPE are specific to your installation) telnet (may be under /bin or /usr/kerberos/bin) vxtest (which should return installdir/vxworks-6.x/vxtest/src/scripts/ vxtest) Windows Hosts To set up the proper environment on Windows hosts, do the following: 1. Start an FTP server on the host. You can launch an FTP server from Start > Programs > Wind River > VxWorks > FTP Server. Set up a user with the user name target and the password vxtarget (Security > Users/rights). Specify a home directory for the target user. Typically, this is set to c:\. 2. When using the command-line interface, all test scripts should be executed from the VxWorks Development Shell (Start > Programs > Wind River > VxWorks > VxWorks Development Shell). 2.3 Configuring Run-Time Support in Workbench To enable support for the Workbench-based BSP VTS in your project, you must include certain options when building your project. To build a suitable validation test suite image using Workbench, do the following: 1. Create a VxWorks Image Project. In Workbench, select File > New > VxWorks Image Project. This launches the New VxWorks Image Project wizard. 2. Provide a name for the project. 3. In the Project Setup dialog: a. Enter the BSP and toolchain options for your the target you wish to test. b. Select the Add Support to Project checkbox in the BSP Validation Test Suite field. 11

18 VxWorks BSP Validation Test Suite User's Guide, 6.7 c. At this time, you can click the Options... button to open the BSP Validation Test Suite Options dialog. You can use this dialog to set the test suite, board, host parameters, and results location configuration options for the project. Set the options (if desired) and then click Close. NOTE: If you do not set options in this dialog, the default options are used. Options can be changed at any time using the project or build properties dialog. 4. Set all other project configuration options as needed and click Finish. 5. Build your project. Right-click the project in Workbench Project Explorer and select Build Project. For more information on configuring VxWorks Image Projects, see Wind River Workbench By Example. 12

19 3 Working with the BSP VTS 3.1 Introduction BSP Validation Test Workflow Running the BSP VTS Reviewing the Test Results Introduction This chapter discusses how to configure, build, and run the BSP VTS tests using both Workbench and a command-line interface. This chapter also describes how to display your test results and provides some guidelines for analyzing the results. 3.2 BSP Validation Test Workflow This section describes the general workflow for running the BSP VTS. 13

20 VxWorks BSP Validation Test Suite User's Guide, 6.7 Workbench The workflow for running the BSP VTS within Workbench is as follows: 1. Create a new VIP and add BSP VTS support to the project. 2. (First time only) Configure the BSP VTS properties. During this step you can select the test suite you wish to run and the target board name. Optionally, you can set the host configuration options. 3. (First time only) Configure your target board by using the board configuration script to create a configuration file and then edit the resulting file to specify parameters that are unique to your board. 4. Build and run the test image. When the build and run operation completes, results are displayed automatically in HTML format (Linux and Solaris only; Windows must be run from the command line). Command Line It is not necessary to create a project when running from the command line. Therefore, the workflow is as follows: 1. (First time only) Configure your target board using the boardconfig script and edit the resulting board configuration file template (board.prm). 2. (First time only) Configure the host using the hostconfig script and edit the resulting host configuration file (host.prm). 3. Build and run the test image. When the build and run operation completes, results are displayed automatically in text format in the shell. 3.3 Running the BSP VTS This section describes how to configure the BSP VTS to run the test suite on a specific host and board for a given test run. The workflow described involves running the individual scripts that make up the BSP VTS. 14

21 3 Working with the BSP VTS 3.3 Running the BSP VTS Configuring the Target and Host The following sections describe how to configure the target board and test host using the boardconfig and hostconfig scripts. Instructions are provided for both Workbench and the command-line. Workbench Step 1: Step 2: Create a VxWorks Image Project If you have not already done so, you must create a VIP with appropriate support for the BSP VTS. This procedure is described in 2.3 Configuring Run-Time Support in Workbench, p.11. Configure BSP VTS Properties This step is only required when running the BSP VTS for the first time in a given workspace. If you change your workspace, you must complete this step again for the new workspace. To configure the BSP VTS properties, right-click on your project and select Properties > Build Properties > BSP Validation Test Suite Properties. In this dialog, you can select your test suite and edit the target board name. You can also set host configuration options from this dialog if desired. 15

22 VxWorks BSP Validation Test Suite User's Guide, 6.7 Step 3: Configure your Target Board To configure your target, right-click the project and select BSP Validation Test Suite > Configure Board. 16

23 3 Working with the BSP VTS 3.3 Running the BSP VTS The VxTest Board Configuration File is displayed. You must edit this file to provide your target IP address, the appropriate netmask and gateway, the target s boot device, and the serial port and baud rate. The default configuration file is bsp.prm. 17

24 VxWorks BSP Validation Test Suite User's Guide, 6.7 Command Line Step 1: Configure your Target Board This step is performed for the initial configuration only. It does not need to be performed again unless you change the board configuration. To configure your board, execute the following from the VxWorks Development Shell (or a command shell on the host with a properly set environment; see 2.2 Setting Your Host Environment, p.10): -> vxtest boardconfig -template board You must then edit the resulting board configuration file to suit your target hardware. Step 2: Configure the Host This step is performed for the initial configuration only. It does not need to be performed again unless you change the host configuration. 18

25 3 Working with the BSP VTS 3.3 Running the BSP VTS To configure your host environment, execute the following from the VxWorks Development Shell (or a command shell on the host with a properly set environment; see 2.2 Setting Your Host Environment, p.10): -> vxtest hostconfig You must then edit the resulting host configuration file to suit your test host Building the Tests The following sections describe how to build the VxWorks test image (vxworks) using the suitebuild script. Instructions are provided for both Workbench and the command-line. Workbench To build the configured test suite, right-click the project and select BSP Validation Test Suite > Build Tests. Once the build completes, a Suite Results file is displayed. This summary indicates a build status of PASS or FAIL. If a build failure occurs, examine the Build Console output for information regarding the failure. (For more information on troubleshooting the tests, see A. Troubleshooting the BSP VTS.) Command Line You can build and run the test suite from the command-line as a single command. For example: $ vxtest suitebuild -bsptoolboard bsp tool board When the tests finish executing, test results are displayed in text format in the shell window. For more information, see Running the Tests, p Running the Tests The following sections describe how to run the test suite using the suiterun script. Instructions are provided for both Workbench and the command-line. Note that for subsequent or similar test runs, much of the configuration and build steps described previously can be modified and executed using this script. 19

26 VxWorks BSP Validation Test Suite User's Guide, 6.7 Workbench To run the test suite, right-click the project and select BSP Validation Test Suite > Run Tests. On Linux and Solaris hosts, this loads and runs the test image automatically. On Windows hosts, follow the instructions shown in the Build Console for manually loading and running the test image, and copying the results to the proper directory. When you execute the tests on a Linux or Solaris host, a terminal window is launched outside of Workbench. This window displays the real-time output from the test suite that is running on the target. Upon completion of the tests, this window closes and the Test Result Summary displays complete build and test results. Command Line You can build and run the test suite from the command-line as a single command. Execute the following from the VxWorks Development Shell (or a command shell on the host with a properly set environment; see 2.2 Setting Your Host Environment, p.10): -> vxtest suiterun -bsptoolboard bspname toolchain boardname When test execution completes, test results are displayed automatically. For more information on displaying the test results, see Command Line, p Reviewing the Test Results This section describes how to display and analyze your test results Displaying the Test Results BSP VTS test results are automatically displayed at the end of the test run (see Running the Tests, p.19). However, if you wish to display results for a given project at a later time, you can use the instructions provided in the following sections. 20

27 3 Working with the BSP VTS 3.4 Reviewing the Test Results Workbench At the conclusion of a test run, detailed HTML-format test results are displayed in Workbench automatically. You can also choose to display test results for a given project at any time by right-clicking the project and selecting BSP Validation Test Suite > Show Results. This launches the Suite Results pane in Workbench. This pane provides comprehensive build and test results in HTML format. Clicking on any colored result provides you with detailed text from the log file for that test. Command Line At the conclusion of a test run, the suiterun script prints a text-based summary of the test results to your terminal. To better understand the meaning of the various statistics that are reported, refer to the API reference for suiteresults. If there are exceptions, aborts, or failures reported in this summary, you may wish to generate a detailed HTML report. This format provides easy access to failure points in the test logs. Invoke suiteresults as follows: $ vxtest suiteresults -v -htmldir browseraccessibledirectory where browseraccessibledirectory is a directory that is accessible by your Web browser. If it does not exist, suiteresults creates the specified directory. The script then populates the directory with the main test report (suiteresults.html), and all build and test logs that are found by suiteresults. Typically, with Apache-based Web servers, the public_html subdirectory under your home directory can be accessed with the following URL: In this case, you could enter the following command: $ vxtest suiteresults -v -htmldir ~/public_html Then, browse the test results at the following URL: suiteresults.html shows the summary test results, and provides hyperlinks to detailed results for each BSP that was tested. At the bottom of the table in the BSP column is the word TOTALS. This is a hyperlink to a detailed report showing combined results for all BSPs and each test case. 21

28 VxWorks BSP Validation Test Suite User's Guide, Analyzing Your Test Results The following figure shows an example of the generated HTML report. The suite results report provides a summary of the test results for your BSPs. The Log Dir field at the top of the report shows the directory that contains the log files for this test run. When running BSP VTS from the command line, this directory can be specified as an optional parameter. The Test Result Summary table includes the following information: BSP Name of the BSP that was used for the test run. This is specified using the bspname parameter provided with the -bsptoolboard flag during test configuration. Clicking the name of the BSP in the Suite Results log takes you to the detailed test report for that BSP. Tool Indicates the toolchain used to build the test. This is specified during using the toolchain parameter provided with the -bsptoolboard flag during test configuration. 22

29 3 Working with the BSP VTS 3.4 Reviewing the Test Results Board Name of the target hardware that the test was executed on. This is specified by the boardname parameter provided with the -bsptoolboard flag during test execution. Profile Indicates which VxWorks configuration profile (if any) was used to build the test image. VSB This contains the leaf directory name of the VxWorks Source Build (VSB) directory used to build the test image. (For more information on VSBs, see the VxWorks Command-Line Tools User's Guide). MP Indicates the processor configuration used for the test execution. The following values are possible: UP uniprocessor SMP symmetric multiprocessor AMP asymmetric multiprocessor (not currently supported in the BSP VTS) Suite Name of the test suite that was run. Build Indicates the pass or fail (PASS or FAIL) status for building the test suite. For information on troubleshooting test build failures, see A. Troubleshooting the BSP VTS. This field also provides a hyperlink to the build log for the test suite. Run Indicates the status of the test run. This can appear as PASS, FAIL, EXC (indicates test harness exceptions), ABORT (indicates test case aborts), or UNKN (results are inconclusive). A PASS status is indicated only when the test run encounters no exceptions, aborts, or failures, and the number of passed test cases plus skipped test cases is equal to the number of expected test cases for the suite. This field also provides a hyperlink to the test run log file. For more information on interpreting run results, see the reference entry for suiteresults. For more information on troubleshooting your BSP code, see the VxWorks BSP Developer s Guide. Exceptions This indicates the number of test harness exceptions. Note that test harness exceptions can be one of two types: host script exceptions or driver exceptions. 23

30 VxWorks BSP Validation Test Suite User's Guide, 6.7 For more information on test harness exceptions, see the reference entry for suiteresults. Passes Number of test cases that passed. Fails Number of test cases that failed. Aborts Number of test cases that were aborted. Skips Number of test cases skipped. Counted Tests Number of test case results counted by suiteresults when scanning the test logs. Expected Tests Number of test cases that are expected to run for a given test suite. Clicking the TOTALS link at the bottom of the Test Results Summary table gives a detailed report showing combined results for all BSPs. The Test Module Build Logs section provides links to the build logs for each test module. These logs can be useful for analyzing test build failures. 24

31 A Troubleshooting the BSP VTS A.1 Introduction 25 A.2 Harness Exceptions 25 A.3 Test Case Aborts and Failures 28 A.4 Re-running Failed Test Cases 30 A.5 Debugging Test Cases 31 A.1 Introduction This section describes some actions you can take if your test results indicate that there are one or more harness exceptions, test aborts, or test failures. For more information on troubleshooting your BSP, see the VxWorks BSP Developer s Guide. A.2 Harness Exceptions Harness exceptions are unexpected errors encountered by the harness itself. The number of harness exceptions encountered in the test run is displayed in the 25

32 VxWorks BSP Validation Test Suite User's Guide, 6.7 Exceptions column of the Test Result Summary report generated by suiteresults (for an example, see Analyzing Your Test Results, p.22). To see the exception messages, do the following: 1. Generate a detailed HTML report (as instructed in Displaying the Test Results, p.20). 2. From the summary table, click on the BSP hyperlink corresponding to a BSP test run that reported an Exception count. This opens the BSP Detailed Test Report for that BSP. The following shows an example BSP test report with exceptions: 26

33 A Troubleshooting the BSP VTS A.2 Harness Exceptions 3. The exception messages are displayed at the top of the BSP test report. Each exception message is followed by one or more hyperlinks leading to the exact point in the test log where the exception occurred. Exception messages come in two varieties host exception messages (HEXC) and driver exception messages (DEXC). Host exception (HEXC) messages are issued by the host scripts. The name of the script that issued the message is shown in the message itself. If the reason for the exception is not obvious from the message, you can look at the script source code to find out more. The script source code is located in installdir/vxworks-6.x/vxtest/src/scripts. Driver exception (DEXC) messages are issued by the vxtestv2 driver that executes on the target and conducts the execution of the test cases. If the meaning of the DEXC message is not clear, you may be able to find out 27

34 VxWorks BSP Validation Test Suite User's Guide, 6.7 more by finding the message in the driver source code under installdir/vxworks-6.x/vxtest/src/drv. A.3 Test Case Aborts and Failures Test case aborts are typically issued by test cases when some part of the test case setup and initialization fails. In this case, the test case cannot be completed. This is different than a failure, which indicates that the test case completed but failed. The number of test case aborts and failures are displayed in the Aborts and Fails columns of the Test Result Summary. To see more details about an abort or failure, do the following: 1. Generate a detailed HTML report (as instructed in Displaying the Test Results, p.20). 2. From the summary table click on the BSP hyperlink corresponding to the BSP test run that reported the abort or failure. This opens the BSP Detailed Test Report for that BSP. 28

35 A Troubleshooting the BSP VTS A.3 Test Case Aborts and Failures 3. Abort and Fail Results are displayed in the BSP Detailed Test Report, sorted by test module name and test case name. Each abort or fail entry contains one or more hyperlinks to the test log where the abort or fail result occurred. For SMP test runs, there is a separate result and hyperlink for each CPU on which the test was run. For UP, or for SMP runs where there is no CPU affinity set for the test case, the result is listed under the CPU: - column as shown below. 29

36 VxWorks BSP Validation Test Suite User's Guide, If you are still unsure why the abort or fail occurred after following the abort or fail hyperlink into the test log, you may need to consult the test module API reference and inspect the test case source code in order to gain further clarification. To consult the test module API reference, open the online Help Contents and search for the test module base name. The source code for a given module can be found in a subdirectory of installdir/vxworks-6.x/vxtest/src/tests. The file containing the source code is module.c. The test case given by the test report is actually a routine name within the test module. A.4 Re-running Failed Test Cases In the process of troubleshooting test failures, it may be convenient to run only the test cases that resulted in exceptions, failures, or aborts. This can be accomplished by invoking suiterun with the -nopass option. Issue the exact suiterun command you used to create the test report, and add the -nopass option. For example: $ vxtest suiterun -nopass -bsptoolboard bsp tool board... 30

37 A Troubleshooting the BSP VTS A.5 Debugging Test Cases This causes the test image to be re-built and run with only failed and aborted test cases. If a test case caused the target to reboot or hang during a prior test run, then that test case is also run. To re-run the full test suite again (including tests that passed), you must issue the -rebuild option to suiterun as follows: $ vxtest suiterun -rebuild -bsptoolboard bsp tool board... The -nopass option is not currently available from within Workbench. You must perform this from the VxWorks Development Shell. Re-running Selected Test Modules You may wish to create and run your own test suite consisting of a small subset of test modules that you are debugging. This can be done with the -suite and -test options. $ vxtest suiterun -suite mysuite -test module1.c module2.c -bsptoolboard bsp tool board... where mysuite is an arbitrary suite name, and module1.c and module2.c are test modules located in any subdirectory under installdir/vxworks-6.x/vxtest/src/tests. A.5 Debugging Test Cases To better understand the root cause of a test case failure, you may wish to use the Workbench source level debugger. To create a test image with source-level debugging, you first need to build the kernel and test module libraries with source-level debugging enabled. The best way to accomplish this, without modifying your existing operating system libraries, is to create a VxWorks Source Build (VSB) project. 1. From the VxWorks Development Shell, invoke vxprj to create a VSB in a directory of your choice and base it on your BSP: $ vxprj vsb create -bsp bspname vsbdir 2. When prompted whether to Set advanced options, specify y. 3. In response to all prompts having to do with ADDEDCFLAGS, specify -g. 31

38 VxWorks BSP Validation Test Suite User's Guide, Accept the defaults for all other prompts. 5. Set your current directory to vsbdir and type make to build the target libraries and test code. 6. Build a test image based on the VSB, including only the test module of interest: $ vxtest suitebuild -vsb vsbdir -suite mysuite -test module1.c -bsptoolboard bspname toolchain boardname The path to the test image is printed at the conclusion of the suitebuild command. 7. From Workbench, create a target server, connect to the target, and load and boot the test image. 8. Set one or more breakpoints within the test case function of interest. 9. To initiate the test run, open a target shell console, and enter the following in the target shell: -> vxtestv2 "-em -v 4" 10. To execute a single test case from the target shell, enter the following: -> vxtestv2 "-em -v 4 -tc testcase" where testcase is the name of the TestCase column of the BSP detailed test report. You should now be able to perform source-level debugging on the test case, and any kernel or BSP code that is relevant. If you find that you need to modify the test case, and would like suitebuild to rebuild the corresponding test module, invoke suitebuild with the -libbuild option as follows: $ vxtest suitebuild -libbuild -vsb vsbdir -suite mysuite -test module1.c -bsptoolboard bspname toolchain boardname 32

39 B Glossary ITC file See image test configuration file. image test configuration file test case test harness test module test suite An intermediate file used to pass the test configuration information found in the test modules to the test suite build script. The individual tests that make up a given test module. The BSP VTS test infrastructure. This includes the BSP VTS test scripts and the test harness driver. A collection of test cases that collectively exercise a certain piece of BSP code or functionality. A collection of test modules is referred to as a test suite. A collection of test modules (see test module) that are configured and run for a given BSP VTS test run. 33

40 VxWorks BSP Validation Test Suite User's Guide,

41 Index A aborts 24 ADDEDCFLAGS 31 adding support for BSP VTS in a VIP 11 additional documentation 7 allowing RSH connections 10 analyzing the test results 22 API references 3 B board see target hardware board.prm 4 boardconfig 4, 7, 15 booting test images 4 BSP VTS API reference 3 Overview 2 properties 18 running 14 workflow 13 BSPs developing 1 displayed in test results 22 -bsptoolboard 19 build displayed in test results 23 building the VxWorks test image 6, 19 VxWorks Image Projects 6 C CDF file 6 command line BSP VTS workflow 14 building the VxWorks test image 19 displaying the test results 21 running the test suite 20 command line options see options configuration boardconfig script 4 host 5, 18 suiteconfig script 4 target hardware 4 configuration files 3 board.prm 4 host.prm 5 suite.prm 4 target hardware 4 35

42 VxWorks BSP Validation Test Suite User's Guide, 6.7 configuring a test run with suiterun 7 BSP VTS properties 18 host and target hardware parameters 7 support for BSP VTS in Workbench 11 the test suite 3 your host 15 your target 15 counted tests 24 creating a VxWorks Image Project 11, 15 the host configuration file 5 the target hardware configuration file 4 the test suite configuration file 3 D debugging test cases 31 developing BSPs 1 DISPLAY variable 10 displaying test results 7, 20 documentation related 7 VxWorks BSP Developer s Guide 1 VxWorks BSP VTS API Reference 8 Wind River Workbench By Example 8 Wind River Workbench User s Guide 8 E exception messages 26 exceptions 23 expect 11 expected tests 24 F failures 24 files board.prm 4 host.prm 5 image test configuration (ITC) 4 log 7 suite.prm 4 suiteresults.html 21 FTP 3, 10 starting an FTP server on Windows 11 G generating an HTML report 20 the host configuration file 15, 18 the target hardware configuration file 4 H harness exceptions 25 host configuration 5, 15 requirements 2 host configuration file 18 host.prm 5 hostconfig 5, 7, 15, 18 HTML report analyzing 22 board 23 BSP 22 build 23 counted tests 24 exceptions 23 expected tests 24 failures 24 generating 20 passed tests 24 processor configuration 23 profile 23 skipped tests 24 test aborts 24 test suite 23 tests run 23 toolchain 22 VSB 23 36

43 Index -htmldir 21 -test 4 I ifconfig 10 image test configuration file see ITC file installation 3 ITC file 4, 6 P passed tests 24 prerequisites system requirements 2 profile 23 programming the target boot shell 4 L -libbuild 32 Linux hosts setting the host environment 10 loading test images 4 locating build and test logs 7 Log Dir field 22 log files 7 M managing host configuration parameters 5 test parameters 4 MP 23 N -nopass 30 O options -bsptoolboard 19 -libbuild 32 -nopass 30 -rebuild 31 R -rebuild 31 requirements host and target 2 re-running failed test cases 30 results see test results reviewing the test results 20 RSH 3, 10 running the test suite 7, 14, 19 from the command-line 9 in Workbench 9 S scripts boardconfig 4, 7, 15 hostconfig 5, 7, 15, 18 suitebuild 4, 6, 7, 19 suiteconfig 4, 7 suiteresults 6, 7, 21 suiterun 4, 7, 19 setting configuration options in Workbench 12 setting up your test environment 9 setting your host environment Linux and Solaris hosts 10 Windows hosts 11 wrenv utility 10 sh 10 skipped tests 24 37

44 VxWorks BSP Validation Test Suite User's Guide, 6.7 Solaris hosts setting the host environment 10 specifying host information 5 starting an FTP server 10 Windows hosts 11 suite.prm 4 suitebuild 4, 6, 7, 19 suiteconfig 4, 7 suiteresults 6, 7, 21 suiteresults.html 21 suiterun 4, 7, 19 configuring a test run with 7 T target hardware CDF 6 configuration 4, 15 information in test results 23 requirements 2 telnet 11 -test 4 test cases aborts and failures 28 debugging 31 re-running 30 test environment 9 test images booting and loading 4 test result summary table 22 test results analyzing 22 board 23 BSP 22 build 23 counted tests 24 display formats 7 displaying 7 displaying from the command line 21 displaying in HTML format 21 displaying in Workbench 21 exceptions 23 expected tests 24 failures 24 harness exceptions 25 passed tests 24 processor configuration 23 profile 23 skipped tests 24 test aborts 24 test case aborts 28 test case failures 28 test suite 23 tests run 23 toolchain 22 VSB 23 test scripts 3 test suite 23 configuring the 3 tests run 23 toolchain 22 troubleshooting 25 U unsupported features in the BSP 6 V -v 21 VIP see VxWorks Image Project VSB 23 vxtest 11 VxWorks BSP Developer s Guide 1, 8 VxWorks BSP VTS API Reference 8 VxWorks Development Shell executing test scripts from 11 vxworks image 19 VxWorks Image Project 15 adding support for BSP VTS 11 VxWorks Source Build project 23 VxWorks test image building 19 38

45 Index W which 10 Wind River Workbench By Example 8 Wind River Workbench User s Guide 8 Windows hosts setting the host environment 11 starting an FTP server 11 Workbench BSP VTS workflow 14 building the VxWorks test image 19 configuring support for BSP VTS 11 displaying the test results 21 running the test suite 20 setting test configuration options 12 workflow 13 command-line 14 using Workbench 14 wrenv utility 10 X xterm 10 39