os-testr Documentation

Transcription

1 os-testr Documentation Release Matthew Treinish July 17, 2015

2

3 CONTENTS 1 os-testr Features Installation 5 3 Usage ostestr subunit-trace subunit2html Contributing 13 5 Work Items for os-testr Short Term Long Term Indices and tables 17 i

4 ii

5 Contents: CONTENTS 1

6 2 CONTENTS

7 CHAPTER ONE OS-TESTR A testr wrapper to provide functionality for OpenStack projects Free software: Apache license Documentation: Source: Bugs: Features ostestr: a testr wrapper that uses subunit-trace for output and builds some helpful extra functionality around testr subunit-trace: an output filter for a subunit stream which provides useful information about the run subunit2html: generates a test results html page from a subunit stream Release Notes Adds support for comments in a blacklist file and printing comments for each exclude Several bugfixes for subunit-trace Switched subunit-trace behavior to disable printing percent change in run time by default and make it optional First release which includes: ostestr, subunit-trace, and subunit2html 3

8 4 Chapter 1. os-testr

9 CHAPTER TWO INSTALLATION At the command line: $ pip install os-testr Or, if you have virtualenvwrapper installed: $ mkvirtualenv os-testr $ pip install os-testr 5

10 6 Chapter 2. Installation

11 CHAPTER THREE USAGE This section contains the documentation for each of tools packaged in os-testr 3.1 ostestr The ostestr command provides a wrapper around the testr command included in the testrepository package. It s designed to build on the functionality included in testr and workaround several UI bugs in the short term. By default it also has output that is much more useful for OpenStack s test suites which are lengthy in both runtime and number of tests. Please note that the CLI semantics are still a WIP as the project is quite young, so default behavior might change in future version Summary ostestr [-b blacklist_file <blacklist_file>] [-r regex REGEX] [-p pretty] [ no-pretty] [-s subunit] [-l list] [-n no-discover <test_id>] [ slowest] [ no-slowest] [ pdb <test_id>] [ parallel] [ serial] [-c concurrency <workers>] [ until-failure] [ print-exclude] Options --blacklist_file BLACKLIST_FILE, -b BLACKLIST_FILE Path to a blacklist file, this file contains a separate regex exclude on each newline --regex REGEX, -r REGEX A normal testr selection regex. If a blacklist file is specified, the regex will be appended to the end of the generated regex from that file --pretty, -p --no-pretty --subunit, -s --list, -l Print pretty output from subunit-trace. This is mutually exclusive with subunit Disable the pretty output with subunit-trace output the raw subunit v2 from the test run this is mutuall exclusive with pretty List all the tests which will be run. --no-discover TEST_ID, -n TEST_ID Takes in a single test to bypasses test discover and just excute the test specified --slowest --no-slowest After the test run print the slowest tests After the test run don t print the slowest tests 7

12 --pdb TEST_ID --parallel --serial Run a single test that has pdb traces added Run tests in parallel (this is the default) Run tests serially --concurrency WORKERS, -c WORKERS The number of workers to use when running in parallel. By default this is the number of cpus --until-failure --print-exclude Run the tests in a loop until a failure is encountered. Running with subunit or prettyoutput enable will force the loop to run testsserially If an exclude file is used this option will prints the comment from the same line and all skipped tests before the test run Running Tests os-testr is primarily for running tests at it s basic level you just invoke ostestr to run a test suite for a project. (assuming it s setup to run tests using testr already) For example: $ ostestr This will run tests in parallel (with the number of workers matching the number of CPUs) and with subunit-trace output. If you need to run tests in serial you can use the serial option: $ ostestr --serial Or if you need to adjust the concurrency but still run in parallel you can use -c/ concurrency: $ ostestr --concurrency 2 If you only want to run an individual test module or more specific (a single class, or test) and parallel execution doesn t matter, you can use the -n/ no-discover to skip test discovery and just directly calls subunit.run on the tests under the covers. Bypassing discovery is desirable when running a small subset of tests in a larger test suite because the discovery time can often far exceed the total run time of the tests. For example: $ ostestr --no-discover test.test_thing.testthing.test_thing_method Additionally, if you need to run a single test module, class, or single test with pdb enabled you can use pdb to directly call testtools.run under the covers which works with pdb. For example: $ ostestr --pdb tests.test_thing.testthing.test_thing_method Test Selection ostestr is designed to build on top of the test selection in testr. testr only exposed a regex option to select tests. This equivalent is functionality is exposed via the regex option. For example: $ ostestr --regex 'magic\.regex' This will do a straight passthrough of the provided regex to testr. Additionally, ostestr allows you to specify a a blacklist file to define a set of regexes to exclude. You can specify a blacklist file with the blacklist-file/-b option, for example: $ ostestr --blacklist_file $path_to_file The format for the file is line separated regex, with # used to signify the start of a comment on a line. For example: 8 Chapter 3. Usage

13 # Blacklist File ^regex1 # Excludes these tests.*regex2 # exclude those tests Will generate a regex to pass to testr which will exclude both any tests matching ^regex1 and.*regex2. If a blacklist file is used in conjunction with the regex option the regex specified with regex will be appended to the generated output from the blacklist_file. Also it s worth noting that the regex test selection options can not be used in conjunction with the no-discover or pdb options described in the previous section. This is because the regex selection requires using testr under the covers to actually do the filtering, and those 2 options do not use testr. It s also worth noting that you can use the test list option to dry run any selection arguments you are using. You just need to use list/-l with your selection options to do this, for example: $ ostestr --regex 'regex3.*' --blacklist_file blacklist.txt --list This will list all the tests which will be run by ostestr using that combination of arguments. Please not that all of this selection functionality will be expanded on in the future and a default grammar for selecting multiple tests will be chosen in a future release. However as of right now all current arguments (which have guarantees on always remaining in place) are still required to perform any selection logic while this functionality is still under development Output Options By default ostestr will use subunit-trace as the output filter on the test run. It will also print the slowest tests from the run after the run is concluded. You can disable the printing the slowest tests with the no-slowest flag, for example: $ ostestr --no-slowest If you d like to disable the subunit-trace output you can do this using no-pretty: $ ostestr --no-pretty ostestr also provides the option to just output the raw subunit stream on STDOUT with subunit/-s. Note if you want to use this you also have to specify no-pretty as the subunit-trace output and the raw subunit output are mutually exclusive. For example, to get raw subunit output the arguments would be: $ ostestr --no-pretty --subunit An additional option on top of the blacklist file is print-exclude option. When this option is specified when using a blacklist file before the tests are run ostestr will print all the tests it will be excluding from the blacklist file. If a line in the blacklist file has a comment that will be printed before listing the tests which will be excluded by that line s regex. If no comment is present on a line the regex from that line will be used instead. For example, if you were using the example blacklist file from the previous section the output before the regular test run output would be: $ ostestr -b blacklist-file blacklist.txt --print-exclude Excludes these tests regex1_match regex1_exclude exclude those tests regex2_match regex2_exclude ostestr 9

14 3.1.6 Notes for running with tox If you use tox _ for running your tests and call ostestr as the test command.. _tox: it s recommended that you set a posargs following ostestr on the commands stanza. For example: [testenv] commands = ostestr {posargs} this will enable end users to pass args to configure the output, use the selection logic, or any other options directly from the tox cli. This will let tox take care of the venv management and the environment separation but enable direct access to all of the ostestr options to easily customize your test run. For example, assuming the above posargs usage you would be to do: $ tox -epy regex ^regex1 or to skip discovery: $ tox -epy n test.test_thing.testthing.test_thing_method 3.2 subunit-trace subunit-trace is an output filter for subunit streams. It is often used in conjunction with test runners that emit subunit to enable a consistent and useful realtime output from a test run Summary subunit-trace [ fails -f] [ failonly] [ perc-diff -d] [ no-summary] [ diff-threshold -t <threshold>] Options --no-failure-debug, -n Disable printing failure debug information in realtime --fails, -f --failonly --perc-diff, -d Print failure debug information after the stream is proccesed Don t print success items Print percent change in run time on each test --diff-threshold THRESHOLD, -t THRESHOLD Threshold to use for displaying percent change from the avg run time. If one is not specified the percent change will always be displayed. --no-summary Don t print the summary of the test run after completes Usage subunit-trace will take a subunit stream in via STDIN. This is the only input into the tool. It will then print on STDOUT the formatted test result output for the test run information contained in the stream. A subunit v2 stream must be passed into subunit-trace. If only a subunit v1 stream is available you must use the subunit-1to2 utility to convert it before passing the stream into subunit-trace. For example this can be done by chaining pipes: 10 Chapter 3. Usage

15 $ cat subunit_v1 subunit-1to2 subunit-trace Adjusting per test output subunit-trace provides several options to customize it s output. This allows users to customize the output from subunittrace to suit their needs. The output from subunit-trace basically comes in 2 parts, the per test output, and the summary at the end. By default subunit-trace will print failure messages during the per test output, meaning when a test fails it will also print the message and any traceback and other attachments at that time. However this can be disabled by using no-failure-debug, -n. For example: $ testr run --subunit subunit-trace --no-failure-debug Rhere is also the option to print all failures together at the end of the test run before the summary view. This is done using the fails/-f option. For example: $ testr run --subunit subunit-trace --fails Often the fails and no-failure-debug options are used in conjunction to only print failures at the end of a test run. This is useful for large test suites where an error message might be lost in the noise. To do this $ testr run --subunit subunit-trace --fails --no-failure-debug By default subunit-trace will print a line for each test after it completes with the test status. However, if you only want to see the run time output for failures and not any other test status you can use the failonly option. For example: $ testr run --subunit subunit-trace --failonly The last output option provided by subunit-trace is to diable the summary view of the test run which is normally displayed at the end of a run. You can do this using the no-summary option. For example: $ testr run --subunit subunit-trace --no-summary Show per test run time percent change subunit-trace provides an option to display the percent change in run time from the previous run. To do this subunittrace leverages the testr internals a bit. It uses the times.dbm database which, the file repository type in testrepository will create, to get the previous run time for a test. If testr hasn t ever been used before or for whatever reason subunittrace is unable to find the times.dbm file from testr no percentages will be displayed even if it s enabled. Additionally, if a test is run which does not have an entry in the times.dbm file will not have a percentage printed for it. To enable this feature you use perc-diff/-d, for example: $ testr run --subunit subunit-trace --perc-diff There is also the option to set a threshold value for this option. If used it acts as an absolute value and only percentage changes that exceed it will be printed. Use the diff-threshold/-t option to set a threshold, for example: $ testr run --subunit subunit-trace --perc-diff --threshold 45 This will only display percent differences when the change in run time is either >=45% faster or <=45% slower. 3.3 subunit2html subunit2html is a tool that takes in a subunit stream file and will output an html page 3.3. subunit2html 11

16 3.3.1 Summary subunit2html subunit_stream [output] Usage subunit2html takes in 1 mandatory argument. This is used to specify the location of the subunit stream file. For example: $ subunit2html subunit_stream By default subunit2html will store the generated html results file at results.html file in the current working directory. An optional second argument can be provided to set the output path of the html results file that is generated. If it is provided this will be the output path for saving the generated file, otherwise results.html in the current working directory will be used. For example: $ subunit2html subunit_stream test_results.html will write the generated html results file to test_results.html in the current working directory 12 Chapter 3. Usage

17 CHAPTER FOUR CONTRIBUTING If you would like to contribute to the development of OpenStack, you must follow the steps in this page: Once those steps have been completed, changes to OpenStack should be submitted for review via the Gerrit tool, following the workflow documented at: Pull requests submitted through GitHub will be ignored. Bugs should be filed on Launchpad, not GitHub: 13

18 14 Chapter 4. Contributing

19 CHAPTER FIVE WORK ITEMS FOR OS-TESTR 5.1 Short Term Expose all subunit-trace options through ostestr Add html option to ostestr to run testr with subunit2html output Add unit tests * For ostestr test selection api * Response code validation on more argument permutations 5.2 Long Term Lock down test selection CLI * When this is done it will become release Add subunit-trace functional tests ** Sample subunit streams and test output from subunit-trace Add testing for subunit2html 15

20 16 Chapter 5. Work Items for os-testr

21 CHAPTER SIX INDICES AND TABLES genindex modindex search 17