Tempest: Integrated OpenStack Testing

Transcription

1 Tempest: Integrated OpenStack Testing Matthew Treinish May 12, 2014

2 OpenStack is Complicated

3 Testing before Tempest Every project had its own unit tests Some projects had functional tests There was nothing to test the whole picture

4 What is Tempest? The official OpenStack integration test suite Only black-box testing, only interacts with OpenStack REST APIs Over 2300 tests covering every incubated and integrated project Used as the gating tests for every integrated project

5 Tempest Growth Started in late Diablo to solve the lack of integrated testing Became a gating test suite in the Folsom cycle Was one of the 5 most active projects in Icehouse Testing with tempest is now a project graduation requirement Being used in refstack as part of defcore validation

6 Growing Pains Huge continued increase in the number of projects and tests The common code to handle all the APIs becomes more complex Managing individual tests becomes very difficult Total run time continues to grow Tempest configuration becomes more involved to handle all the services and configurations

7 Design Principles Should be able to run against any OpenStack Cloud from devstack to a public cloud Everything explicit Only use public interfaces (REST APIs) Should be self cleaning Self Testing

8 Basic Tempest Architecture

9 API Tests Directly test the OpenStack REST APIs Uses a unified tempest REST client Fall into 2 categories positive and negative Negative tests are starting to be automatically generated from api schemas

10 Scenario Tests Through path tests of functionality Designed to simulate a typical end user action workflow Use official python-*clients Ideally will involve integration points between projects

11 CLI Tests Test the official client CLIs Quick sanity checks of functionality in an OpenStack environment Only perform read-only operations Mostly a matter of convenience for gating

12 Stress Tests Use tempest to generate load on a cloud Dedicated scenarios for generating load Can specify any existing tempest test to also be a stress generator All scenario tests are automatically added as stress tests Run stress tests serially or in parallel looping for a set period of time

13 3rd Party and Unit Tests 3rd Party Tests Tests for in-tree non-openstack APIs Currently only tests Nova s EC2 implementation Unit Tests Tempest eventually reached a size where its operations were too complex to take for granted Started late in Havana because of bugs in wrapper scripts super passing the gate Unit testing coverage was expanded significantly during Icehouse

14 Parallel Tempest Tempest is capable of running it s tests in parallel More closely simulates real load on a cloud Greatly improves speed of running tests: Serially gate runs took 1 hr with <1000 tests Current gate runs are 40min to 1 hr with 2300 tests

15 Issue with single user in parallel

16 Tenant Isolation and Parallel Execution Tenant isolation enables running each test class in isolation When enabled each test class creates an isolated user and tenant Execute tests in parallel at the class level

17 Branchless Tempest Starting with Icehouse tempest will no longer have stable branches For gating purposes we will use master tempest against stable branch proposed commits All proposed tempest commits will run against master on the other projects and the currently supported stable releases (Icehouse forward) This should help us ensure greater api consistency between releases

18 Tempest Configuration Around 200 config options currently Intent of config file is to tell tempest what can be run What should be run is the domain of the test runner Sample config file documents every option Intended to be able to cover any OpenStack configuration Should have sane defaults (ie match project defaults) To setup a basic config file: Configuring only basic auth and connection info should be needed Also specify flavor and image ids to use for creating servers List which services are available and any required configuration for those services

19 Future Work Expansion of autogenerated negative tests More Unit test coverage Expansion of feature flags in config file Configuration file validation and generation tooling Possibly a new service for running tempest

20 Where to get more information Tempest Repository Tempest Documentation IRC #openstack-qa channel on Freenode

21 Questions?