15. Regression testing

Transcription

1 Outline 15. Regression testing Tom Verheyen, Jelle Slowack, Bart Smets, Glenn Van Loon Introduction - What, why, when, how - Regression faults - Test automation - Test suite maintenance - Reducing a test suite Patterns - Test All - Test Risky Use Cases - Test by Profile - Test Changed Segments - Test Within Firewall 2 What is regressions testing? Baseline version: component/system passed test suite Delta version: component/system not passed regression test Delta build: executable configuration SUT containing baseline and delta components Regression test case: test case baseline passed, expected to pass on delta build Regression test suite: composition of regression test cases Regression fault: revealed by test case that no longer passes 3 Why should we regression test? Ariane 5 First test flight: crash! - Data conversion from 64-bit floating point to 16-bit signed integer - Software (Ada) reused from Ariane 4 - No regression tests done Estimate cost: $350 million - $2,5 billion 4 1

2 When can we use regression test? What kind of errors can be found? - Delta side effects - Delta - baseline incompatibilities - Undesirable feature interactions baseline - delta - Bad fixes IBM: bad fixes injection rates 2% - 20% When should we run a regression test? When a new subclass has been developed When a superclass is changed When a server class is changed When a bug fix has been completed When a new system build has been generated When a new increment is generated for system scope integration testing When a system has stabilized and the final release build has been generated 5 6 How to run a regression test? Basic procedure same for all situation 1. Remove broken test cases 2. Choose a regression test suite (full/ reduced: see Regression patterns) 3. Set up test configuration 4. Run the regression test suite 5. Do something with the results Regression faults Combination of baseline B and delta D fails - D has side effect on B - C is client/server of B - D is incompatible with B (Ariane 5!) - D s and B s contract specifications differ -... (Extended list of errors: Binder p ) Faults at intercomponent, subsystem and system scope 7 8 2

3 Test automation Manual testing is not a good thing - Person must reenter it and judge result - If number of tests and time to rerun increases: 1. Rerun fewer baseline test cases + focus on tests to validate final requirements 2. Add more people to enter test cases and judge results (usually not possible) - Time to ship decrease # regressions tests approaches 0 9 Test automation: requirements Effective automated RT requires capabilities: - Version control, compare baseline delta results, smart comparator,... - Want to know more? Binder p.764 Test environment should be controllable - Some variable factors can result in less-than identical test configurations Test environment itself, generator with clock as seed,... - Extended list of factors: Binder p Test suite maintenance A regression test suite can rapidly grow large Test decay is inevitable - Broken testcase, redundant test cases, Example: aerospace manufacturer - Test suite of test cases +/- 90% test cases: redundant Documentation inadequate Some parts not tested: 3000 new test cases only test cases remained 11 Test suite maintenance: procedure 1. Run baseline test suite on delta build. Remove broken test cases. 2. Correct relevant bugs. 3. Merge delta component scope tests with baseline test suite. 4. Use a coverage analyzer. 5. Rerun the test suite (to know coverage) 6. Analyze tests with same entry-exit paths (still necessary?) 7. If code is not tested: develop new test case. 8. Rerun test suite again. 9. Check revised baseline test suite as new baseline test suit. 12 3

4 Reducing a test suite Even with maintance, tests can become too large - Use reduced regression test, with selected tests How to safely reduce a regression test? - Safe: all tests that could possibly exhibit different output when run on the delta build 13 Reducing a test suite 4 criteria - Inclusiveness: percentage of BL tests that may show regression faults and are in the RT suite Safe RT suite is 100% inclusive - Precision: percentage BL tests in a RT that cannot reveal regression faults and are not selected in the reduced RT suite 100 test cases: 1 cannot reach changed code 99% precise Precise RT suite: no tests that cannot produce different output - Efficiency: cost of identifying a reduced RT suite - Generality: range of application for the selection strategy Retest All: default 2. Retest Risky Use Cases 3. Retest by Profile 4. Retest Changed Segments Test patterns 1. Retest All Rerun entire baseline test suite Context - Any scope Fault model - Failures because of incompatibility, side effects, undesirable feature interaction

5 1. Retest All Strategy - Test model: baseline test suite is reused - Test procedure: rerun after removing broken tests - Oracle: should be same results as previous run - Automation: see slides Entry criteria - Delta components pass component scope testing - Suitable baseline test suite exists - Test environment has same configuration as previous runs (see slide 13) 1. Retest All Exit criteria - No pass test cases reveal bugs that are acceptable - Remaining test cases pass Consequences - Inclusiveness, precision, efficiency, generality - See further slides Retest Risky Use Cases Use risk-based metrics to select partial RT suite Context - Too few time, personnel or equipement Fault model - Failures because of incompatibility, side effects, undesirable feature interaction 2. Retest Risky Use Cases Strategy - Test model: risk criteria to select test cases Suspicious use cases: unstable, complex use cases Critical use cases: necessary for safe, effective operation - Test procedure: Identify, develop en run test suite - Oracle: should be same results as previous run - Automation: see slides Entry criteria

6 2. Retest Risky Use Cases Exit criteria Consequences - Inclusiveness, precision, efficiency, generality - See further slides Intent - Partial regression test - Budget-constrained Context - Too few time, personnel or equipement - Greatest dependability within budget? - Applicable at any scope Fault model - Allocate tests by profile: frequency-based testing vs reliability Strategy: Test model - # test cases for each use case? => total budget Example Total budget 6000 mins Run 1 test case 5 mins Chance on bug reveal 0.5 % Bug fix requires 200 mins Baseline test suite test cases How many of test cases (T) can we run? 6000 = 5T + (0.005 x 200) 5 => T

7 Allocation of Regression Test Time by Use Case frequency Use case Probability Number of tests Cash withdrawal Deposit Balance inquiry TOTAL Identify, develop, run reduced test suite 2. Verify: critical use cases & use case variants included Strategy: Oracle - should be same results as previous run Strategy: Automation - Slides Automate selection in prev. example: of test cases => ¼ test random selector (each test 25% chance of being selected) Consequences - Inclusiveness, precision, efficiency, generality - See further slides Entry & exit criteria

8 Intent - Partial regression test - Code-change analysis Context - Too few time, personnel or equipement - Applicable at class, cluster or subsystemscope Fault model - See pattern Retest All Strategy: Test model - Select all baseline tests that have reached: A changed code segment Or a deleted code segment -> Graph walk technique (Rothermel and Harrold) Strategy: Test model (2) - Basic model does NOT consider: Inheritance Dynamic binding Data flow Control flow Other dependencies arising from state-based behaviour, iteration or recursion 1. Obtain a report from coverage analyzer, that lists codesegments by testcase Testcase T1 T2 T3 Codesegments B1,B3,B6,B7 B1,B4,B8 B2,B

9 2. Extract pairs with each segment and testcase Testcase Codesegments T1 T2 T3 B1,B3,B6,B7 B1,B4,B8 B2,B5 PAIR B1 T1 B3 T1 B6 T1 B7 T1.. B5 T Use a version control tool to generate a report on the changes between baseline & delta Segment B1 B2 B3.... B9 Change Same Deleted Changed New Concatenate test (step 2) & change file (step 3). Sort! Concat B1 Same B1 T1 B1 T2 B2 Deleted B2 T Selection rules for tests: Tests under same skip Tests under delete include Tests under changedinclude Tests under new skip Concat B1 Same B1 T1 B1 T2 B2 Deleted B2 T

10 Strategy: Oracle Strategy: Automation - see slides Entry & exit criteria Consequences - See further slides Intent - Partial regression test - Code-dependency analysis Context - Too few time, personnel or equipement - Applicable at class, cluster or subsystemscope Fault model - See pattern Retest All Strategy: Test Model - Firewall = set of components whose test cases will be included in a regression test - Firewall set is identified by analysis of changes to each component in the SUT and its dependencies. Strategy: Test Model (2) Each pair of components (A,B) is analyzed Either or both may be changed: 1. Contract change alters external interface and/or externally visible contract (eg. alteration of public methods,..) 2. Implementation change other changes that are not visible to clients

11 Strategy: Test Model (3) Dependency relationship between each pair is used to select test cases 4 Relationships: 1. B uses A 2. B is a subclass of A 3. B overrides A 4. B is a server of A Strategy: Test Model (4) For each relationship: baseline testcase that apply to A,B or AB may be reused in one of 4 ways: Level 0 No testcases can be rerun Level 1 The state setup and test messages can be rerun. Expected results must be redeveloped. The sequence of testcases may need to be reworked. Level 2 The state setup, test messages and expected results can be rerun. The sequence of testcases may need to be reworked. Level 3 The test cases can be rerun as is Strategy: Test Model (5) Example - Class Account (A) and class Money (B) - Account uses a variable amount of type Money - Implementation change to account Changes Retesting requirement A B Dependency A Implementation None B is server of A Relationship: Money is a server of Account -.. Level 2 reuse of the test cases for Account (decision table for selecting regression tests 15.7 p 791)

12 1. Develop a dependency matrix 2. Apply the decision rules (table 15.7) 3. Rerun tests according to the testing levels Strategy: Oracle Strategy: Automation - see slides Entry & exit criteria Consequences - See next slide! Pattern Comparison Inclusiveness All: Safe, all tests selected Risky: Unsafe, selection of testcases not by analysis / dependencies Profile: Unsafe, same as Risky Use Case Changed: Safe, all baseline tests that can produce a different result are selected Firewall: Safe Pattern Comparison Precision All: no tests are skipped, least precise Risky: some tests that could be skipped Profile: ditto Changed: few tests that could be skipped most precise of the white box partial regression strategies Firewall: few tests that could be skipped

13 Pattern Comparison Pattern Comparison Efficiency All: lowest analysis & setup cost high run cost Risky: time and cost are constrained (budget) selection based on use cases (<-> implementation) => analysis can be done without code analyzers / technical knowledge of SUT Profile: Same as Risky Use Case but if the operational profile is established, then low cost of selection analysis Efficiency (2) Changed: High setup cost Run cost = size of deltas Cost can be greater than Retest All: reselection Firewall: Highest setup cost Run cost = size of firewall Pattern Comparison Generality All: applicable in any circumstances easy to set up Risky: nearly any circumstances, any scope Profile: many circumstances, system scope with accurate operational profile Changed: applicable at class scope requires skills in code analysis Firewall: applicable at cluster scope requires skills in code analysis 51 13