Regression Tes+ng. Midterm Wednesday Oct. 26, 7:00-8:30pm Room 142

Transcription

1 Regression Tes+ng Computer Science Fall 2011 Prof. L. J. Osterweil Material adapted from slides originally prepared by Prof. L. A. Clarke Midterm Wednesday Oct. 26, 7:00-8:30pm Room 142 Closed book eam Includes all the material covered up through the end of last Thursday s lecture Need to understand the concepts, apply the concepts, know the basic terminology

2 Review Session for Midterm will be held in CS 150 today at 4:00 Regression Tes+ng Retes+ng sovware aver it has been modified Goal is to: Make sure that required changes work as specified AND that unchanged requirements are s+ll met Most sovware tes+ng is actually regression tes+ng Most sovware development is actually evolu+on

3 SoVware Development/V&V Lifecycle Requirements! Specification! Architecting! Implementation! Designing! Coding! System Test! Plan! Software Sys.! Test Plan! Integration! Test Plan! Unit! Test! Plan! System! Testing! Software! Sys Testing! Integration! Testing! Unit Testing! Validation Flow of control edge (the ImmFol relation) Data Flow edge (artifacts flow from tail to head) Regression Tes+ng Changed! Requirements! Specification! System Test! Plan! Architecting! Software Sys.! Test Plan! Implementation! Designing! Integration! Test Plan! Unit! Test! Plan! Coding! System! Testing! Software! Sys Testing! Integration! Testing! Unit Testing!

4 Why is regression tes+ng a problem? Large systems can take a l.o.n.g +me to retest e.g., 6 months of regression tes+ng before every release Some+mes it is difficult and +me consuming to create the tests Some+mes it is difficult and +me consuming to evaluate the tests e.g., may not be able to automa+cally determine if the results are correct For eis+ng and/or new test cases May require a person in the loop (GUI and simula+on eamples) to create and evaluate the results Cost of regression tes+ng can prevent sovware improvements Regression Tes+ng Primarily selec+ng from eis+ng test cases Plus, adding some new test cases Perhaps, dele+ng or upda+ng some old test cases Usually view this as dele+on plus addi+on

5 Steps in regression tes+ng Given: a program P originally tested with test set T producing results R and a modified version of the program P Iden+fy the changes to P Select T a subset of T to eecute P Maybe not a proper subset Test P with T : reestablish correctness of P with resp. to T Create new tests T as necessary Automated support for Regression tes+ng Test data selec+on support Select a subset of the eis+ng test cases Based on what? May not be a proper subset Select new test data to eercise new func+onality Based on what? Typically coverage criteria and func+onal test cases

6 Eample: Dealing with new code a c f Dealing with new code a b Test cases that eercise any of the immediate predecessor nodes of a new statement are assumed to eercise the new statement c f Also need to select new test cases to improve branch coverage

7 Regression Tes+ng Criteria (Rothermel and Harrold) A test case t T is fault- revealing if it produces wrong outputs for P In general, can not determine which elements of T are fault revealing A test case t T is modifica+on- revealing if it produces different outputs for P than for P Modifica+on- revealing test cases may also be fault revealing In general, can not determine which elements of T are modifica+on revealing A test case t T is modifica+on- traversing if it eecutes a statement in P that has changed Modifica+on- traversing cases include modifica+on revealing cases Such test cases can be computed based upon iden+fica+on of Changed code Removed code New code Retest selec+on Some Alternatives: Fault revealing (Conservative and Precise) -impossible to compute Modification revealing (Conservative, but not precise) -also impossible to compute Modification traversing -easier to compute Retest all (Conservative, but not precise )- trivial to compute

8 SAFE Regression Tes+ng Criteria A selec+on criterion is safe if all of the eis+ng test cases that could epose a fault have been selected In other words, (T - T ) cannot uncover any faults in the system For t (T - T ), then either t is no longer in the domain or Statements eecuted by t are not impacted by the changes to the code I.e., no statements in the eecu+on of these test cases were changed An Empirical Study T. L. Graves, M. J. Harrold, J. M. Kim, A. Porter and G. Rothermel, "An Empirical Study of Regression Test Selec+on Techniques," ACM Transac+ons on SoVware Engineering and Methodology, 10 (2), April 2001, pp Eperiment to evaluate Fault detec+on effec+veness Regression tes+ng is usually not more effec+ve than the original test set Retest- all has good fault detec+on effec+veness, but may not be cost effec+ve Cost effec+veness Are there techniques that have similar fault detec+on effec+veness but the cost of the test case selec+on analysis is significantly less than the test cases it eliminates? Cost to compute T can be considerable but Should be less than the cost of eecu+ng (T- T ) while s+ll assuring lijle or no loss in effec+veness

9 Programs studied 7 C++ programs from Siemens LOCs Many versions of each 9-41 versions Each version had one seeded fault 2 larger programs 6 Klocs/ 33 versions/mul+ple faults 49 Klocs/ 5 versions/ mul+ple faults Admittedly these are small programs Test pools, test suites, test cases Test pools Test suites with known edge coverage 1000 edge- coverage test suites selected from the pool randomly Selected test suites to achieve edge- coverage Assume n k test cases in the kth suite 1000 non- edge coverage test suites Selected randomly from the pool Kth test suite has n k test cases, so non- edge coverage has a buddy edge- coverage test suite of the same size

10 Regression tes+ng techniques studied Minimiza+on - select test cases from the test suite so that every edge or node associated with the change is eercised oven resulted in a single test case Safe select every test case in a suite that eercises a statement that has been deleted, modified, or is new Regression tes+ng techniques studied (con+nued) Data flow- select every test case in a test suite that eercises a def- use pair affected by a deleted or modified statement Not quite safe (what about new statements?) Not full dependence Random- select 25%/50%/75% of the test cases in a suite chosen randomly Retest- all

11 Test case size reduc+on Random and test- all select a test suite size that is 25%, 50%, 75%, and 100%, respec+vely, of T by defini+on Minimiza+on: ~1% test suite size Safe:~60% test suite size Data flow: 54% test suite size Fault detec+on effec+veness For minimiza+on, random, and test- all The larger the test suite size the bejer the fault detec+on Improvement diminishes as the % gets higher testall 100% random75% random 50% random25% minimization Effectiveness (%) But, of course, all faults are not equal in importance

12 Fault detec+on effec+veness: safe Safe test suite size averaged 60% of original, but only performed slightly bejer than random (75%) There was significant variance in the test suite reduc+on Some programs resulted in almost no reduc+on in original test suite size Larger programs tended to have a larger reduc+on in the test suite size for some programs the payoff was significant best case: 5% of the test cases were required Size reduc+on oven depended on where a change was located a b d c e Changes to leaf nodes/ components required few test cases Method Invocation Graph f Changes to root node/ component required all test cases

13 Regression Tes+ng Conclusion Regression tes+ng is a serious problem for some systems Want to reduce test suite size but not fault detec+on Need selec+on and priori+za+on models to select test cases Priori+za+on models Safe selec+on techniques might s+ll return too many test cases Need to combine with priori+za+on techniques Simple selec+on techniques, such as random selec+on, might be most cost efficient