COMP 3705 Advanced Software Engineering: Software Testing

Transcription

1 COMP 3705 Advanced Software Engineering: Software Testing Prof. Matt Rutherford Class (c) 2009 Matthew J. Rutherford For Next Week Readings Chapters 7 and 9 Homework #8 See website Class (c) 2009 Matthew J. Rutherford 1

2 Final Exam In two weeks First hour will be review / summary Exam will be the last 2 hours Open book / notes Comprehensive Mix of short answers (a paragraph) and problems to work Class (c) 2009 Matthew J. Rutherford Today Finish up Mutation Testing Chapter 6: Regression testing Integration testing Process / Test plans Oracles Class (c) 2009 Matthew J. Rutherford 2

3 Midterm Exam Excellent Job! Median / Average: 90% Std. Deviation: 5% Class (c) 2009 Matthew J. Rutherford Testing Programs with Mutation Prog Input test method Create mutants Run equivalence detector Generate test cases Run T on P Define threshold no Run mutants: schema-based weak selective Threshold reached? Eliminate ineffective TCs Fix P no P (T) correct? yes Ammann & Offutt 3

4 Homework #7 Class (c) 2009 Matthew J. Rutherford Why Mutation Works This is not an absolute! The mutants guide the tester to a very effective set of tests A very challenging problem : Find a fault and a set of mutation-adequate tests that do not find the fault Of course, this depends on the mutation operators Ammann & Offutt 4

5 Exercise Collaborate with those in your row to come up with the sneakiest bug you can We ll add the bug as a mutant and see if a mutationadequate test finds it Class (c) 2009 Matthew J. Rutherford Designing Mutation Operators At the method level, mutation operators for different programming languages are similar Mutation operators do one of two things: Mimic typical programmer mistakes ( incorrect variable name ) Encourage common test heuristics ( cause expressions to be 0 ) Researchers design lots of operators, then experimentally select the most useful 5

6 Subsumption of Other Criteria Mutation is widely considered the strongest test criterion And most expensive! Mutation subsumes other criteria by including specific mutation operators Subsumption actually only makes sense for weak mutation other criteria impose local requirements, like weak mutation Node coverage Edge coverage Clause coverage General active clause coverage Correlated active clause coverage All-defs data flow coverage Integration and Object-Oriented Testing In Java, testing the way classes, packages and components are connected Component is used as a generic term This tests features that are unique to object-oriented programming languages inheritance, polymorphism and dynamic binding Integration testing is often based on couplings the explicit and implicit relationships among software components 6

7 Integration Mutation (5.3.2) Faults related to component integration often depend on a mismatch of assumptions Callee thought a list was sorted, caller did not Callee thought all fields were initialized, caller only initialized some of the fields Caller sent values in kilometers, callee thought they were miles Integration mutation focuses on mutating the connections between components Sometimes called interface mutation Both caller and callee methods are considered Four Types of Mutation Operators Change a calling method by modifying values that are sent to a called method Change a calling method by modifying the call Change a called method by modifying values that enter and leave a method Includes parameters as well as variables from higher scopes (class level, package, public, etc.) Change a called method by modifying return statements from the method 7

8 Five Integration Mutation Operators 1. IPVR Integration Parameter Variable Replacement Each parameter in a method call is replaced by each other variable in the scope of the method call that is of compatible type. This operator replaces primitive type variables as well as objects. 2. IUOI Integration Unary Operator Insertion Each expression in a method call is modified by inserting all possible unary operators in front and behind it. The unary operators vary by language and type 3. IPEX Integration Parameter Exchange Each parameter in a method call is exchanged with each parameter of compatible types in that method call. max (a, b) is mutated to max (b, a) Five Integration Mutation Operators (2) 4. IMCD Integration Method Call Deletion Each method call is deleted. If the method returns a value and it is used in an expression, the method call is replaced with an appropriate constant value. Method calls that return objects are replaced with calls to new () 5. IREM Integration Return Expression Modification Each expression in each return statement in a method is modified by applying the UOI and AOR operators. UOR Unary Operator Insertion (add: +, -,! and ~) AOR Arithmetic Operator Replacement (+, -, *, /,**, and %) 8

9 Integration Mutation Operators Example 1. IPVR Integration Parameter Variable Replacement 2. IUOI Integration Unary Operator Insertion MyObject a, b;... callmethod (a); Δ callmethod (b); callmethod (a); Δ callmethod (a++); Integration Mutation Operators Example 3. IPEX Integration Parameter Exchange Max (a, b); Δ Max (b, a); 4. IMCD Integration Method Call Deletion X = Max (a, b); Δ X = new Integer (3); 5. IREM Integration Return Expression Modification int mymethod () { return a; Δ return ++a; } 9

10 Object-Oriented Mutation integration mutation operators These five operators can be applied to non-oo languages C, Pascal, Ada, Fortran, They do not support object oriented features Inheritance, polymorphism, dynamic binding Two other language features that are often lumped with OO features are information hiding (encapsulation) and overloading Even experienced programmers often get encapsulation and access control wrong Object-Oriented Language Features (Java) Method overriding Allows a method in a subclass to have the same name, arguments and result type as a method in its parent Variable hiding Achieved by defining a variable in a child class that has the same name and type of an inherited variable Class constructors Not inherited in the same way other methods are must be explicitly called Each object has a declared type : Parent P; an actual type : P = new Child (); or assignment : P = Pold; Declared and actual types allow uses of the same name to reference different variables with different types 10

11 OO Language Feature Terms Polymorphic attribute An object reference that can take on various types Type the object reference takes on during execution can change Polymorphic method Can accept parameters of different types because it has a parameter that is declared of type Object Overloading Using the same name for different constructors or methods in the same class Overriding A child class declares an object or method with a name that is already declared in an ancestor class Easily confused with overloading because the two mechanisms have similar names and semantics Overloading is in the same class, overriding is between a class and a descendant More OO Language Feature Terms Members associated with a class are called class or instance variables and methods Static methods can operate only on static variables; not instance variables Instance variables are declared at the class level and are available to objects 20 object-oriented mutation operators defined for Java mujava Broken into 4 general categories 11

12 Class Mutation Operators for Java (1) Encapsulation AMC (2) Inheritance HVD, HVI, OMD, OMM, OMR, SKD, PCD (3) Polymorphism ATC, DTC, PTC, RTC, OMC, OMD, AOC, ANC (4) Java-Specific TKD, SMC, VID, DCD OO Mutation Operators Inheritance 2. HVD Hiding Variable Deletion Each declaration of an overriding or hiding variable is deleted. 3. HVI Hiding Variable Insertion A declaration is added to hide the declaration of each variable declared in an ancestor. 4. OMD Overriding Method Deletion Each entire declaration of an overriding method is deleted. 5. OMM Overridden Method Moving Each call to an overridden method is moved to the first and last statements of the method and up and down one statement. 12

13 OO Mutation Operators Example 2. HVD Hiding Variable Deletion 3. HVI Hiding Variable Insertion point int x; int y; point int x; int y; colorpoint int x; Δ1 // int x; int y; Δ2 // int y; colorpoint Δ1 int x; Δ2 int y; OO Mutation Operators Example 4. OMD Overriding Method Deletion 5. OMM Overriding Method Moving point void set (int x, int y) colorpoint void set (int x, int y) Δ // void set (int x, int y) point void set (int x, int y) { width = 5; } colorpoint void set (int x, int y) { super.set (x, y); width = 10;} Δ { width=10; super.set (x, y); } 13

14 OO Mutation Operators Inheritance 6. OMR Overridden Method Rename Renames the parent s versions of methods that are overridden in a subclass so that the overriding does not affect the parent s method. 7. SKD Super Keyword Deletion Delete each occurrence of the super keyword. 8. PCD Parent Constructor Deletion Each call to a super constructor is deleted. OO Mutation Operators Example 6. OMR Overriding Method Rename point void set (int x, int y) Δ void setp (int x, int y) void setdimension (int d) { set (x, y); Δ setp (x, y); } point p; p = new colorpoint (); p.set (1, 2); p.setdimension (3); 7. SKD Super Keyword Deletion point int getx() colorpoint int getx () { return super.x; Δ return x; } colorpoint void set (int x, int y) 14

15 OO Mutation Operators Example 8. PCD Parent Constructor Deletion point point (int x, int y) colorpoint colorpoint (int x, int y, int color) { super (x, y); Δ // super (x, y); } OO Mutation Operators Language Specific 17. TKD this Keyword Deletion Each occurrence of the keyword this is deleted. 18. SMC Static Modifier Change Each instance of the static modifier is removed, and the static modifier is added to instance variables. 19. VID Variable Initialization Deletion Remove initialization of each member variable. 20. DCD Default Constructor Delete Delete each declaration of default constructor (with no parameters). 15

16 OO Mutation Operators Example 17. JTD This Keyword Deletion 18. JSD Static Modifier Change point void set (int x, int y) { this.x = x; Δ1 x = x; this.y = y; Δ2 y = y; } point public static int x = 0; Δ 1 public int x = 0; public int Y = 0; Δ2 public static int y = 0; OO Mutation Operators Example 19. VID Variable Initialization Deletion 20. DCD Default Constructor Delete point int x = 0; Δ int x; point point() { } Δ // point() { } 16

17 Integration Mutation Summary Integration testing often looks at couplings We have not used grammar testing at the integration level Mutation testing modifies callers and callees OO mutation focuses on inheritance, polymorphism, dynamic binding, information hiding and overloading The access levels make it easy to make mistakes in OO software mujava is an educational / research tool for mutation testing of Java programs Chapter 6: Practical Issues Regression Testing Integration Testing Process Test Plans Oracles Class (c) 2009 Matthew J. Rutherford 17

18 The Toolbox Chapters 1-5 fill up a toolbox with useful criteria for testing software To move to level 3 (reducing risk) or level 4 (mental discipline of quality), testing must be integrated into the development process Most importantly : In any activity, knowing the tools is only the first step The key is utilizing the tools in effective ways Topics : Integrating software components and testing Integrating testing with development Test plans Checking the output Introduction to Software Testing (Ch 6), Regression Testing (6.1) Most software today has very little new development Correcting, perfecting, adapting, or preventing problems with existing software Composing new programs from existing components Applying existing software to new situations Because of the deep interconnections among software components, changes in one method can cause problems in methods that seem to be unrelated Not surprisingly, most of our testing effort is regression testing Large regression test suites accumulate as programs (and software components) age Introduction to Software Testing (Ch 6), 18

19 Automation and Tool Support Too many tests to be run by hand Tests must be run and evaluated quickly often overnight, or more frequently for web applications Testers do not have time to view the results by inspection Types of tools : Capture / Replay Capture values entered into a GUI and replay those values on new versions Version control Keeps track of collections of tests, expected results, where the tests came from, the criterion used, and their past effectiveness Scripting software Manages the process of obtaining test inputs, executing the software, obtaining the outputs, comparing the results, and generating test reports Tools are plentiful and inexpensive (often free) Introduction to Software Testing (Ch 6), Managing Tests in a Regression Suite Test suites accumulate new tests over time Test suites are usually run in a fixed, short, period of time Often overnight, sometimes more frequently, sometimes less At some point, the number of tests can become unmanageable We cannot finish running the tests in the time allotted We can always add more computer hardware But is it worth it? How many of these tests really need to be run? Introduction to Software Testing (Ch 6), Ammann & Offutt 38 19

20 Policies for Updating Test Suites Which tests to keep can be based on several policies Add a new test for every problem report Ensure that a coverage criterion is always satisfied Sometimes harder to choose tests to remove Remove tests that do not contribute to satisfying coverage Remove tests that have never found a fault (risky!) Remove tests that have found the same fault as other tests (also risky!) Reordering strategies If a suite of N tests satisfies a coverage criterion, the tests can often be reordered so that the first N-x tests satisfies the criterion so the remaining tests can be removed Introduction to Software Testing (Ch 6), Ammann & Offutt 39 When a Regression Test Fails Regression tests are evaluated based on whether the result on the new program P is equivalent to the result on the previous version P-1 If they differ, the test is considered to have failed Regression test failures represent three possibilities : The software has a fault Must fix the fix The test values are no longer valid on the new version Must delete or modify the test The expected output is no longer valid Must update the test Sometimes hard to decide which!! Introduction to Software Testing (Ch 6), Ammann & Offutt 40 20

21 Evolving Tests Over Time Changes to external interfaces can sometimes cause all tests to fail Modern capture / replay tools will not be fooled by trivial changes like color, format, and placement Automated scripts can be changed automatically via global changes in an editor or by another script Adding one test does not cost much but over time the cost of these small additions start to pile up Introduction to Software Testing (Ch 6), Ammann & Offutt 41 Choosing Which Regression Tests to Run When a small change is made in the software, what portions of the software can be impacted by that change? More directly, which tests need to be re-run? Conservative approach : Run all tests Cheap approach : Run only tests whose test requirements relate to the statements that were changed Realistic approach : Consider how the changes propagate through the software Clearly, tests that never reach the modified statements do not need to be run Lots of clever algorithms to perform CIA have been invented Few if any available in commercial tools Introduction to Software Testing (Ch 6), Ammann & Offutt 42 21

22 Rationales for Selecting Tests to Re-Run Inclusive : A selection technique is inclusive if it includes tests that are modification revealing Unsafe techniques have less than 100% inclusiveness Precise : A selection technique is precise if it omits regression tests that are not modification revealing Efficient : A selection technique is efficient if deciding what tests to omit is cheaper than running the omitted tests This can depend on how much automation is available General : A selection technique is general if it applies to most practical situations Introduction to Software Testing (Ch 6), Ammann & Offutt 43 Identifying Correct Outputs (6.5) With simple software methods, we have a very clear idea whether outputs are correct or not But for most programs it s not so easy This section presents four general methods for checking outputs: Direct verification Redundant computation Consistency checks Data redundancy Introduction to Software Testing (Ch 6), 22

23 Direct Verification Appealing because it eliminates some human error Fairly expensive requiring more programming Verifying outputs is deceptively hard One difficulty is getting the post-conditions right Not always possible we do not always know the correct answer Flow calculations in a stream the solution is an approximation based on models and guesses; we don t know the correct answers! Probability of being in a particular state in a Petri net again, we don t know the correct answer Introduction to Software Testing (Ch 6), Ammann & Offutt 45 Direct Verification Example Consider a simple sort method Post-condition : Array is in sorted order Input Output Oops! Output Oops! Post-condition : Array sorted from lowest to highest and contains all the elements from the input array Input Output Oops! Post-condition : Array sorted from lowest to highest and is a permutation of the input array Introduction to Software Testing (Ch 6), Ammann & Offutt 46 23

24 Direct Verification Example Cont. Input : Array A Make copy B of A Sort A // Verify A is a permutation of B Check A and B are of the same size For each object in A Check if object appears in A and B the same number of times // Verify A is ordered for each index I but last in A Check if A [i] <= A [i+1] This is almost as complicated as the sort method under test! We can easily make mistakes in the verification methods Introduction to Software Testing (Ch 6), Ammann & Offutt 47 Redundant Computation Write two programs check that they produce the same answer Very expensive! Problem of coincident failures That is, both programs fail on the same input Sadly, the independent failure assumption is not valid in general This works best if completely different algorithms can be used Not clear exactly what completely different means Consider regression testing Current software checked against prior version Special form of redundant computation Clearly, independence assumption does not hold But still extremely powerful Introduction to Software Testing (Ch 6), Ammann & Offutt 48 24

25 Consistency Checks Check if a probability is negative or larger than one Check assertions or invariants No duplicates Cost is greater than zero Internal consistency constraints in databases or objects These are only partial solutions and does not always apply, but is very useful within those limits Introduction to Software Testing (Ch 6), Ammann & Offutt 49 Data Redundancy Check for identities Testing sin (x) : sin(a+b) = sin(a)cos(b) + cos(a)sin(b) Choose a at random Set b=x-a Note failure independence of sin(x), sin(a) Repeat process as often as desired; choose different values for a Possible to have arbitrarily high confidence in correctness assessment Inserting an element into a structure and removing it These are only partial solutions and does not always apply, but is very useful within those limits Introduction to Software Testing (Ch 6), Ammann & Offutt 50 25

26 Exercise Think of the software component in the systems you work on that is the most difficult to verify. Identify possibilities for the four oracle techniques discussed: Direct verification Redundant computation Consistency checks Data redundancy Class (c) 2009 Matthew J. Rutherford Integration and Testing (6.2) The polite word for this is risky Less polite words also exist The usual method is to start small, with a few classes that have been tested thoroughly Add a small number of new classes Test the connections between the new classes and pre-integrated classes Integration testing : testing interfaces between classes Should have already been tested in isolation (unit testing) Introduction to Software Testing (Ch 6), 26

27 Methods, Classes, Packages Integration can be done at the method level, the class level, package level, or at higher levels of abstraction Rather than trying to use all the words in every slide Or not using any specific word We use the word component in a generic sense A component is a piece of a program that can be tested independently Integration testing is done in several ways Evaluating two specific components Testing integration aspects of the full system Putting the system together piece by piece Introduction to Software Testing (Ch 6), Software Scaffolding Scaffolding is extra software components that are created to support integration and testing A stub emulates the results of a call to a method that has not been implemented or integrated yet A driver emulates a method that makes calls to a component that is being tested Driver Makes calls to methods in CUT Component Under Test (example: ADT) Stubs Emulates methods the CUT calls Introduction to Software Testing (Ch 6), 27

28 Stubs The first responsibility of a stub is to allow the CUT to be compiled and linked without error The signature must match What if the called method needs to return values? These values will not be the same the full method would return It may be important for testing that they satisfy certain limited constraints Approaches: Return constant values from the stub Return random values Return values from a table lookup Return values entered by the tester during execution Processing formal specifications of the stubbed method More costly / more effective Introduction to Software Testing (Ch 6), Mock Objects How can you unit-test program units that have dependencies? Mock Objects are a special kind of stub that is used when testing higher-level OO classes Class (c) 2009 Matthew J. Rutherford 28

29 Mock Objects Class (c) 2009 Matthew J. Rutherford Mock Objects Class (c) 2009 Matthew J. Rutherford 29

30 Drivers Many good programmers add drivers to every class as a matter of habit Instantiate objects and carry out simple testing Criteria from previous chapters can be implemented in drivers Test drivers can easily be created automatically Values can be hard-coded or read from files Introduction to Software Testing (Ch 6), Class Integration and Test Order (CITO) Old programs tended to be very hierarchical Which order to integrate was pretty easy: Test the leaves of the call tree Integrate up to the root Goal is to minimize the number of stubs needed OO programs make this more complicated Lots of kinds of dependencies (call, inheritance, use, aggregation) Circular dependencies : A inherits from B, B uses C, C aggregates A CITO : Which order should we integrate and test? Must break cycles Common goal : least stubbing Designs often have few cycles, but cycles creep in during implementation Introduction to Software Testing (Ch 6), 30

31 Test Activities Introduction to Software Testing (Ch 6), Test Activities (2) Introduction to Software Testing (Ch 6), 31

32 Test Plans (6.4) The most common question I hear about testing is How do I write a test plan? This question usually comes up when the focus is on the document, not the contents It s the contents that are important, not the structure Good testing is more important than proper documentation However documentation of testing can be very helpful Most organizations have a list of topics, outlines, or templates Introduction to Software Testing (Ch 6), Standard Test Plan ANSI / IEEE Standard is ancient but still used Many organizations are required to adhere to this standard Unfortunately, this standard emphasizes documentation, not actual testing often resulting in a well documented vacuum Introduction to Software Testing (Ch 6), 32

33 Types of Test Plans Mission plan tells why Usually one mission plan per organization or group Least detailed type of test plan Strategic plan tells what and when Usually one per organization, or perhaps for each type of project General requirements for coverage criteria to use Tactical plan tells how and who One per product More detailed Living document, containing test requirements, tools, results and issues such as integration order Introduction to Software Testing (Ch 6), Purpose Test Plan Contents System Testing Target audience and application Deliverables Information included Introduction Test items Features tested Features not tested Test criteria Pass / fail standards Criteria for starting testing Criteria for suspending testing Requirements for testing restart Hardware and software requirements Responsibilities for severity ratings Staffing & training needs Test schedules Risks and contingencies Approvals Introduction to Software Testing (Ch 6), 33

34 Test Plan Contents Tactical Testing Purpose Outline Test-plan ID Introduction Test reference items Features that will be tested Features that will not be tested Approach to testing (criteria) Criteria for pass / fail Criteria for suspending testing Criteria for restarting testing Test deliverables Testing tasks Environmental needs Responsibilities Staffing & training needs Schedule Risks and contingencies Approvals Introduction to Software Testing (Ch 6), 34