MTAT : Software Testing

Transcription

1 MTAT : Software Testing Lecture 03: White-Box Testing (Textbook Ch. 5) Spring 2013 Dietmar Pfahl

2 Lecture Chapter 5 White-box testing techniques (Lab 3)

3 Structure of Lecture 3 Introduction to White-Box Testing Control-Flow Testing Data-Flow Testing Loop Testing Fault-Based Testing (Mutation Testing)

4 Testing Strategies requirements output input events Black Box Testing White Box Testing

5 Types of Testing system integration module unit Level of detail portability maintainability efficiency usability reliability functionality white box black box Accessibility Characteristics

6 White-Box Testing (Ch 5) Exhaustive Testing If-then-else There are many possible paths! 5 20 (~10 14 ) different paths loop < 20x Selective Testing

7 Selective Testing a selected path Selective: Control flow testing Data flow testing Loop testing Fault-based testing

8 Code Coverage (Test Coverage) Measures the extent to which certain code items related to a defined test adequacy criterion have been executed (covered) by running a set of test cases (= test suites) Goal: Define test suites such that they cover as many (disjoint) code items as possible

9 Code Coverage Measure Example Statement Coverage (CV s ) Portion of the statements tested by at least one test case. S CV t s 100% S p S : number of statements tested S t p : total number of statements

10 Code Coverage Analysis Code coverage analysis is the process of: Finding areas of a program not exercised by a set of test cases Creating additional test cases to increase coverage Determining a quantitative measure of code coverage, which is (believed to be) a predictor of code quality Code coverage analyzers automate this process Additional aspect of code coverage analysis: Identifying redundant test cases that do not increase coverage

11 Main Classes of Test Adequacy Criteria Control Flow Criteria: Statement, decision (branch), condition, and path coverage are examples of control flow criteria They rely solely on syntactic characteristics of the program (ignoring the semantics of the program computation) Data Flow Criteria: Require the execution of path segments that connect parts of the code that are intimately connected by the flow of data

12 Structure of Lecture 3 Introduction to White-Box Testing Control-Flow Testing Data-Flow Testing Loop Testing Fault-Based Testing (Mutation Testing)

13 Overview of Control Flow Criteria s 1 c 2 d 3 d 1 d 2 d s 6 3 d c 7 d 5 c 14 3 d 8 d 9 c 1 d 10 c 4 d 4 s 4 d 5 d s 12 2 d 13 d 11 If c1 then if c2 then s1 s2 while c3 do s3 else if c4 then repeat s4 until c5 endif endif

14 Overview of Control Flow Criteria s 1 c 2 d 3 d 1 d 2 d s 6 3 d c 7 d 5 c 14 3 d 8 d 9 c 1 d 10 c 4 d 4 s 4 d 5 d s 12 2 d 13 d 11 Statement Coverage Decision (or Branch) Coverage Condition Coverage Condition/Decision Coverage Multiple Condition Coverage Modified Condition Decision Coverage (MC/DC) Simple Paths Linearly Independent Paths Linear Code Sequence and Jump (LCSAJ) Visit-Each Loop All Paths

15 Life Insurance Example bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; else accept := age < 80; return accept d2 d1 c2 d3 s1 d4 c1 d5 c3 d6 s2 d8 d7

16 Statement Coverage Execute each statement at least once Tools are used to monitor execution A possible concern may be: Dead code

17 Statement Coverage bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; else accept := age < 80; return accept AccClient(83, female)->accept AccClient(83, male) ->reject c1 d2 c2 s1 d1 d4 d3 d6 d5 c3 s2 d8 d7

18 Condition Coverage Test all possible conditions in a program A condition in a program may contain: Boolean operators and variables Relational operators Arithmetic expressions.

19 (Simple) Condition Coverage bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; else accept := age < 80; return accept AccClient(83, female)->accept AccClient(83, male) ->reject c1 d2 c2 s1 d1 d4 d3 d6 d5 c3 s2 d8 d7

20 Decision (Branch) Coverage /1 bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; true else accept := age < 80; return accept AccClient(83, female)->accept AccClient(83, male) ->reject c1 d2 c2 s1 d1 d4 d3 d6 d5 c3 s2 d8 d7

21 Decision (Branch) Coverage /2 bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; else accept := age < 80; false return accept AccClient(83, female)->accept AccClient(83, male) ->reject c1 d2 c2 s1 d1 d4 d3 d6 d5 c3 s2 d8 d7

22 Decision (Branch) Coverage /3 bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; true else accept := age < 80; false return accept AccClient(83, female)->accept AccClient(83, male) ->reject c1 d2 c2 s1 d1 d4 d3 d6 d5 c3 s2 d8 d7

23 Advanced Condition Coverage Condition/Decision Coverage (C/DC) as DC plus: every condition in each decision is tested in each possible outcome Modified Condition/Decision coverage (MC/DC) as above plus, every condition shown to independently affect a decision outcome (by varying that condition only) a condition independently affects a decision when, by flipping that condition and holding all the others fixed, the decision changes this criterion was created at Boeing and is required for aviation software according to RCTA/DO-178B Multiple-Condition Coverage (M-CC) all possible combinations of conditions within each decision taken

24 CC, DC, C/DC, M-CC, MC/DC Examples If (A<10 and B>250) then Condition: (TT) A = 2; B = 300 (True) Decision: (TT) A = 2; B = 300 (True) (FT) A = 12; B = 300 (False) Condition/Decision: (TT) A = 2; B = 300 (True) (FF) A = 12; B = 200 (False) Multiple Condition: (TT) A = 2; B = 300 (True) (FT) A = 12; B = 300 (False) (TF) A = 2; B = 200 (False) (FF) A = 12; B = 200 (False) Modified Condition/Decision: (TT) A = 2; B = 300 (True) (FT) A = 12; B = 300 (False) (TF) A = 2; B = 200 (False)

25 Independent Path Coverage McCabe cyclomatic complexity estimates number of test cases needed The number of independent paths needed to cover all paths at least once in a program Visualize by drawing a flow graph CC = #(edges) #(nodes) + 2 CC = #(decisions) + 1 while-loop if-then-else case-of

26 Independent Paths Coverage Example Independent Paths Coverage Requires that a minimum set of linearly independent paths through the control flow-graph be executed This test strategy is the rationale for McCabe s cyclomatic number (McCabe 1976) which is equal to the number of test cases required to satisfy the strategy. Cyclomatic Complexity = = 6

27 Independent Paths Coverage Example Edges: Path1: Path2: Path3: Path4: Path5: Path6:

28 Independent Paths Coverage Example Edges: Why no need to cover Path7??? Path7: Because it equals Path1+Path2-Path3!!! Path1: Path2: P1+P2: Path3: P3:

29 How to find Test Cases? Requirements yielding an outcome at each predicate node contained in a path Consider all requirements together Guess a value that will satisfy these requirements --- Can we improve on this?

30 Symbolic Execution How to find test inputs to exercise a path? Need to make a choice at each predicate node Give a symbolic value to each variable Walk the path collecting requirements on symbolic input Then have a set of (in)equalities to solve Example: Find test cases for each path by symbolic execution

31 Program CFG Program finds the smallest factor of a prime decomposition of a given number smallest(int p) (*p>2*) { int q = 2; while(p mod q > 0 AND q < sqrt p) do q := q+1 ; if (p mod q = 0) then print(q, is factor ) else print(p, is prime ) ; } p. Cyclomatic Complexity: v(g) = = = 3 Sets of linearly independent paths: Example Set 1: Example Set 2:

32 Symbolic Execution Tree Solve path conditions Test input! p: x q: 2 PC: x > 2 and (2 sqrt(x) or x = 2*n) p: x PC: x>2 p: x q: 2 PC: x> PC: path condition p: x q: 2 PC: x > 2 and 2 < sqrt(x) and x <> 2*n p: x is prime q: 2 q: 3 PC: x > 2 and (2 sqrt(x) and x = 2*n + 1) p: x q: 2 and factor 4 2*n + 1 PC: x > 2 and n = 1 x = 3 x = 2*n

33 Control-Flow Coverage Relationships Subsumption: a criterion C1 subsumes another criterion C2, if any test set {T} that satisfies C1 also satisfies C2 Multiple Condition MC/DC C/DC Condition Decision Complete Path Linearly Indep. Path Statement

34 Structure of Lecture 3 Introduction to White-Box Testing Control-Flow Testing Data-Flow Testing Loop Testing Fault-Based Testing (Mutation Testing)

35 Data Flow Testing Identifies paths in the program that go from the assignment of a value to a variable to the use of such variable, to make sure that the variable is properly used. X:=14;.. Y:= X-3;

36 Data Flow Testing Definitions Def assigned or changed Uses utilized (not changed) C-use (Computation) e.g. right-hand side of an assignment, an index of an array, parameter of a function. P-use (Predicate) branching the execution flow, e.g. in an if statement, while statement, for statement. Example: All def-use paths (DU) requires that each DU chain is covered at least once

37 Data Flow Testing Example [1] bool AccClient(agetype age; gndrtype gender) [2] bool accept [3] if(gender=female) [4] accept := age < 85; [5] else [6] accept := age < 80; [7] return accept Considering age, there are two DU paths: (a)[1]-[4] (b)[1]-[6] Test case conditions: AccClient(*, female)-> * AccClient(*, male)-> *

38 Data Flow Testing Example [1] bool AccClient(agetype age; gndrtype gender) [2] bool accept [3] if(gender=female) [4] accept := age < 85; [5] else [6] accept := age < 80; [7] return accept Considering gender, there is one DU path: (a) [1]-[3] Test case conditions: AccClient(*, *)-> *

39 Data Flow Criteria All c-uses All defs All p-uses Weaker All c-uses, some p-uses All p-uses, some c-uses All uses All def-use paths Stronger # tests

40 Structure of Lecture 3 Introduction to White-Box Testing Control-Flow Testing Data-Flow Testing Loop Testing Fault-Based Testing (Mutation Testing)

41 Loop Testing simple loop nested loops concatenated loops unstructured loops

42 Loop Testing: Simple Loops Minimum conditions - simple loops 1. skip the loop entirely 2. only one pass through the loop 3. two passes through the loop 4. m passes through the loop m < n 5. (n-1), n, and (n+1) passes through the loop where n is the maximum number of allowable passes

43 Nested Loops Extend simple loop testing Reduce the number of tests: start at the innermost loop; set all other loops to minimum values conduct simple loop test; add out of range or excluded values work outwards while keeping inner nested loops to typical values continue until all loops have been tested

44 Structure of Lecture 3 Introduction to White-Box Testing Control-Flow Testing Data-Flow Testing Loop Testing Fault-Based Testing (Mutation Testing)

45 Let s count marbles... a lot of marbles Suppose we have a big bowl of marbles How can we estimate how many? I don t want to count every marble individually I have a bag of 100 other marbles of the same size, but a different color What if I mix them?

46 Estimating marbles... I mix 100 black marbles into the bowl Stir well... I draw out 100 marbles at random 20 of them are black How many marbles were in the bowl to begin with? 400

47 Estimating Test Suite Quality Now, instead of a bowl of marbles, I have a program with faults (bugs) I add 100 new faults Assume they are exactly like real faults in every way I make 100 copies of my program, each with one of my 100 new faults I run my test suite on the programs with seeded faults and the tests reveal 20 of the faults (the other 80 program copies do not fail) What can I infer about my test suite?

48 Basic Assumptions We d like to judge effectiveness of a test suite in finding real faults, by measuring how well it finds seeded fake faults. Valid to the extent that the seeded bugs are representative of real bugs Not necessarily identical (e.g., black marbles are not identical to clear marbles); but the differences should not affect the selection E.g., if I mix metal ball bearings into the marbles, and pull them out with a magnet, I don t learn anything about how many marbles were in the bowl

49 Mutation Testing Assumptions Competent programmer hypothesis: Programs are nearly correct Real faults are small variations from the correct program => Mutants are reasonable models of real faulty programs Coupling effect hypothesis: Tests that find simple faults also find more complex faults Even if mutants are not perfect representatives of real faults, a test suite that kills mutants is good at finding real faults too

50 Fault-Based Testing (Mutation Testing) Terminology Mutant new version of the program with a small deviation (=fault) from the original version Killed mutant new version detected by the test suite Live mutant new version not detected by the test suite

51 Mutation Testing A method for evaluation of test suite effectiveness not for designing test cases! 1. Take a program and test data generated for that program 2. Create a number of similar programs (mutants), each differing from the original in a small way 3. The original test data are then run through the mutants 4. If tests detect all changes in mutants, then the mutants are dead and the test suite adequate Otherwise: Create more test cases and iterate 2-4 until a sufficiently high number of mutants is killed

52 Example Mutation Operations Change relational operator (<,>, ) Change logical operator (II, &, ) Change arithmetic operator (*, +, -, ) Change constant name / value Change variable name / initialisation Change (or even delete) statement

53 Example Mutants if (a b) c = a + b; else c = 0; if (a b) c = a + b; else c = 0; if (a && b) c = a + b; else c = 0; if (a b) c = a * b; else c = 0;

54 Types of Mutants Stillborn mutants: Syntactically incorrect killed by compiler, e.g., x = a ++ b Trivial mutants: Killed by almost any test case Equivalent mutant: Always acts in the same behaviour as the original program, e.g., x = a + b and x = a (-b) None of the above are interesting from a mutation testing perspective Those mutants are interesting which behave differently than the original program, and we do not (yet) have test cases to identify them (i.e., to cover those specific changes)

55 Equivalent Mutants if (a == 2 && b == 2) c = a + b; else c = 0; int index=0; while (...) {...; index++; if (index==10) break; } if (a == 2 && b == 2) c = a * b; else c = 0; int index=0; while (...) {...; index++; if (index>=10) break; }

56 Program Example nbrs = new int[range] public int max(int[] a) { int imax := 0; for (int i = 1; i <= range; i++) if a[i] > a[imax] imax:= i; return imax; a[0] a[1] a[2] max TC TC TC }

57 Variable Name Mutant nbrs = new int[range] public int max(int[] a) { int imax := 0; for (int i = 1; i <= range; i++) if i > a[imax] imax:= i; return imax; a[0] a[1] a[2] max TC TC TC }

58 Relational Operator Mutant nbrs = new int[range] public int max(int[] a) { int imax := 0; for (int i = 1; i <= range; i++) if a[i] >= a[imax] imax:= i; return imax; a[0] a[1] a[2] max TC TC TC } Need a test case with two identical max entries in a[.], e.g., (1, 3, 3)

59 Variable Operator Mutant nbrs = new int[range] public int max(int[] a) { int imax := 0; for (int i = 0; i < range; i++) if a[i] > a[imax] imax:= i; return imax; a[0] a[1] a[2] max TC TC TC } Need a test case detecting wrong loop counting

60 In real life... Fault-based testing is a widely used in semiconductor manufacturing With good fault models of typical manufacturing faults, e.g., stuck-at-one for a transistor But fault-based testing for design errors is more challenging (as in software) Mutation testing is not widely used in software industry But plays a role in software testing research, to compare effectiveness of testing techniques Some use of fault models to design test cases is important and widely practiced

61 Why is white-box testing not enough? Missing features Missing code Different states of the software Infinite amount of different paths through the software Different paths can reveal different defects Variations of perspectives Control flow/data flow Input/Output behaviour Test data generation Quality attributes Performance Robustness Reliability Usability

62 Recommended Textbook Exercises Chapter 5 2, 5, 6, 9, 10, 11, 14

63 Next Week Lecture 4: Static Testing (Inspection) and Defect Estimation Lab 3: White-Box Testing In addition to do: Continue working on project Read textbook chapter 5 (available via OIS)