Structural Testing & Mutation

Size: px

Start display at page:

Download "Structural Testing & Mutation"

Adam Arnold
5 years ago
Views:

1 Structural Testing & Mutation Filippo Ricca DISI, Università di Genova, Italy 1

2 White vs. Black box testing A white box testing is based upon explicit knowledge of the SUT and its structure Called also structural testing Statement coverage testing is an example The aim is to create enough testcases to ensure every statement is executed at least once Inputs Outputs A black box testing approach will device testcases without any knowledge of the SUT or any aspect of its structure Called also functional testing Usually testcases are generated starting from requirements/specifications Inputs Outputs 2

3 What is adequacy? Test suite T P Fault detection what if no faults are found? How good this test suite is? Has P been tested thoroughly?, or Is T adequate? Two possibilities: Test Coverage and program mutation 3

4 Test Coverage 4

5 Test Coverage Coverage can be based on: source code model control flow graph (extended) finite state machines data flow graph requirements checklist... 5

6 Coverage: what to measure? For any coverage measure, we need: A coverage unit: an element with the properties: We can count the total number of units in the software We can identify which units were hit during a single execution run This means that we can determine the percentage of units hit during one or more execution runs 89% 6

7 Control flow coverage Statement coverage Branch coverage (also called decision coverage) Minimum coverage specified by the IEEE unit test standard Multiple Condition coverage Covers combinations of condition in decisions If (x=6) && (y=7) then Path coverage 100% path coverage impossible in practice Loops... 7

8 Control Flow graph 1 int proc(int a, int b, int x) { if ((a>1) && (b==0)) // 1 { x = x/a; // 3 if ((a==2) (x>1)) // 4 { x = x+1; // 5 return x; // a > 1 AND b = 0 false a == 2 OR x>1 false true true x x/a x x

9 Statement Coverage Criterion: All statements must be covered during test execution This is the weakest form of coverage Some branches may be missed Procedure: Find paths that cover all statements Choose input data that will result in the selected paths 9

10 Statement Coverage The following path is sufficient for statement coverage: 1 a > 1 AND b = 0 true false x x/a 3 int proc(int a, int b, int x) Possible input: 4 a == 2 OR x>1 true a = 2, b = 0, x = 4 6 false x x

11 Not covered branches 1 a > 1 AND b = 0 true 2 false x x/a 3 4 a == 2 OR x>1 true 6 false x x

12 Branch Coverage Criterion: At any branch point, each branch must be covered during test execution The true and false branch of a 2-way if statement Each case in a switch statement Loops Procedure: While, for, goto,. Find paths that cover all branches Choose input data that will result in the selected paths Branch coverage includes statement coverage! 12

13 Branch Coverage The following paths are sufficient for branch coverage: 1 a > 1 AND b = 0 true Possible input: 1. a = 2, b = 2, x = a = 3, b = 0, x = false a == 2 OR x>1 false true x x/a x x

14 Multiple Condition Coverage Criterion: Every atomic (i.e. does not include AND or OR) condition must be true and false at some point during test execution In a compound logical statement (i.e. includes AND and OR), every combination of atomic conditions must be covered during test execution Achieving multiple condition coverage also satisfies statement and branch coverage 14

15 Multiple Condition Coverage int proc(int a, int b, int x) { Need cases where if ( (a>1) && (b==0) ) 1. { a > 1 is true and b = 0 is true x = x/a; 2. a > 1 is true and b = 0 is false 3. a > 1 is false and b = 0 is true if ( (a==2) (x>1) ) 4. a > 1 is false and b = 0 is false { 5. a = 2 is true and x > 1 is true x = x+1; 6. a = 2 is true and x > 1 is false 7. a = 2 is false and x > 1 is true return x; 8. a = 2 is false and x > 1 is false 15

16 Multiple Condition Coverage 1. a > 1 is true and b = 0 is true 2. a > 1 is true and b = 0 is false 3. a > 1 is false and b = 0 is true 4. a > 1 is false and b = 0 is false 5. a = 2 is true and x > 1 is true 6. a = 2 is true and x > 1 is false Possible input: a = 2, b = 0, x = 2 [1][5] a = 2, b = 1, x = 0 [2][6] a = 0, b = 0, x = 2 [3][7] a = 0, b = 1, x = 0 [4][8] 7. a = 2 is false and x > 1 is true 8. a = 2 is false and x > 1 is false 16

17 Multiple Condition Coverage Multiple condition coverage covers all branches and statements. Input values: a = 2, b = 0, x = 2 a = 2, b = 1, x = 0 a = 0, b = 0, x = 2 a = 0, b = 1, x = 0 Paths covered Equal a > 1 AND b = 0 false a == 2 OR x>1 7 1 false true true x x/a x x

18 All Paths Coverage Criterion: All paths through the code must be covered This is typically infeasible when loops are present A version of this coverage with loops is to treat loops as having two paths: 1. The loop is executed (normally, once) 2. The loop is skipped Some paths may also be infeasible because there is no combination of data conditions that permit a path to be taken See example below 18

19 All Paths Coverage Set of all paths: a > 1 AND b = 0 false true x x/a 3 Input values: a = 0, b = 1, x = 0 a = 3, b = 0, x = 0 a = 2, b = 1, x = 0 a = 2, b = 0, x = a == 2 OR x>1 false true x x

20 Comparison From the previous two examples, we can see that: Multiple condition coverage does not imply all paths coverage All paths coverage does not imply multiple condition coverage 20

21 Infeasible paths Set of paths: true a > false x x/a 3 To be able to take this path, we would have to have: 4 a > 4 true a <= 1 AND a > 4 which is logically impossible! 6 false x x

22 Code Coverage Tools Clover: Emma: Coverlipse: Cobertura:

23 Emma Open-source tool Supports class, method, and line coverage Fractional line coverage supported, but not branch coverage Only part of a line covered Standalone version Eclipse plugin EclEmma also available Installing the plug-in In Eclipse: Help -> install new software -> 23

24 Testcases are written in Junit! Emma coverage report 24

25 Emma source code annotations 25

26 Fractional line coverage Only part of conditional executed Loop increment not executed green for fully covered lines, yellow for partly covered lines and red for lines that have not been executed at all 26

27 EMMA example public class Calc { int tot; Calc(){ tot=0; int somma (int x) { tot=tot + x; return tot; int sottrai (int x) { tot=tot - x; return tot; int moltiplica (int x) { tot=tot * x; return tot; public class CalcTest { Calc public void setup(){ c= new public void testsomma() { assertequals(c.somma(3), public void testsottrai() { assertequals(c.sottrai(0), public void testmoltiplica() { c.somma(2); assertequals(c.moltiplica(3), 6);

28 Output EMMA Coverage = 100%

29 Program Mutation 29

30 What is program mutation? P is alterated in several istances (automatically) For several reasons: Computing test adequacy of a testsuite Improving/completing a testsuite Detecting new errors 30

31 Mutation terms Mutant: a copy of the original program with a small change (seeded fault) Mutation operator: applied to make change (automatically) to the original program Es. + --> * Mutant killed: if its behaviors/outputs differ from those of the original program Live otherwise 31

32 Mutation operator 32

33 Method-level operator (examples) Mutant operator In P In mutant Variable replacement z=x*y+1; x=x*y+1; Relational operator replacement if (x<y) z=x*x+1; if(x>y) if(x<=y) Off-by-1 z=x*y+1; z=x*(y+1)+1; z=(x+1)*y+1; Replacement by 0 z=x*y+1; z=0*y+1; Arithmetic operator replacement z=x*y+1; z=0; z=x*y-1; z=x+y-1; 33

34 Class-level operator (examples) Access modifier change: The AMC operator changes the access level for instance variables and methods to other access levels Hiding variable deletion: The IHD operator deletes a hiding variable, a variable in a subclass that has the same name and type as a variable in the parent class 34

35 Mutant score Mutant killed: if its behaviors/outputs differ from those of the original program testcase 35

36 Test adequacy using mutation Mutant/Mutation score can be also computed for the entire Testsuite MS(T)=number killed mutants/number mutants MS(T)=1 means that all the mutant are killed => the testsuite is 100% adequate Best case... In same cases it is not possible to reach MS(T)=1 Mutants equivalent to P! Ex. x+y --> x+y+0 Automatic tools should avoid it! 36

37 Computing Adequacy Foo Tools Usage Foo Foo Foo MS(FooTest)=? Mutating Foo Tests: FooTest Mutation points = 12, unit test time limit 2.02s.. M FAIL: Foo:31: negated conditional M FAIL: Foo:33: negated conditional M FAIL: Foo:34: - -> + M FAIL: Foo:35: negated conditional... Score: 67% 37

38 Test enhancement using mutation One has the opportunity to enhance a test set T after having assessed its adequacy If the mutation score (MS) is 1, then some other technique, or a different set of mutants, needs to be used to help enhance T If the mutation score (MS) is less than 1, then there exist live mutants. Each live mutant needs to be distinguished from P Adding testcases to the testuite!!!!! 38

39 Error detection using mutation There is no guarantee that tests derived to distinguish live mutants will reveal a yet undiscovered error in P Nevertheless, empirical studies have found to be the most powerful of all enhancement techniques The next simple example illustrates how test enhancement using mutation detects errors 39

40 Foo (1) Consider the following function foo that is required to return the sum of two integers x and y. Clearly foo is incorrect int foo(int x, y) { return (x-y); Now suppose that foo has been tested using a test set T that contains two tests T={ t1: <x=1, y=0>, t2: <x=-1, y=0> This should be return (x+y) Foo(1,0)=1 and Foo(-1,0)=-1 Correct? 40

41 Foo (2) int foo(int x, y) { return (x-y); mutants M1: int foo(int x, y) { M2: int foo(int x, y) { M3: int foo(int x, y) { return (x+y); return (x-0); return (0+y); Testsuite is not adequate! Next we execute each mutant against tests in T MS(T)= 1/3 T={ t1: <x=1, y=0>, t2: <x=-1, y=0> Test (t) foo(t) M1(t) M2(t) M3(t) t t Live Live Killed 41

42 Foo (3) int foo(int x, y) { return (x-y); M1: int foo(int x, y) { return (x+y); Let us examine M1 A test that distinguishes M1 from foo must satisfy the following condition: x-y x+y => -y y => -2y 0 => y 0 Hence we get t3: <x=1, y=1> Executing foo on t3 gives us foo(t3)=0 However, according to the requirements we must get foo(t3)=2 Thus t3 distinguishes M1 from foo and also reveals the error 42

Tools for mutation testing: features A typical tool for mutation testing offers the following features: MuClipse A selectable palette of mutation

43 Tools for mutation testing: features A typical tool for mutation testing offers the following features: MuClipse A selectable palette of mutation operators Management of test set T Ex. Junit Generation of mutants Execution of the program under test and mutants against T Report Mutation score (adequacy) 43

44 Tools Free for mutation testing Proteum is the first mutation testing tool that implements all mutation operators designed for the ANSI C programming language µjava (mujava) is a mutation system for Java programs MuClipse is an Eclipse plugin verion of µjava (mujava) Jumble is a mutation system for Java programs with JUnit tests. Jumble can be used from the command line or as an Eclipse plugin The bytecode is mutated Others at: 44

45 Jumble installation 1. Download Jumble from: 2. Unzip the downloaded file in a folder: jumble_1_1_0 3. Copy or Move the folder jumble (jumble_1_1_0 > eclipseplugin > plugin-export) to your Eclipse plugins folder: Eclipse >plugins, or Eclipse > dropins > eclipse > plugins 4. Restart Eclipse 45

46 Run default mutation testing If : class to be mutated <YourClass> and your test are in the same package and your test s name follows this pattern: <YourClass>Test.java, then you can run Jumble 1. Right click on <YourClass> 2. Choose Jumble > Jumble Class 3. Look at the Console view to see the mutant score 46

47 How Jumble work? class "Foo JUnit tests "FooTest" Jumble starts by running the unit tests (in FooTest.class) on the unmodified Foo class to check that they all pass, and to measure the time taken by each test Then it will mutate Foo in various ways and run the tests again to see if they detect the mutation It continues this process until all mutations of Foo have been tried It provides the output 47

48 Jumble Output Jumble found 12 different mutants of Foo Mutating Foo Tests: FooTest Mutation points = 12, unit test time limit 2.02s.. M FAIL: Foo:31: negated conditional M FAIL: Foo:33: negated conditional M FAIL: Foo:34: - -> + M FAIL: Foo:35: negated conditional... Score: 67% unit tests kill the mutant (indicated by a '.') if (C) decision on line 31 was mutated to if (!C) Overall, 67% (8/12) of the mutations were killed by the unit tests, which means that they probably need to be improved 48

49 Jumble mutations Conditionals: Jumble replaces each condition with its negation the condition x > y would mutate to become!(x > y) Arithmetic Operations: Jumble replace arithmetic operations A fixed table is used + --> -, * --> / Increments: Increments, decrement are mutated for example, i++ becomes i-- Constants: Jumble can change the value of literal constants for example, 0 --> 1 Return Values: Jumble can change return values for example, return X --> return 0 Switch Statements: Jumble can mutate each case of a switch swapping 49

50 Jumble Example public class Calc { int tot; Calc(){ tot=0; int somma (int x) { tot=tot + x; return tot; int sottrai (int x) { tot=tot - x; return tot; int moltiplica (int x) { tot=tot * x; return tot; public class CalcTest { Calc public void setup(){ c= new public void testsomma() { Assert.assertEquals(c.somma(3), public void testsottrai() { Assert.assertEquals(c.sottrai(0), public void testmoltiplica() { c.somma(2); Assert.assertEquals(c.moltiplica(3), 6);

51 Mutant non killed Jumble output Mutating Calc Tests: CalcTest Mutation points = 7, unit test time limit 2.31s...M FAIL: Calc:15: - -> +... Score: 85% public class Calc { int tot; Calc(){ tot=0; int somma (int x) { tot=tot + x; return tot; int sottrai (int x) { tot=tot + x; return tot; int moltiplica (int x) { tot=tot * x; return tot; 51

52 public class CalcTest { Calc public void setup(){ c= new public void testsomma() { Assert.assertEquals(c.somma(3), public void testsottrai() { Assert.assertEquals(c.sottrai(3), -3); Test enhancement Mutating Calc Tests: CalcTest Mutation points = 7, unit test time limit 2.31s... Score: public void testmoltiplica() { c.somma(2); Assert.assertEquals(c.moltiplica(3), 6);

53 Possible exercises at the exam Given a piece of code: Build the CFG Find paths that cover all statements, branches, conditions, or paths Choose input data that will result in the selected paths See exercise Delete rows from array at Enhance a test set T after having assessed its adequacy Given P, T (Junit tests) and a set of Jumble mutations Foo:33: negated conditional Error detection using mutation Ex. Foo 53

54 References (used to prepare these slides) Slides of the book Foundations of software testing by Aditya P. Mathur InstructorSlides.html Slides of Alan Williams, University of Ottawa Glover, Don t be fooled by the Coverage Report, IBM developer works article Slides of Cu Nguyen Duy, Software Analysis and Testing Wikipedia 54

Structural Testing. White Box Testing & Control Flow Analysis

Structural Testing. White Box Testing & Control Flow Analysis Structural Testing White Box Testing & Control Flow Analysis Functional vs. Structural Functional Have I built the right product? Tests derived from the program specification Internal Structure ignored