软件测试 05 变异测试 玄跻峰 武汉大学计算机学院. URL:

Transcription

1 软件测试 05 变异测试 玄跻峰 武汉大学计算机学院 URL: 1

2 回顾白盒测试 /* Class under test */ class TextEditor { Menu _menu; ToolBar _toolbar; ExitButton _exitbutton; 语句覆盖分支覆盖 } public TextEditor ( ) { _menu =... } public int intdisplay display ( )( {) {... } public void addoneplugin (Plugin plugin) {... } if (_menu == null && _toolbar == null) throw new NullPointerException; if (_menu!= null && _toolbar!= null && _exitbutton.isclicked() ) return EXIT; } waitforclickingexit ();... 2

3 Java 语言回顾 Class Method Field Variable Type (int, double, String, Class, ) Access (public, protected, default, private) public int foo (double a, double b) { Apple apple = new Apple( ); apple.updateos( 9.0 ); } 3

4 JUnit public void test1() { } 4

5 白盒测试回顾 public class Triangle { private double a, b, c; public Triangle(double ain, double bin, double cin) { a = ain; b = bin; c = cin; if (a <= 0 b <= 0 c <= 0) throw new IllegalArgumentException("Must be greater than zero."); if ((a >= b + c) (b >= a + c) (c >= a + b)) throw new IllegalArgumentException("Not a triangle."); } public String classify() { String result; if ((a == b) && (b == c)) result = "Equilateral"; else if ((a!= b) && (a!= c) && (b!= c)) result = "Scalene"; else result = "Isosceles"; return result; } 5

6 本课程的大作业 6

7 引子 2014 年, 法国本地新闻称巴黎某家庭有 21 个孩子 父母不工作, 靠孩子每月领取的超过 3000 欧元家庭保障金生活 CAF( 家庭保障 局 ) 称制度漏洞 7

8 问题 : 保障金制度多年没有出现问题, 为什么问题隐藏这么久才发现? 8

9 问题 : 保障金制度多年没有出现问题, 为什么问题隐藏这么久才发现? 预料之外的变化, 能够很好的发现隐藏的问题 9

10 变异测试 Mutation Testing 10

11 Introduction Quis custodiet ipsos custodes? Who Watches The Watchmen? In this case: What Tests The Tests? Mutation Testing is a method of inserting faults into programs to test whether the tests pick them up, thereby validating or invalidating the tests

12 History of Mutation Can trace birth of Mutation Testing back to a student paper written in 1971 More interest in the late 70s (DeMillo et al.) Died down due to problems of cost Being researched again recently due to availability of much higher computing power Is most recently being used on nonimperative languages such as Java and XML

13 Goals To assess the quality of the tests by performing them on mutated code To use these assessments to help construct more adequate tests To thereby produce a suite of valid tests which can be used on real programs

14 Testing Method Mutant processes are created to try to mimic typical syntactic errors made by programmers Many differing mutants are run against the specified tests to assess the quality of the tests The tests are attributed a score between 0 and 1, as to whether they can distinguish between the original and the mutants

15 Mutation testing history / aim Pioneered in the 1970s Aim: to locate and expose weaknesses in test suites. 15

16 Mutation testing overview Mutation testing is done by selecting a set of mutation operators and then applying them to the source program one at a time for each applicable piece of the source code. The result of applying one mutation operator to the program is called a mutant. 16

17 Mutation testing overview (2) If the test suite is able to detect the change in the mutant code, then the mutant is said to be killed. When this happens, the mutant is considered dead and no longer needs to remain in the testing process since the faults represented by that mutant have been detected, and more importantly, it has satisfied its requirement of identifying a useful test case. 17

18 Mutation testing overview (3) If a mutant gives the same outcome as the original with the test data, then it is said to be live. The quality of a set of test cases is measured by the percentage of mutants killed by the test data. 18

19 Definition 19

20 Example The original code: if(a && b)c = 1; else c = 0; We replace '&&' with ' ' and produce the following mutant: if(a b)c = 1; else c = 0; Now, for the test to kill this mutant, the following condition should be met: 1) Test input data should cause different program states for the mutant and the original program. For example, a test with a=1 and b=0 would do this (enough for Weak mutation testing). 2) The value of 'c' should be propagated to the program's output and checked by the test (necessary for Strong mutation testing). 20

21 Weak mutation testing Weak mutation is an approximation technique that compares the internal states of the mutant and original program immediately after execution of the mutated portion of the program. It requires much less computing power to ensure that the test suite satisfies weak mutation testing. Mutation Testing 21

22 Strong mutation testing When strong mutation testing is executing, the values produced by mutated code are passing further and changes (or not) the output of the module or subroutine. If the mutant is not killed, then considering whether the test data could be enhanced to kill this live mutant or whether no test data could ever be constructed to kill the mutant. Strong mutation testing requires much more computing power. 22

23 Equivalent mutants The original code: int index=0; while ( ){ index++; if (index == 10) break; } Replace "==" with ">=" and produce the following mutant: int index=0; while ( ){ index++; if (index >= 10) break; } However, it is not possible to find a test case which could kill this mutant. Mutation Testing 23

24 Selective mutation testing Selective mutation testing is based on mutants generation only by selected set of mutation operators. Generated by specific set of operators, mutants are more different and represent more different code mistakes. 24

25 Traditional mutation operators 1. Statement deletion 2. Boolean expression changes (true and false) 3. Arithmetic operation changes (+ and *, - and /) 4. Boolean operator changes (> and >=, == and <=) 5. Declared in the same scope variables changes (variable types should be the same) 25

26 Class-level mutation operators 1. Polymorphic 2. Method Overloading 3. Method Overriding/Hiding in Inheritance 4. Field Variable Hiding in Inheritance 5. Information Hiding (Access Control) 6. Static/Dynamic States of Objects 7. Exception Handling 26

27 1. Polymorphic mutations class T { } interface K { } class S extends T implements K { } class U extends S { } The original code: S s = new S(); Mutants: a)t s = new S(); b)k s = new S(); c)s s = new T(); d)s s = new U(); 27

28 2. Method overloading mutations 1. public LogMessage(int level, String logkey, Object[]inserts) { } 2. public LogMessage(int level, String logkey, Object insert) { } Mutant: public LogMessage(String logkey, int level, Object[]inserts) { } 28

29 3. Method Overriding/Hiding in Inheritance public class Animal { } public void speak(){..} public void bite(){..} public class Cat extends Animal { } Mutant: public void speak() {..} public void bite(){..} public class Cat extends Animal { //public void speak() {..} public void bite(){..} } 29

30 4. Field Variable Hiding in Inheritance public class Animal { public String kind; public int legcount; } public class Cat extends Animal { public String kind; public int legcount; } Mutant: public class Cat extends Animal { public String kind; //public int legcount; } Mutation Testing 30

31 Possible mutants 31

32 5. Information Hiding (Access Control) The original code: protected Address address; Mutants: a)public Address address; b)private Address address; c)address address; //default (no access modifier defined) 32

33 6. Static/Dynamic States of Objects The original code: public static int VALUE = 100; private String s; Mutants: a)public int VALUE = 100; //static is removed. b)private static String s; //static is added. 33

34 7. Exception Handling The original code: String formatmsg(logmessage msg){ try { } catch(missingresourceexception mre){ } } Mutant: String formatmsg(logmessage msg) throws MissingResourceException { //try-catch block removed } 34

35 变异测试 欢迎提问 35