What is software testing? Software testing is designing, executing and evaluating test cases in order to detect faults.

Transcription

1 ϖοιδ τεσταδδανδχουντ() { ασσερ τεθυαλσ(1, ο.αδδανδχουντ(νεω ΑρραψΛιστ()); ϖοιδ τεσταδδανδχουντ() { ασσερ τεθυαλσ(1, ο.αδδανδχουντ(νεω ΑρραψΛιστ()); ιντ αδδανδχουντ(λιστ λιστ) { ρετυρν λιστ.σιζε(); ιντ αδδανδχουντ(λιστ λιστ) { ρετυρν λιστ.σιζε(); What is software testing? Code Coverage & Mutation Testing Static code analysis, lecture #8 Software testing is designing, executing and evaluating test cases in order to detect faults. Requirements, Design, Code... Test models (frameworks) classical approach to software testing bartosz.bogacki@cs.put.poznan.pl Application modelling Evaluated knowledge Test model based on application Results analysis Tests designing Results Testing strategy System testing Tests implementation Test cases How to assure high quality of test cases? Code coverage analysis Quality of test cases is low if tests don t execute relevant parts of code Test cases: Program code:? Code coverage (test coverage): describes the degree to which the source code of a program has been tested Code coverage analysis is the process of: Finding areas of a program not exercised by test cases, Creating additional test cases to increase coverage, and Determining a quantitative measure of code coverage, which is an indirect measure of tests quality. Test coverage types Main test coverage types Statement Coverage Basic Block Coverage Decision Coverage Condition Coverage Path Coverage Function/Method Coverage Call Coverage Data Flow Coverage...many, many more Coverage methods important for practitioners are: Statement coverage Basic block coverage Decision coverage Condition coverage Path coverage Method coverage Class coverage 1

2 Statement coverage Statement coverage Reports whether each executable statement was encountered Other known names line coverage, segment coverage Example code and test case int addandcount(list list) { list.add( Sample text ); return list.size(); void testaddandcount() { assertequals(1, o.addandcount(new ArrayList()); void testaddandcount() { assertequals(1, o.addandcount(new ArrayList()); int addandcount(list list) { list.add( Sample text ); return list.size(); Coverage = 100% int addandcount(list list) { list.add( Sample text ); return list.size(); What if list was null? 100% of statement coverage doesn t guarantee, that the code was completely tested. Basic block coverage Basic block coverage The same as statement coverage except the unit of code measured is each sequence of non-branching statements A non-branching statement is one that does not affect chain of execution Branching statements include all looping statements, conditional statement, break, continue, and return Example code and coverage values void addconst(list list) { list.add( Sample text 1 ); list.add( Sample text 2 ); list.add( Sample text 2 ); else { System.out.println( Empty ); *assumption: if/else were counted as 2 statements assertequals(1, o.addconst(new ArrayList())); assertequals(1, o.addconst(null)); single unit of code Statement coverage Basic block coverage 83% 75% 50% 75% Decision coverage Decision coverage Reports whether boolean expressions tested in control structures evaluated to both true and false The entire boolean expression is considered one true-or-false predicate regardless of whether it contains logical-and (&&) or logical-or ( ) operators Other known names: branch coverage, all-edges coverage, C2 Example code and coverage values void addconst(list list) { list.add( Sample text 1 ); list.add( Sample text 2 ); list.add( Sample text 2 ); else { System.out.println( Empty ); assertequals(1, o.addconst(new ArrayList())); assertequals(1, o.addconst(null)); Decision coverage 50% 50% 2

3 Decision coverage Condition coverage Short-circuit operators issue if (condition1 && (condition2 o2.method())) { System.out.println( Test case 1 ); else { System.out.println( Test case 2 ); Test case 1: condition1=true; condition2=true if statement value = true Test case 2: condition1=false; if statement value = false Coverage = 100%, but o2.method() was never executed! Similar to decision coverage but has better sensitivity to the control flow Reports the true or false outcome of each boolean sub-expression, separated by logical-and and logical-or if they occur Path coverage Path coverage Reports whether each of the possible paths in each method have been followed A path is a unique sequence of branches within the method A large number of variations of this measure exist to cope with loops Boundary-interior path testing For loops: zero repetitions and more than zero repetitions For do-while loop: one iteration and more than one iteration The number of paths is exponential to the number of branches a method containing 10 if-statements has 1024 paths to test adding one more if-statement doubles the count to 2048 Sometimes 100% of coverage is not feasible, example: System.out.println ( Value + list.get(1)); logger.debug ( Control point 1 ); list.add( Bartek ); Method coverage Class coverage This measure reports whether each method was invoked It is useful during preliminary testing to assure at least some coverage in all areas of the software Looking out for uncovered methods is a good technique for detecting either dead code or code than needs more test attention This measure reports whether each class was accessed A class is considered to have been covered if it has been loaded and initialized by the JVM Similar to the method coverage, useful in preliminary testing to assure at least some coverage in all areas of the software 3

4 ϖοιδ τεσταδδανδχουντ() { ασσερτεθυαλσ(1, ο.αδδανδχουντ(νεω ΑρραψΛιστ()); ϖοιδ τεσταδδανδχουντ() { ασσερτεθυαλσ(1, ο.αδδανδχουντ(νεω ΑρραψΛιστ()); ιντ αδδανδχουντ(λιστ λιστ) { ρετυρν λιστ.σιζ ε() ; ιντ αδδανδχουντ(λιστ λιστ) { ρετυρν λιστ.σιζ ε() ; Popular coverage analysis tools Emma Coverlipse Emma Clover JCoverage Runs as a separate java program Generates hierarchical html reports Uses Class coverage Method coverage Line coverage Basic block coverage License: CPL (Common Public License) Reference: Emma Is a code coverage sufficient? Sample screenshots Imagine the following public class MyClazz { List addperson(list list, String person) { /* list.add(person); */ logger.info ( Person: + person + was added ); return list; but we test nothing here! [the test has no assertions] void testaddperson() { MyClazz obj = new MyClazz(); obj.addperson(new ArrayList(), Bartek ); obj.addperson(null, null); Statement/basic block coverage = 100% Decision/condition coverage = 100% Class/method coverage = 100% Is a code coverage sufficient? How to assure high quality of test cases? The test doesn t check if the person was really added to the list The valid test case should be like: Quality of test cases is low if tests don t execute relevant parts of code (coverage is low) Test cases: Program code: void testaddperson() { MyClazz obj = new MyClazz(); List list = obj.addperson(new ArrayList(), Bartek ) assertequals ( Bartek, (String)list.get(1)); list = obj.addperson(null, null); assertnull (list);...but high code coverage, doesn t implicate high quality of tests. All responses from the tested code need to be checked and validated.? 4

5 Mutation Testing Traditional approach Mutation testing Mutant modified source code Equivalent mutant modified source code that behaves as an original Killed mutant mutant that made at least one test case to fail Live mutant - mutant that didn t make any test case to fail Score - the percentage of changes that made the tests to fail number_of_killed_mutants score = number_of_killed_mutants+number_of_live_mutants Mutation testing Mutation testing Consider the example: public int bar(int a) Mutate the code: throws IllegalArgumentException if { ((a < 5) (a < 1)) { if ((a > 5) (a < 1)) { throw new IllegalArgumentException(); int c = a; for (int i = 0; i < a; i++) { c *= 10; return c; public void testbar1 () { assertequals (3000, new Foo().bar(3)); Mutant killed! exception thrown instead of value 3000 public void testbar2 () { Mutant killed! try { exception not thrown new Foo().bar(6); fail ("Exception not thrown"); catch (IllegalArgumentException e) { Consider the example: public int bar(int a) throws IllegalArgumentException { if ((a > 5) (a < 1)) { throw new IllegalArgumentException(); int c = a; for (int i = 0; i < a; i++) { c *= 10; return c; public void testbar1 () { assertequals (3000, new Foo().bar(3)); public void testbar2 () { try { new Foo().bar(6); fail ("Exception not thrown"); catch (IllegalArgumentException e) { Mutate the code: for (int i = 0; i > a; i++) Mutant killed! value 3 returned instead of value 3000 Mutation testing Mutation testing Consider the example: public int bar(int a) throws IllegalArgumentException { if ((a > 5) (a < 1)) { throw new IllegalArgumentException(); int c = a; for (int i = 0; i < a; i++) { c *= 10; return c; public void testbar1 () { assertequals (3000, new Foo().bar(3)); public void testbar2 () { try { new Foo().bar(6); fail ("Exception not thrown"); catch (IllegalArgumentException e) { Mutate the code: c += 10 Mutant killed! value 33 returned instead of value 3000 A test case that kills a mutant must satisfy three conditions: The reachability condition is that the mutated statement must be reached. The necessity condition is that once the mutated statement is executed, the test case must cause the mutant program to behave erroneously. The sufficiency condition states that the incorrect state must propagate through the program's computation to result in a failure. 5

6 Mutation types Mutations at lexical level Mutation can be introduced at different level of program processing Static mutation: lexical analysis syntax analysis bytecode analysis Dynamic mutation: modified run-time All mutations are made in scanner (at lexical analysis stage) After some change was made the code is compiled, test cases are executed, if all tests passed - live mutant was found (proper message saying what was changed should be displayed) Examples of lexical level mutations Lexical level mutations Example 1: List addperson(list list, String person) { /* list.add(person); */ logger.info ( Person: + person + was added ); return list; Example 2: if ((true) (list!= null)) public static boolean issamelength(long[] a1, long[] a2) { if ((a1 == null && a2!= null && a2.length > 0) (a2 == null && a1!= null && a1.length > 0) (a1!= null && a2!= null && a1.length!= a2.length)) { return false; return true; return true; Sample set of mutations: numeric literals (ie. 0 -> 1) true -> false false -> true if( -> if(true if( -> if(false && == ->!=!= -> == ++ -> > ++ Problems with lexical level mutations Problems with lexical level mutations...but what if the source is like: public static boolean toboolean(int value, int truevalue, int falsevalue) { if (value == truevalue) { return true; else if (value == falsevalue) { return false; // no match throw new IllegalArgumentException("The Integer did not match either specified value"); false -> true public static boolean toboolean(int value, int truevalue, int truevalue) { if (value == truevalue) { return true; else if (value == truevalue) { return true; // no match throw new IllegalArgumentException("The Integer did not match either specified value");...or is like: String valuestring (int v1, int v2) { Integer i = new Integer (v1 + v2); String s = "value is: " + i + "."; return s; + -> - String valuestring (int v1, int v2) { Integer i = new Integer (v1 + v2); String s = "value is: " - i + "."; return s; 6

7 Mutations at lexical level Mutations at lexical level Advantages Easy to design and develop Faster than syntax-based approach Easy to use Disadvantages Slow (modify/compile/execute cycle) Mutations are not syntax preserving Small set of applicable mutations Jester the JUnit test tester Syntax level mutations An example of lexical-level mutation system is Jester ( Mutations are made in parser (at syntax analysis stage or during AST analysis) After some change was made the code is compiled, test cases are executed, if all tests passed - live mutant was found (proper message saying what was changed should be displayed) Syntax level mutations Examples of method-level mutations Method-level mutations for Java: Arithmetic Operator Replacement Arithmetic Operator Insertion Arithmetic Operator Deletion Relational Operator Replacement Conditional Operator Replacement Conditional Operator Insertion Conditional Operator Deletion Shift Operator Replacement Logical Operator Replacement Logical Operator Insertion Logical Operator Deletion Assignment Operator Replacement Arithmetic Operator Insertion a -> ++a public int bar(int a) throws IllegalArgumentException { if ((a > 5) (a < 1)) { throw new IllegalArgumentException(); Conditional Operator Replacement (a > 5) -> int!(a > c 5) = a; for (int i = 0; i < a; i++) { c *= -10; return c + a; Assignment Operator Replacement *= -> += Arithmetic Operator Deletion -10 -> 10 Conditional Operator Replacement -> && Relational Operator Replacement a < 1 -> a >= 1 Arithmetic Operator Replacement c + a -> c - a Arithmetic Operator Replacement i++ -> i-- 7

8 Syntax level mutations Syntax level mutations Class-level mutations for Java: Encapsulation Access modifier change Hiding variable deletion Hiding variable insertion Overriding method deletion Overriding method calling position change Inheritance Overriding method rename super keyword insertion super keyword deletion Explicit call to a parent's constructor deletion new method call with child class type Member variable declaration with parent class type Parameter variable declaration with child class type Type cast operator insertion Class-level mutations for Java: Polymorphism Cast type change Type cast operator deletion Reference assignment with other comparable variable Overloading method contents replace Overloading method deletion Arguments of overloading method call change this keyword insertion this keyword deletion static modifer insertion static modifer deletion Syntax level mutations Examples of class-level mutations Class-level mutations for Java: Java-Specific Features Member variable initialization deletion Java-supported default constructor creation Reference assignment and content assignment replacement Reference comparison and content comparison replacement Acessor method change Modifer method change Explicit call of a parent's constructor deletion public Stack (int a) { Access modifier change // super (a); private int size = 10; class MyStack extends SimpleStack { int size = 10; Overriding method deletion public MyStack (int a) //{ void push (int a) { super (a); // ++size; // super.setsize(size); // super.push(a); // void push (int a) { ++size; super.setsize(size); super.push (a); super keyword deletion... setsize(size); super.push(a);... Overridden method calling position change... super.push(a); super.setsize(size);... Examples of class-level mutations Mutations at syntax level new method call with child class type class Client extends stack Dummy = new { MyStack(10); BaseStack stack; public Client () { stack = new BaseStack(10); Argument of overloading method change valuestring (20, 1); Object a_1_20() { valuestring (1, 20); Advantages More precise than lexical-based mutators Greater number of applicable mutants Object valuestring (int v1, int v2) { Integer i = new Integer (v1); Integer j = new Integer (v2); String s = "value: " + i + " " + j; return s; Reference assignment with other compatible type return i; 8

9 Mutations at syntax level Lava Disadvantages Slow (modify/compile/execute cycle) More complicated to develop and maintain than lexical-based mutators An example of syntax-level mutation system is Lava ( Mutations at bytecode level Mutations at bytecode level Mutations are made on intermediate representation (after compilation) After some change was made test cases are executed, if all tests passed - live mutant was found (proper message saying what was changed should be displayed) Developed with bytecode translation tools like: BCEL, BCA, Javassist, JOIE, JMangler, etc. Applicable mutations: Method-level mutations Class-level mutations Bytecode level mutations Bytecode level mutations Arithmetic Operator Replacement new method call with child class type public static int sum(int a, int b) { // return a + b; iload_0 iload_1 iadd ireturn public static int sum(int a, int b) { // return a - b; iload_0 iload_1 isub ireturn ArrayList list; class MyList extends ArrayList {; void assignlist() { // list = new ArrayList (); aload_0 new #18 <Class ArrayList> dup invokespecial #19 <Method void ArrayList()> putfield #21 <Field ArrayList list> return ArrayList list; void assignlist() {// list = new MyList (); aload_0 new #44 <Class MyList> dup invokespecial #45 <Method void MyList()> putfield #21 <Field ArrayList list> return 9

10 Mutations at syntax level Mutations at syntax level Advantages More precise than lexical-based mutators Greater number of applicable mutants Faster than lexical- and syntax-based methods (due to elimination of compilation time) Disadvantages More complicated to develop and maintain than lexical-, or syntax-based mutators mujava (µjava) Research directions An example of bytecode-level mutation system is mujava ( Selective mutations Schema-based mutation analysis Aspect-based mutations (response injection) Summary Further reading Good tests are planned and have sufficient code coverage Basic code coverage methods have lot of weaknesses, aggregated model is preferred Full code coverage doesn t guarantee, that project have good tests Lack of code coverage guarantees that the project has poor tests Mutation testing can be used to test test cases Mothra: Jester: mujava: 10

11 The end Thank you! 11