White-box testing. Software Reliability and Testing - Barbara Russo SERG - Laboratory of Empirical Software Engineering.

Transcription

1 White-box testing Software Reliability and Testing - Barbara Russo SERG - Laboratory of Empirical Software Engineering Barbara Russo 1

2 White-box testing White-box testing is a verification technique that software engineers can use to examine if their code works as specified Barbara Russo 2

3 With WBT Exercise independent paths within a module or unit Exercise logical decisions on both true and false Execute loops at their boundaries and within their operational bounds and Exercise internal data structures to ensure their validity Barbara Russo 3

4 Types of WBT Unit testing Regression testing Integration testing Barbara Russo 4

5 Unit testing Each time you write a code module, you should write test cases for it A possible exception: accessor methods (i.e., getters and setters) Generally, accessor methods will be written error-free Focus testing effort on code that could easily be broken 5

6 Unit testing It focuses on faults within modules 6

7 Test with annotation JUnit4 and older 7

8 Installing JUnit Download from the jar file Read the API latest/ Unzip or import the jar file Add to classpath of your project Project à Properties à Java Build Path Libraries tab Add library 8

9 Installing JUnit Right click on the project folder in package explorer BuildPath à add Libraries tab Or BuildPath àin the library tab select add library 9

10 Installing JUnit 10

11 Example public class Calculator { public int evaluate(string expression) { int sum = 0; for (String summand: expression.split("\\+")) sum += Integer.valueOf(summand); return sum; } } 11

12 Example import static org.junit.assert.assertequals; import org.junit.test; public class CalculatorTest public void evaluatesexpression() { Calculator calculator = new Calculator(); int sum = calculator.evaluate("1+2+3"); assertequals(6, sum); } } 12

13 setup() & teardown() When a test class contains multiple methods to test, you can use the setup() and teardown() methods to initialise and release any common objects under test 13

14 Why annotations The traditional way to indicate test methods in JUnit 3 is by prefixing their names with test This is a very effective method for tagging certain methods in a class as having a special meaning, but the naming doesn t scale very well and is rather inflexible (what if we want to pass additional parameters to the test?). 14

15 Benefits of Using Annotations Annotations were formally added to the Java language in JDK 5 Tagging methods with annotations Hence, method names are not restricted to any pattern or format. We can pass additional parameters to annotations 15

16 Benefits of Using Annotations Annotations are strongly typed, so the compiler will flag any mistakes right away Test classes no longer need to extend anything (such as TestCase, for JUnit 3). 16

17 Basic Describes how to pass parameters to Marks a class or a method as a part of the test. 17

18 @Test It tells that the public method that returns void to which it is attached can be run as a test case JUnit first constructs a new instance of the class then invokes the annotated method Any exceptions thrown by the test will be reported as a failure If no exceptions are thrown, the test succeeds 18

19 Example public class Example public void method() { org.junit.assert.asserttrue(new ArrayList().isEmpty()); } 19

20 Test Fixtures A test fixture is a fixed state of a set of objects used as a baseline for running tests JUnit provides annotations so that test classes can have fixture run before or after every test, or one time fixtures that run before and after only once for all test methods in a class. There are four fixture annotations: two for class-level fixtures and two for method-level ones. At the class level, you and at the method (or test) level, you 20

21 Test The annotated method will be run before any test method in the current class is The annotated method will be run after all the test methods in the current class have run. 21

22 @After Annotating a public method that returns void causes that method to be run before (or after) the Test method methods of superclasses will be run before (or after) those of the current class methods are guaranteed to run even if a Before or Test method throws an exception 22

23 Major annotations - test The annotated method will be run before each test The annotated method will be run after each test method 23

24 Example public class Example { List public void initialize() { mylist= new ArrayList(); public void testsize() { System.out.println{ it uses sizelist }; public void testremove() { System.out.println{ it uses removelist }; } } 24

25 Optional parameters Expected and Timeout Expected: check a test method throws the expected exception If it doesn't throw an exception or if it throws a different exception than the one declared, the test fails 25

26 Example: test public void outofbounds() { new ArrayList<Object>().get(0); } 26

27 ExpectedException rule Careful! with expected exception will pass if any code in the method throws IndexOutOfBoundsException For longer tests, use the ExpectedException rule 27

28 ExpectedException public ExpectedException thrown = public void shouldtestexceptionmessage() throws IndexOutOfBoundsException { List<Object> list = new ArrayList<Object>(); thrown.expect(indexoutofboundsexception.class); thrown.expectmessage("index: 0, Size: 0 ); thrown.expectmessage(junitmatchers.containsstring("size: 0")); // JUnit matcher that finds a string in the exception message } 28

29 Optional parameters Timeout causes a test to fail if it takes longer than a specified amount of clock time (measured in public void infinity() { while(true); } 29

30 Annotating a class When a class is annotated or extends a class annotated JUnit will invoke the class it references to run the tests in that class instead of the runner built into JUnit. 30

31 The test suite class with annotation import org.junit.runner.runwith; import org.junit.runners.suite; BTest.class, CTest.class) public class ABCSuite { } 13/04/16 Barbara Russo 31

32 @SuiteClasses annotation specifies the classes to be executed when a class annotated is run 32

33 Exercise Build test cases for class Bag 33

34 Integration testing 34

35 Integration testing It verifies interface specifications and model breakdown It verifies the software architecture Scaffolding (see later) is extensively required Faults are related to interactions and compatibility 35

36 Integration Faults Inconsistent interpretation of parameters or values Example: Mixed units (meters/yards) in Martian Lander Violations of value domains, capacity, or size limits Example: Buffer overflow 36

37 Integration Faults Side effects on parameters or resources Example: Conflict on (unspecified) temporary file Omitted or misunderstood functionality Example: Inconsistent interpretation of web hits Nonfunctional properties Example: Unanticipated performance issues 37

38 Integration Faults Dynamic mismatches Example: Incompatible polymorphic method calls 38

39 Regression Testing 39

40 Regression testing Premise Adding new/changing module impacts a system: New data flow paths established New I/O may occur New control logic invoked 40

41 Regression testing It is re-execution of subset of tests that have already been run Ensures changes have not propagated unintended side effects 41

42 Regression testing Approaches: Manual testing Capture/Playback tools: 1. Capture test cases and results 2. Playback and 3. Compare 42

43 Regression testing Test suite contains: Representative sample of tests that exercises all software functions Focus on functions likely affected by change Focus on components that have been changed 43

44 Scaffolding 44

45 Why Scaffolding? Test case design includes input and expected output behaviour It may be simply a matter to fill a template (Acceptance test), but 45

46 Why Scaffolding? Testing can be complex: a test specification is connected to several test cases: e.g., a sorted sequence, length greater than 2, with items in ascending order with no duplicates We need a structure to support test execution 46

47 Why Scaffolding? To test in small: independent drivers just test some functionalities of a large user interface To provide hooks for scripting To drive coding (TDD) 47

48 Elements Test driver Test stub Test harness Oracle Barbara Russo 48

49 Test driver To drive program under test through test cases It is a software module used to invoke a module under test and, It often provides test inputs, control and monitor execution, and report test results Drivers can become automated test cases 49

50 Example moveplayer(player1, 2); It calls moveplayer and passes Player1 and moves it two spaces The drivers would likely to be called from main method A white-box test case would execute this driver line of code and check Player.getPosition() to make sure the player is now on the expected cell on the board Barbara Russo 50

51 Test stub A stub is a program statement substituting for the body of a software module that is or will be defined elsewhere or A dummy component or object used to simulate the behaviour of a real component until that component has been developed 51

52 Test stub Developing stubs allows programmers to call a method in the code being developed, even if the method does not yet have the desired behaviour Stubs can be filled in to form the actual method Barbara Russo 52

53 Example If the moveplayer() has not been written yet public void moveplayer(player aplayer, int dicevalue) { player.setposition(1); } or public Cell moveplayer(player aplayer, int dicevalue) { return new Cell(); } or public Cell moveplayer(player aplayer, int dicevalue) { Player player=aplayer; player.setposition(dicevalue); return player.getposition(player); } Barbara Russo 53

54 Test harness Environment in which to execute the tests Substitutes for other parts of the deployed environment Ex: Software simulation of a hardware device 54

55 Oracle A test oracle is a piece of software that provides a pass/fail service to the program execution (e.g., assert) In principle, an oracle classifies every execution and detects every failure, but it can even give false alarms False alarms increase cost of maintenance and reduce resources to dedicate to real failures Thus, there is no perfect oracle 55

56 Using stubs and drivers with TDD Barbara Russo 56

57 What are drivers, stubs, and oracles? public void testcenterline() { Formatter f = new Formatter(); f.setlinewidth(10); assertequals(" word ", f.center("word")); } public void testoddcenterline(){ Formatter f = new Formatter(); f.setlinewidth(10); assertequals( " hello ", f.center("hello")); } public String center(string line) { return a ; }! 57

58 What are drivers, stubs, and oracles? public void testcenterline() { Formatter f = new Formatter(); f.setlinewidth(10); assertequals(" word ", f.center("word")); } public void testoddcenterline(){ Formatter f = new Formatter(); f.setlinewidth(10); assertequals( " hello ", f.center("hello")); } public String center(string line) { return a ; }! 58

59 More on Oracles Software Reliability and Testing - Barbara Russo SERG - Laboratory of Empirical Software Engineering 59

60 Comparison-based oracles To judge oracle correctness: compare the actual with predicted program behaviour Predicted behaviour: either pre-computed as part of test case specifications (when test specifications are simple) or derived in some way independent from the program under test 60

61 Comparison-based oracles Test harnesses often include support for comparison -based oracles: frameworks like JUnit provide methods like assertequals 61

62 Comparison-based oracles Comparison-based oracles can be suitable for more complex programs when supported by other techniques like the capture and replay testing Capture - replay testing saves the predicted behaviour from an earlier execution It useful when we cannot derive the expected behaviour from specification It can be expensive as it requires other implementations of the same feature under test 62

63 Example Save the behaviour (execution) of a trusted alternate version of the program under test For some reason, the current version is not adequate for the product under test Think of OSS development: the open and the enterprise release Replay the open version to test the enterprise one 63

64 Example Save system behaviour (use): used when behaviour cannot avoid the direct intervention of the user Use log of use: they can be reused until the programmer changes that functionality and cannot ensure the same behaviour anymore Sometime useful to model the input in a more abstract way (e.g. instead of mouse click use object selection) 64

65 Other approaches for complex input-output Determine input by knowing the output: It is a misconception that oracles always require a predicted output! 65

66 Example - partial oracles If a program is required to find a bus route from A to B, the oracle needs not to check that a there is a valid route between A and B If a program is required to find the optimal route between A and B, the oracle needs not to check that the valid route is also optimal These oracles are called partial Partial oracles are cheap 66

67 Self-check oracles Self-check: an oracle that is not paired with a specific test case Good design practice: not connected with program states Self-checks can use assertions and can focus on data structure invariants 67

68 Example package org.eclipse.jdt.internal.ui.text; import java.text.characteriterator; import jorg.eclipse.jface.text.assert;... public class SequenceCharacterIterator implements CharacterIteratorf{ //Check the index of iterations; self-check oracle private void invariant(){ Assert.isTrue(fIndex >= ffirst); Assert.isTrue(fIndex <= flast); } it continues 68

69 Example public SequenceCharacterIterator(CharSequence sequence, int first, int last) throws IllegalArgumentException{ if(sequence==null) throw new NullPointerException; if(first<0 first>last) throw new IllegalArgumentException; if(last>sequence.length()) throw new NullPointerException; fsequence = sequence; ffirst = first; flast = last; findex = index; invariant();//check the index of iterations } 69

70 Example... public char setindex(int position){ } if(position>=getbeginindex() && position <= getendindex()) findex=position; else throw new IllegalArgumentException(); invariant(); //Check the index of iterations return current(); } 70