Software Testing - Introduction -

Transcription

1 Software Testing - Introduction - Bugs? What is testing? The RIP model. How important is testing? Test case, test requirement, coverage criteria, coverage level, criteria subsumption. An introduction to JUnit and EclEmma. Testing and the software development process. Testing activities. Testing levels.!! Verificação e Validação de Software! Departamento de Informática! Faculdade de Ciências da Universidade de Lisboa!! Eduardo Marques, edrdo@di.fc.ul.pt

2 Bug as such li/le faults and difficulties are called show themselves, and months of anxious watching, st=dy, and labor are requisite before commercial success or failure is ceraainly reached. [Thomas Edison, 1878] Did You Know? Edison Coined the Term Bug, A. Magoun and P. Israel, The Institute, IEEE ! 2

3 First act=al case of bug being found [!!] Note in Harvard Mark II logbook by Grace Hopper, 1947 [actual moth (bug) part of the logbook], U.S. Naval Historical Center Online Library Photograph 3

4 Other famous bugs Pentium FDIV bug (1994) = Ariane 5 bug (1996) It took the European Space Agency 10 years and $7 billion to produce Ariane 5 [ ]. All it took to explode that rocket less than a minute into its maiden voyage last June, scattering fiery rubble across the mangrove swamps of French Guiana, was a small computer program trying to stuff a 64-bit number into a 16-bit space.! From: A bug and a Crash, James Gleick, ariane.html Apple Maps, 2012 Citius, Ministério da Justiça, Other examples: History s Worst Software Bugs, Wired [link]

5 A simple bug public static int numzero(int[] x) {! // Effects: if x == null throw NullPointerException! // else return the number of occurrences of 0 in x! 1 int count = 0;! 2 for (int i = 1; i < x.length; i++)! 3 if (x[i] == 0)! 4 count++;! 5 return count;! } There is a simple bug in numzero Where is the bug location in the source code? How would you fix it? If the bug is location is reached, how does it corrupt program state? Does it always corrupt program state? If program state is corrupted, does numzero fail? How? 5 The term bug is ambiguous however are we referring to the source code or to outcome of a failed execution? We need clear terminology.

6 Fault, Error, Failure [Falta,Erro, Falha] public static int numzero(int[] x) {! // Effects: if x == null throw NullPointerException! // else return the number of occurrences of 0 in x! 1 int count = 0;! 2 for (int i = 1; i < x.length; i++)! 3 if (x[i] == 0)! 4 count++;! 5 return count;! } Fault: a defect in source code [~ the location of the bug] i = 1 in the code [should be fixed to i = 0] Error: erroneous program state caused by execution of the defect i becomes 1 (array entry 0 is not ever read) Failure: propagation of erroneous state to the program outputs The output value for x = { 0, 1, 0 } is 1 instead of the expected value 2. 6 Failure happens as long as x.length > 0 && x[0] = 0

7 State representation - convention We will represent program states using the notation <var=v1,...,varn=vn, PC=program counter> Example sequence of states in the execution of numzero({0,1,2,0}) 1: < x={0,1,2,0}, PC=[int count=0 (l1)] >! 2: < x={0,1,2,0}, count=0, PC=[i=1 (l2)] >! 3: < x={0,1,2,0}, count=0,i=1,pc=[i<x.length (l2)] >! 4: < x={0,1,2,0}, count=0,i=1,pc=[if(x[i]==0) (l3)] >!...! <x={0,1,2,0}, count=1, PC=[return count;(l5)] > 7

8 Error state - convention We will use the convention: an error state is the first different state in execution in comparison to an execution to the state sequence of what would be the correct program. If we had i=0 the execution of numzero({0,1,2,0}) would begin with states: 1: < x={0,1,2,0}, PC=[int count=0 (l1)] >! 2: < x={0,1,2,0}, count=0, PC=[i=0 (l2)] >! 3: < x={0,1,2,0}, count=0,i=0,pc=[i<x.length (l2)] >...! Instead we have 1: < x={0,1,2,0}, PC=[int count=0 (l1)] >! 2: < x={0,1,2,0}, count=0, PC=[i=1 (l2)] >! 3: < x={0,1,2,0}, count=0,i=1,pc=[i<x.length (l2)] >...! The first error state is 2: < x={0,1,2,0}, count=0, PC=[i=1 (l2)] > 8

9 Basic terminology Recall: Fault: A static defect in the software s source code (defined by one or more source code locations) Error: An incorrect internal state that is the manifestation of some fault [to be further clarified further on]. Failure: External, incorrect behavior with respect to the requirements or other description of the expected behavior. Testing :evaluating software by observing its execution. Test failure : a test execution that results in a failure. Debugging: the process of finding a fault given a failure. 9

10 Test failure - RIP model Reachability The fault is reached. Infection Execution of the fault leads to an error. Propagation Errors are propagated to the program output (failure) If the RIP conditions are met, we say that there is a test failure. 10

11 numzero: execution w/error but no failure reachability + infection but no propagation public static int numzero(int[] x) {! // Effects: if x == null throw NullPointerException! // else return the number of occurrences of 0 in x! 1 int count = 0;! 2 for (int i = 1; i < x.length; i++)! 3 if (x[i] == 0)! 4 count++;! 5 return count;! } Considering an execution where x = { 1, 0, 2, 0} Error: <x={1,0,2,0}, i=1, count=0, PC= if...,l3> deviates from expected internal state <x={1,0,2,0}, i=0,count=1,pc =[if..., l3]>! No Failure! numzero({1,0,2,0}) will return 2 as expected. 11

12 numzero: execution w/error and failure reachability + infection + propagation public static int numzero(int[] x) {! // Effects: if x == null throw NullPointerException! // else return the number of occurrences of 0 in x! 1 int count = 0;! 2 for (int i = 1; i < x.length; i++)! 3 if (x[i] == 0)! 4 count++;! 5 return count;! } Considering an execution where x = { 0, 1, 2, 0} Error: <x={0,1,2,0}, i=1, count=0, PC= if...,l3> deviates from expected internal state <x={1,0,2,0}, i=0,count=1,pc =[if..., l3]>! And failure: numzero({0,1,2,0}) will return 1 rather than 2. 12

13 13 How important is testing?

14 How mature is your testing?! Beizer s scale for test process maturity [ A & O, page 8] Level 0: There s no difference between testing and debugging. Level 1: The purpose of testing is to show that the software works. Level 2: The purpose of testing is to show that the software doesn t work. Level 3: The purpose of testing is not to prove anything specific, but to reduce the risk of using the software. Level 4: Testing is a mental discipline that helps all IT professionals develop higher quality software. 14

15 Debug only? Level 0: There s no d i ff e re n c e b e t w e e n testing and debugging. We should try to prevent bugs early on rather than wait for them to come and haunt us ( m a n y t i m e s w i t h d i s a s t r o u s consequences). 15

16 Developer versus tester? 16 Level 1: The purpose of testing is to show that the software works (developer-biased view ). But generally, correctness is impossible to establish or demonstrate Level 2: The purpose of testing is to show that the software doesn t work ( testerbiased view ). A negative view, still it is in line with the basic aim of software testing Both views fail to answer the question: do we have good software or bad tests?

17 Mature testing Correctness cannot generally be achieved or demonstrated through testing. Testing cannot prove the absence of bugs, but merely expose them. Developers and testers should be on the same boat though. Level 3: The purpose of testing is not to prove anything specific, but to reduce the risk of using the software. A general, overall understanding is required of how to devise tests, and derive confidence/risk levels from their execution. Level 4: Testing is a mental discipline that helps all IT professionals develop higher quality software. The grand philosophical view : testing should be an integral part of the development process and play an important role towards the general the aim of improving software quality. 17

18 More terminology - test case Test case: A test case is composed of the test case values, expected results, prefix values, and postfix values necessary for a complete execution and evaluation of the software under test. Test case values: the input values necessary to complete some execution of the software under test. Expected values: the expected result that will be produced by the test case if and only if the program satisfies its intended behavior. Prefix values/actions: any inputs/commands necessary to put the software into the appropriate state to receive the test case values. Postfix values/actions : any inputs/commands that need to be sent to the software after the test case values are sent. 18

19 Test case execution setup/execute prefix values/actions 1 setup test case values (inputs) 2 execute software under test (SUT) 3 compare test outputs vs expected values 4 setup/execute postfix values/actions 5 We ll focus today only on testing that requires steps

20 Example test cases for numzero public static int numzero(int[] x) {! // Effects: if x == null throw NullPointerException! // else return the number of occurrences of 0 in x! 1 int count = 0;! 2 for (int i = 1; i < x.length; i++)! 3 if (x[i] == 0)! 4 count++;! 5 return count;! } test case test case values (x) expected values actual execution failure? 1 null NullPointerException NullPointerException No 2 { } 0 0 No 3 { 1,2,3 } 0 0 No 4 {1,0,1,0} 2 2 No 20 5 {0,1,2,0} 2 1 Yes

21 More terminology Test set: a set of test cases. We will use notation T for a test set. Test requirement : requirement that should be satisfied by a test set. Test requirements normally come in sets. We use notation TR for the set of test requirements. Coverage criterion: A coverage criterion C is a rule or collection of rules that define a set of test requirements TR(C) to be satisfied by a test set. Coverage level: the percentage of test requirements that are satisfied by a test set. We say T satisfies C if the coverage level of TR(C) by T is 100 %. Infeasible requirement: requirement that cannot be satisfied by any test case. If there are infeasible test requirements, the coverage level will never be 100%. 21

22 Basic coverage criteria public static int numzero(int[] x) {! // Effects: if x == null throw NullPointerException! // else return the number of occurrences of 0 in x! 1 int count = 0; /* I1 */! 2 for (int i = 1 /* I2 */; i < x.length /* I3,B1 */; i++ /* I4 */)! 3 if (x[i] == 0) /* I5,B2 */! 4 count++; /* I6 */! 5 return count; /* I7 */! } Line coverage (LC): cover every line in the SUT. TR(LC) = { line 1, line 2, line 3, line 4, line 5 } Instruction coverage (IC): cover every instruction in the SUT. TR(IC) = { I1, I2, I3, I4, I5, I6, I7 } Branch coverage (BC): cover every instruction, and including all cases at choice points (if, switch-case, etc). 22 TR(BC) = { NPE-B1, B1,!B1, B2,!B2 }

23 LC, IC, BC for numzero public static int numzero(int[] x) {! // Effects: if x == null throw NullPointerException! // else return the number of occurrences of 0 in x! 1 int count = 0; /* I1 */! 2 for (int i = 1 /* I2 */; i < x.length /* I3,B1 */; i++ /* I4 */)! 3 if (x[i] == 0) /* I5,B2 */! 4 count++; /* I6 */! 5 return count; /* I7 */! } test case test case values (x) expected values exec. result test fails? LC IC BC t1 null NPE NPE no 1 2 i1 i2 i3 NPE-B1 t2 { } 0 0 no i1 i2 i3 i7!b1 t3 {1,2} 0 0 no All except i6 B1,!B1,!B2 t4 {0,0 } 2 1 yes All All B1,!B1, B2 23 t5 {1,1,0} 1 0 no All All B1,!B1, B2,!B2

24 LC, IC, BC for numzero T (test set) LC level IC level BC level { t1 } 40 % (2/5) 42 % (3/7) 20 % (1/5) { t1, t2 } 60 % (3/5) 57 % (4/7) 40 % (2/5) { t2, t3 } 80 % (4/5) 85 % (6/7) 60 % (3/5) { t4 } 100 % (5/5) 100 % (7/7) 60 % (3/5) { t1, t5} 100 % (5/5) 100 % (7/7) 100 % (5/5) test case test case values (x) expected values exec. result test fails? LC IC BC t1 null NPE NPE no 1 2 i1 i2 i3 NPE-B1 t2 { } 0 0 no i1 i2 i3 i7!b1 t3 {1,2} 0 0 no All except i6 B1,!B1,!B2 t4 {0,0} 2 1 yes All All B1,!B1, B2 24 t5 {1,1,0} 1 0 no All All B1,!B1, B2,!B2

25 LC, IC, BC for numzero T (test set) LC level IC level BC level { t1 } 100 % 40 coverage % (2/5) for all criteria 42 % (3/7) 20 % (1/5) { t1, t2 } but bug is not exposed!!! 60 % (3/5) 57 % (4/7) t1 and t5 do not fail 40 % (2/5) { t2, t3 } 80 % (4/5) 85 % (6/7) 60 % (3/5) { t4 } 100 % (5/5) 100 % (7/7) 60 % (3/5) { t1, t5} 100 % (5/5) 100 % (7/7) 100 % (7/7) test case test case values (x) expected values exec. result test fails? LC IC BC t1 null NPE NPE no 1 2 i1 i2 i3 NPE-B1 t2 { } 0 0 no i1 i2 i3 i7!b1 t3 {1,2} 0 0 no All except i6 B1,!B1,!B2 t4 {0,0} 2 1 yes All All B1,!B1, B2 25 t5 {1,1,0} 1 0 no All All B1,!B1, B2,!B2

26 Criteria subsumption Criteria Subsumption: A coverage criterion C1 subsumes C2 if and only if every test set that satisfies criterion C1 also satisfies C2. For instance: instruction coverage subsumes line coverage branch coverage subsumes instruction coverage The inverse is not true. In the previous slide: If count++ appeared in the same line as if (x[i] == 0), test case t3 would cover all lines but not all instructions (instruction i6 is not be executed by t3). Test t4 covers all instructions, but not all branches. 26

27 Introduction to JUnit JUnit is a testing framework for Java (the most widely used one). Eclipse SDK includes JUnit and a GUI interface for it. Today we ll look at: JUnit test classes Anatomy of basic JUnit test methods JUnit assertions and other misc. features Associated patterns for test programming 27

28 JUnit - test classes 28 import static org.junit.assert.*;! import static vvs.arrayoperations.numzero;! import org.junit.test;!! public class ArrayOperationsNumZeroTest public void testnumzeroemptyarray() {!!! int x[] = {}; // zero-sized array!!! int n = numzero(x);!!! assertequals("0 zeros", 0, n);!! public void testnumzeroarraywithnozeros() {!!! int[] x = { 1,2, 3 };!!! int n = numzero(x);! imports test class test method annotation - one per test case - another test method/case!! assertequals("0 zeros in an array with no zeros", 0, n);!! }!...

29 JUnit/xUnit - conventions Testclass class pattern [G.Meszaros page 373] Group related test methods in a single test class. The name of test packages/classes/methods should at least transmit: The name of the SUT class (e.g., TestArrayOperationsNumZero) The name of the method or feature being tested (e.g., TestArrayOperationsNumZero) The relevant characteristics of the test case values per each test method (e.g., testnumzeroemptyarray) See G. Meszaros, xunit Testing patterns, page 158 It is also common to prefix or suffix test classes with Test and prefix test methods with test. 29

30 JUnit - test public void testnumzeroarraywithnozeros() {!! int[] x = { 1,2, 3 };! 1) setup test case values!!! int n = numzero(x);! 2) execute SUT!!! assertequals("0 zeros in an array with no zeros", 0, n);! }!... Test design atterns 3) assert expected vs. test outputs Setup + execute SUT + verify expected results (+ teardown) expected Use assertion methods provided by JUnit to verify expected results exec. output Use assertion messages together with assertion methods to give an indication of what went worn 30 Four-phase test, Assertion Method, and Assertion Message patterns [G. Meszaros, pages ]

31 JUnit assertions Method assertequals(msg,expected,value)! assertnotequals(msg,expected,value) asserttrue(msg, value)! assertfalse(msg, value) assertnull(msg, expression)! assertnotnull(msg, expression) assertarrayequals(msg, vexp, vval) assertsame(msg, expected, value) Checked condition value.equals(expected)!!value.equals(expected) value == true! value == false value == null! value!= null Arrays vexp and vval have the same contents. value == expected! (exactly the same object reference)! Full list at 31

32 Other JUnit features: exceptions as expected (expected = NullPointerException.class)!! public void testnumzerowithnullargument() {!!! int[] x = null;! numzero(x);!! } equivalent public void testnumzerowithnullargument1() {! int[] x = null;! try {! numzero(x);! fail("expected NullPointerException");! } catch (NullPointerException e) { }! } 32 Note: the 2nd pattern is more verbose and unnecessary in this case. It is useful in situations when we wish to perform other assertions beyond the expected exception behavior.

33 Other JUnit features: prefix/postfix actions public static void globalsetup() {! // prefix actions executed once before any test!...! public static void globalteardown() {! // prefix actions executed once after all tests!...! public void pertestsetup() {! // prefix actions executed before each test!...! public static void pertestteardown() {! // actions executed after each test!...! }

34 Test engineering - terminology Validation [A & O, p.11] The process of evaluating software at the end of the software development to ensure compliance with intended usage. not done by developers, but by experts in the intended usage of the software Verification [A & O, p.11] The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase. requires technical background on the software, normally done by developers at the various stages of development 34

35 Test engineering - activities Test Manager } design Test Designs instantiate Executable Tests [A & O, p. 4]! Test Engineer Test Engineer Test Engineer P Computer execute Output Evaluate 35 different activities, technical backgrounds, levels of automation...

36 Types of Test Activities Summary 1a. Design Design test values to satisfy engineering goals Criteria Requires knowledge of discrete math, programming and testing 1b. Design Design test values from domain knowledge and intuition Human Requires knowledge of domain, UI, testing 2. Automation Embed test values into executable scripts Requires knowledge of scripting 3. Execution Run tests on the software and record the results Requires very little knowledge 4. Evaluation Evaluate results of testing, report to developers Requires domain knowledge These four general test activities are quite different It is a poor use of resources to use people inappropriately Introduction to Software Testing (Ch 1) Ammann & Offutt

37 Test engineering - overall goals Validation [A & O, p.11] The process of evaluating software at the end of the software development to ensure compliance with intended usage. not done by developers, but by experts in the intended usage of the software Verification [A & O, p.11] The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase. requires technical background on the software, normally done by developers at the various stages of development 37

38 Testing and the SW development lifecycle Requirements Analysis Test Acceptance Test Validation Architectural Design Design Information System Test Subsystem Design Integration Test Detailed Design Module Test Verification Implementation Unit Test 38 [A & O, p 6]

39 Old : Testing at Different Levels main Class P Acceptance testing: Is the software acceptable to the user? Class A Class B System testing: Test the overall functionality of the system method ma1() method mb1() method ma2() method mb2() This view obscures underlying similarities Integration testing: Test how modules interact with each other Module testing: Test each class, file, module or component Unit testing: Test each unit (method) individually Introduction to Software Testing (Ch 1) Ammann & Offutt 53

40 Exercise 1 [Adapted from Exercise 3, page 16 from A & O, code in provided Eclipse project] F o r e a c h o f t h e f o u r m e t h o d s findlast( ), countpositive(), lastzero(), and oddorpos() (a) Identify the fault and what would be the correct code. (b) If possible, identify a test case that does not execute the fault (reachability condition not met). (c) Identify a test case that leads to failure (reachability, infection and propagation conditions all met). Identify the first error state. (d) If possible identify a test case that results in an error, but not a failure (propagation condition not met). Identify the first error state. (e) If possible, identify a test case that executes the fault but does not lead to an error state (infection condition not met). 40

41 Exercise 2 Write a JUnit test class for each method in exercise 1: (a) Start by writing tests that satisfy Line Coverage (b) Enhance the tests (if necessary) to satisfy Instruction Coverage (c) Enhance the tests (if necessary) to satisfy Branch Coverage Assess the coverage of your tests using EclEmma See Eclipse update site: 41