SE 3S03. Zahra Ali. Week of Feb 22, 2016

Transcription

1 SE 3S03 Department of Computing and Software McMaster University Week of Feb 22, 2016 Coverage Coverage 1/34

2 Outline Coverage Coverage Coverage Coverage 2/34

3 Recall that means using the information about the internal structure of the software to determine test cases. It may also be called unit structural testing Coverage Coverage 3/34

4 Coverage a test set T such that, by executing P for each d in T, each elementary statement of P is executed at least once. P behaves as a function d is an element in the domain D of P elementary statements are usually assignment statements, I/O statements and procedure calls Generally means execute every line of source code at least once Can you think of any weakness in just fulfilling this criterion? Coverage Coverage 4/34

5 Coverage a test set T such that, by executing P for each d in T, each elementary statement of P is executed at least once. P behaves as a function d is an element in the domain D of P elementary statements are usually assignment statements, I/O statements and procedure calls Generally means execute every line of source code at least once Can you think of any weakness in just fulfilling this criterion? It fails to measure if statements with a false decision outcome. Coverage Coverage 4/34

6 Coverage Simple example: if (x < 6) { x += 1; } How many test cases do we need to make all lines of code execute? Coverage Coverage 5/34

7 Coverage Simple example: if (x < 6) { x += 1; } How many test cases do we need to make all lines of code execute? Only 1 case, for example: <x = 4> Coverage Coverage 5/34

8 Coverage Simple example: if (x < 6) { x += 1; } How many test cases do we need to make all lines of code execute? Only 1 case, for example: <x = 4> Note that the case where x 6 does not get checked using this criterion. Coverage Coverage 5/34

9 Coverage Consider the following change to the example: while (x < 6) { x += 1; } Using the same value of x = 4, does this code have the false condition left unchecked? Coverage Coverage 6/34

10 Coverage Consider the following change to the example: while (x < 6) { x += 1; } Using the same value of x = 4, does this code have the false condition left unchecked? No! The false condition executing is the only way the while loop will terminate. Coverage Coverage 6/34

11 Coverage Example: If we look at the function, we know that the highlighted conditional statements determine whether or not the body code is executed. Coverage Coverage 7/34

12 Coverage Our goal is to ensure every line of code is executed, so we simply have to create test cases that satisfy the different conditions. However, remember we want to try and keep our test set minimal for this testing technique. So, wherever we see nested conditional statements, we just need to have one test that satisfies all to meet the criterion. Coverage Coverage 8/34

13 Coverage So we begin by defining a test with inputs inputvalid = false && count = 0 For the exception to run, we need the inputvalid = false (to enter the while loop) and count 4 so let s pick inputvalid = false && count = 4 So to complete statement coverage testing on the above, we only need 2 test cases. Coverage Coverage 9/34

14 Coverage Continuing on to the process() function, we follow the same procedure: A test to enter the conditional loop and test the body code needs linenum RecordReader.MAX LINE NUM and endreached = false so let s pick linenum = 8, RecordReader.MAX LINE NUM = 50 and endreached = false Coverage Coverage 10/34

15 Coverage To satisfy these three for loops and the if statement, linenum > 0, fieldchars.length > 0, chars.length > 1, and have at least one occcurence of fieldchars[j] == chars[k]. Keeping in mind we want to keep test cases minimal, let s pick linenum = 8, fieldchars = { A, D, W }, chars = { A, D, W }, Coverage Coverage 11/34

16 Coverage With our previous test using linenum = 8, we are sure to enter the while loop and the for loop to print the first half of the line. In addition, the same test will allow us to enter the second half of the line when i = 1 and execute the for block since chars.length would equal 3. Coverage Coverage 12/34

17 Coverage The code defaults the condition (i + SUBSECTION LENGTH linenum) to this else-block, and we currently do not have a set of inputs to test this. To enter the while-block, linenum must be at least 2. Since ( ) let s pick linenum = 2 Coverage Coverage 13/34

18 Coverage An example to satisfy these conditions could include, i = 5 and chars.length = 3. In context of testing, we need a test case to have at least 5 lines, and we do - with linenum = 8 and chars as previously defined. Coverage Coverage 14/34

19 Coverage Here we need an i such that ((i+1) % (2*6)!= 6), so we can choose i = 1 for that case. Our test condition with linenum = 8 will execute this code. Coverage Coverage 15/34

20 Coverage As a quick overview of what we just defined, Looking at each test case and bearing in mind the initial conditions, we can minimize the tests: Coverage Coverage 16/34

21 Coverage If we look at the conditions we need for each case, we can see that some tests must be executed together. So we continue to minimize... Leaving us with a final count of 3 test cases to invoke all the statements within the code: Coverage Coverage 17/34

22 Recall the edge coverage criterion from class: a test set T such that, by executing P for each d in T, each edge of P s control flow graph is traversed at least once. P is a program fragment d is an element in the domain D of P Can you think of any weakness in just fulfilling this criterion? Coverage Coverage 18/34

23 Recall the edge coverage criterion from class: a test set T such that, by executing P for each d in T, each edge of P s control flow graph is traversed at least once. P is a program fragment d is an element in the domain D of P Can you think of any weakness in just fulfilling this criterion? Compound boolean expressions (c 1 c 2 ) used in branching clauses can end up having unchecked constituents that may cause errors later on. Coverage Coverage 18/34

24 Simple example: if (x > 3 y = true) { x += 1; } How many test cases do we need to make sure the if-block is executed, and also that it is skipped? Coverage Coverage 19/34

25 Simple example: if (x > 3 y = true) { x += 1; } How many test cases do we need to make sure the if-block is executed, and also that it is skipped? 2 cases: one where the if-condition is met, and another where it is not. for example: < x = 4, y = true>, <x = 4, y = false> Coverage Coverage 19/34

26 Simple example: if (x > 3 y = true) { x += 1; } How many test cases do we need to make sure the if-block is executed, and also that it is skipped? 2 cases: one where the if-condition is met, and another where it is not. for example: < x = 4, y = true>, <x = 4, y = false> Coverage Coverage Note that 2 cases are not covered: one where x 3 and y = false, and another where x 3 and y = false. 19/34

27 Back to our example Our statement coverage has executed all edges of the control flow graph that have a line of code explicitly written. To ensure edge coverage, we need to go back through the program, and analyze which conditional statements we do not satisfy so we can make sure our tests also take the path where they skip the conditional block of code. Coverage Coverage 20/34

28 Looking at the main function again, we have: Our while loop is always guaranteed to execute the path where the condition is not met- that s how it terminates. Our only worry is the if-statement block here, in which both outcomes have been covered already by our test cases. Coverage Coverage 21/34

29 Let s consider our process function again: Again, this is a while loop, so we re fine. Coverage Coverage 22/34

30 Process function consideration cont d: Our only change we must make is ensure we have a test case in which the character in the third field is not one of the characters we re looking for. Instead of making a new test case, let s just alter the ones we have. Coverage Coverage 23/34

31 Our modified test cases now include the case where the if-condition is not met: Coverage Coverage 24/34

32 Our only concern is making sure a test case doesn t meet the condition- (i + SUBSECTION LENGTH < linenum)... Coverage Coverage 25/34

33 ... which we already covered to meet the else condition: Coverage Coverage 26/34

34 The same goes for the next if-else condition, since we needed to execute both blocks of code to meet the statement coverage criterion. Coverage Coverage The if (k < chars.length - 1) block does not execute at the end of each printed line, which covers the case when printing **** is skipped. 27/34

35 Which brings us to the end of our edge coverage consideration, and we finish with the following test cases: Coverage Coverage 28/34

36 Coverage Recall the condition coverage criterion from class: a test set T such that, by executing P for each element in T, each edge of P s control flow graph is traversed, and all possible values of the constituents of compound conditions are exercised at least once. Notice that this criterion strengthens the edge coverage criterion. Can you think of any weakness in just fulfilling this criterion? Coverage Coverage 29/34

37 Coverage Recall the condition coverage criterion from class: a test set T such that, by executing P for each element in T, each edge of P s control flow graph is traversed, and all possible values of the constituents of compound conditions are exercised at least once. Notice that this criterion strengthens the edge coverage criterion. Can you think of any weakness in just fulfilling this criterion? We may traverse through all edges of the flow graph and possible constituent values, but this does not protect us from errors like division by 0. Coverage Coverage 29/34

38 Coverage Simple example: if (x % 2 = 0 y % 2 = 0) { z = x/y; } How many test cases do we need to cover the condition coverage criterion? Coverage Coverage 30/34

39 Coverage Simple example: if (x % 2 = 0 y % 2 = 0) { z = x/y; } How many test cases do we need to cover the condition coverage criterion? 4 cases: test each constituent in the compound boolean expression <x is even, y is even>,<x is odd, y is even>,<x is even, y is odd>,<x is odd, y is odd>: for example: < x = 4, y = 6>, <x = 5, y = 6> < x = 4, y = 5>, <x = 5, y = 5> Coverage Coverage 30/34

40 Coverage Simple example: if (x % 2 = 0 y % 2 = 0) { z = x/y; } How many test cases do we need to cover the condition coverage criterion? 4 cases: test each constituent in the compound boolean expression <x is even, y is even>,<x is odd, y is even>,<x is even, y is odd>,<x is odd, y is odd>: for example: < x = 4, y = 6>, <x = 5, y = 6> < x = 4, y = 5>, <x = 5, y = 5> Coverage Coverage Note that even covering every case here does not protect us from cases like division by 0. 30/34

41 Recall the path coverage criterion from class: a test set T such that, by executing P for each d in T, all paths leading from the initial to the final node of P s control flow graph are traversed. Note that path coverage criterion edge coverage criterion. Can you think of any weakness in just fulfilling this criterion? Coverage Coverage 31/34

42 Recall the path coverage criterion from class: a test set T such that, by executing P for each d in T, all paths leading from the initial to the final node of P s control flow graph are traversed. Note that path coverage criterion edge coverage criterion. Can you think of any weakness in just fulfilling this criterion? Generally, this criterion is impractical as in the general case the number of execution paths is very large. This makes traversing through all control paths infeasible. Coverage Coverage 31/34

43 Simple example: if (x % 2 = 0 y % 2 = 0) { z = x/y; } How many test cases do we need to cover the condition coverage criterion? Coverage Coverage 32/34

44 Simple example: if (x % 2 = 0 y % 2 = 0) { z = x/y; } How many test cases do we need to cover the condition coverage criterion? Many, many cases- we would need to test every possible value of x and y that could make its way into the program. We could catch the division by 0 error from before. Coverage Coverage 32/34

45 Simple example: if (x % 2 = 0 y % 2 = 0) { z = x/y; } How many test cases do we need to cover the condition coverage criterion? Many, many cases- we would need to test every possible value of x and y that could make its way into the program. We could catch the division by 0 error from before. Coverage Coverage Note that this testing could end up taking a very long time as programs grow and become more complex- it s simply infeasible in most cases. 32/34

46 We covered a brief overview of statement coverage criterion, edge coverage criterion, conditional coverage criterion, and path coverage criterion and noted issues with each. We can see the value in testing programs with multiple techniques. If code is missing some sort of implementation, white box testing won t find it, but black box testing will! Coverage Coverage 33/34

47 References I Appendix References C. Ghezzi, M.Jazayeri, D.Mandrioli; Fundamentals of Software Engineering Prentice Hall PTR, /34