MSc Software Testing and Maintenance MSc Prófun og viðhald hugbúnaðar Fyrirlestrar 31 & 32 Structural Testing White-box tests. 27/1/25 Dr Andy Brooks 1
Case Study Dæmisaga Reference Structural Testing of Programs, A Survey, A A Omar and F A Mohammed, ACM SIGSOFT Software Engineering Notes vol 14 no 2, April 1989, pp 62-7. plus extra teacher s notes 27/1/25 Dr Andy Brooks 2
Structural testing method The static analyis phase involves deriving the program flowgraph as a collection of vertices and edges. Each vertex represents a statement. Each edge between two vertices represents the control flow. The path selection phase involves choosing a finite set of paths in the flowgraph with a view to satisfying one or more coverage criteria. 27/1/25 Dr Andy Brooks 3
Structural testing method The test case generation phase involves generating the input data necessary to test the selected paths. The dynamic analysis phase involves executing the program using the test data and analysing the output produced. compare expected output with actual output 27/1/25 Dr Andy Brooks 4
Static Analysis The program is assumed to have compiled successfully. Each executable statement is assigned a number. Each statement is represented as a node with its number inside. An arc represents the transition from one statement to another. A sequence of nodes with only one incoming and one outgoing arc can be reduced to a single node. 27/1/25 Dr Andy Brooks 5
Figure 1 Sample Fortran program 1 READ(1,3) I, J, K 3 FORMAT(3I4) 2 IF (I.EQ.) GO TO 3 3 A = SQRT(I**2 + J**2) 4 B = I+J 5 GO TO 4 6 3 A = SQRT(J**2 + K**2) 7 B = J+K 8 4 WRITE(2,4) A, B 4 FORMAT(2F8.3) 9 STOP 27/1/25 Dr Andy Brooks 6
27/1/25 Dr Andy Brooks 7 1 2 3 4 5 8 9 6 7 1 2 3 8 9 6 9 1 8 1 6 1 3 1 1 2 1 1 9 8 6 3 2 1 program graph reduced program graph matrix representation of the reduced graph
extra Drawing program graphs Sequence If While Switch Non-branching sequence of statements 27/1/25 Dr Andy Brooks 8
extra Cyclomatic Complexity (CC) The cyclomatic complexity of a program graph may be calculated in 3 ways. 1. Number of regions no. bounded areas + the unbounded area surrounding graph 2. Number of edges - number of nodes + 2 3. Number of predicate nodes + 1 switch is not a predicate node 27/1/25 Dr Andy Brooks 9
extra Cyclomatic Complexity (CC) The value of cyclomatic complexity defines the number of independent paths and provides an upper bound for the number of tests that must be conducted to ensure all statements have been executed at least once. An independent path must traverse at least one edge that has not been traversed before. Basis path testing involves writing test cases to execute all the independent paths. 27/1/25 Dr Andy Brooks 1
extra Cyclomatic Complexity is 5. At least 5 test cases are needed to ensure statement coverage and that every predicate has been evaluated true and false at least once. Code with high complexity (high CC) may get targeted for additional tests. But note that there are disagreements over the theoretical foundation and usefulness of the CC metric. MAXV is the maximum cyclomatic complexity value for any single method within a class. 27/1/25 Dr Andy Brooks 11
path selection criteria/control flow criteria 3.1.1 execute each statement at least once This criteria is poor and various classes of errors are undetected. If X=X+Y is erroneously written X=X-Y and the test case sets Y to, no error will be detected. If X=X*2 is erroneously written X=X+2 and the test case sets X to 2, no error will be detected. 27/1/25 Dr Andy Brooks 12
path selection criteria/control flow criteria 3.1.1 execute each statement at least once F P erroneous transfer of control T S1 S2 If the test data causes P to be true then correct results are output. But the erroneous transfer of control when P is false will be undetected. 27/1/25 Dr Andy Brooks 13
path selection criteria/control flow criteria 3.1.2 exhaustive path testing In exhaustive path testing, each path in the program is executed at least once. However, even for small programs, the number of paths can be extremely large. Exhaustive testing is an impossible task. Even if exhaustive testing were possible, a program can still contain errors: There is no guarantee the program meets its specification. A program may be incorrect because of missing paths. If A-B is written instead of ABS(A-B), this error may not be detected by a test. 27/1/25 Dr Andy Brooks 14
path selection criteria/control flow criteria 3.1.2 exhaustive path testing 2 iterations A The number of unique logical paths between A and B is: 5 2 + 5 19 +... + 5 1 K 1 14 Executing a path every 5 minutes would take around one billion years. Executing a path every second would take around three million years. B 27/1/25 Dr Andy Brooks 15
path selection criteria/control flow criteria 3.1.3 decision coverage (branch testing) Each branch is executed at least once. This means each decision has a true and false outcome at least once. This means each statement is executed at least once. For multi-way decisions (case, switch), the test cases exercise all possible outcomes. 27/1/25 Dr Andy Brooks 16
path selection criteria/control flow criteria 3.1.4 essential branches criteria In normal branch testing, all branches are treated equally. This means that there are many branches which are executed by many test data. In essential branch testing, redundant test cases are removed by considering only essential branches. essential essential 27/1/25 Dr Andy Brooks 17
path selection criteria/control flow criteria 3.1.5 condition coverage criterion But condition coverage may not satisfy branch coverage. Assume IF (A and B) is being tested. One test case could be: A is true, B is false Another test case could be: A is false, B is true But these two test cases do not cause the THEN clause of the IF to be executed. 27/1/25 Dr Andy Brooks 18
path selection criteria/control flow criteria 3.1.6 decision/condition coverage criterion Each condition in a decision takes on all possible outcomes at least once. Each decision has a true and false outcome at least once. But: if an OR is found to be true early, subsequent conditions need not be evaluated. if an AND is found to be false early, subsequent conditions need not be evaluated. errors in logical expressions are not necessarily made visible. 27/1/25 Dr Andy Brooks 19
path selection criteria/control flow criteria 3.1.7 multiple-condition coverage criterion Test cases are written to cover all possible combinations of condition outcomes. Multiple-condition coverage also satisfies decision coverage, condition coverage, and decision/condition coverage. DO I = 1 TO M WHILE N... END 1. I < M and N is true 2. I < M and N is false 3. I > M and N is true 4. I > M and N is false If (A<1) & (B > 2) Draw a truth table for & and create test data for each outcome. A B Outcome 1 1 T 1 F 1 F F extra 27/1/25 2
extra Just another condition testing strategy If Grade == A test with: Grade < A Grade == A Grade > A 27/1/25 Dr Andy Brooks 21
extra Loop testing strategy Skipping the loop Go round once Go round twice Go round m times where m < n, the maximum number of iterations n-1, n, n+1 27/1/25 Dr Andy Brooks 22
Infeasible paths Testing can be problematic if infeasible paths are present. In traditional approaches, paths were chosen without regard to whether or not they were feasible. if (age < 21) System.out.println ("Youth is wonderful. Enjoy."); else if (age < 21) System.out.println ("No test will ever get here."); 27/1/25 Dr Andy Brooks 23
path selection criteria/data flow criteria 3.2. Data flow criteria During program execution a statement may: define a variable (assign a value) reference a variable (access a value) undefine a variable (variable destroyed) loop control variables are undefined once the loop terminates Four types of error are possible: A variable is undefined and then referenced. A variable is defined and then undefined. A variable is defined and then defined again. A variable is defined yet never referenced. A compiler might detect some of these. Which? By following the path of a variable, these four types of error can be detected. 27/1/25 Dr Andy Brooks 24
path selection criteria/data flow criteria 3.2. Data flow criteria 1. For each variable, select a path which contains a definition use of the variable, then either a computational or predicate use of that variable. 2. For each variable, select all the paths which include both a definition of the variable and a predicate use of the variable. 3. For each variable, select all the paths which include both a definition of the variable and a computation use of the variable. 4. For each variable, select all the paths which define the variable, then make computation and predicate use of it. This family of test criteria fall between branch testing and every path testing in terms of difficulty of fulfillment. 27/1/25 Dr Andy Brooks 25
4. Test case generation Work out the test inputs required for each selected test path. Performing this task manually can take a lot of time. Alternatively, generate test data randomly and obtain feedback on which paths have been executed by instrumenting the software under test. Continue to generate test data at random until all the selected test paths have been executed at least once. Information about the number of times a test path has been executed can be useful. 27/1/25 Dr Andy Brooks 26
7. Reliability and testing strategies adequacy A set T of test data for a program P is reliable if it reveals that P contains an error whenever P is incorrect. But we cannot construct such reliable test data. Empirically, Howden (1976) found that path testing was almost reliable for about 65% of program errors across 11 programs. But the programs involved were very simple in which each path could be tested. And not all error types were considered. 27/1/25 Dr Andy Brooks 27
8. Conclusion Structural testing comprises four phases: static analysis phase flowgraph developed path selection phase control flow criteria data flow criteria test case generation phase possibly using random test data and instrumented software under test dynamic analysis phase compare expected and actual output 27/1/25 Dr Andy Brooks 28
8. Conclusion But although we can find a systematic and automated method of testing, yet the testing theory has failed to yield any meaningful comparisons of methods. 27/1/25 Dr Andy Brooks 29
8. Conclusion What we shall try to do is to try different combination of methods to extract empirically the optimal combination. 27/1/25 Dr Andy Brooks 3