MSc Software Testing MSc Prófun hugbúnaðar

Transcription

1 MSc Software Testing MSc Prófun hugbúnaðar Fyrirlestrar 7 & 8 Structural Testing White-box tests. 29/8/27 Dr Andy Brooks 1

2 Case Study Dæmisaga Reference Structural Testing of Programs, A Survey, A A Omar and F A Mohammed, ACM SIGSOFT Software Engineering Notes vol 14 no 2, April 1989, pp plus extra teacher s notes 29/8/27 Dr Andy Brooks 2

3 Structural testing method The static analyis phase involves deriving the program flowgraph as a collection of vertices and edges. Each vertex represents a statement. Each edge between two vertices represents the control flow. The path selection phase involves choosing a finite set of paths in the flowgraph with a view to satisfying one or more coverage criteria. 29/8/27 Dr Andy Brooks 3

4 Structural testing method The test case generation phase involves generating the input data necessary to test the selected paths. The dynamic analysis phase involves executing the program using the test data and analysing the output produced. compare expected output with actual output 29/8/27 Dr Andy Brooks 4

5 tornado/skýstrókur extra Comparing expected output with actual output No Year (y) Month (m) Day (d) Wind (w) Expected Actual Result Please retry your input was incorrect On that Sunday it was an F1 tornado On that Monday it was an F1 tornado On that Thursday it was an F2 tornado On that Friday it was an F3 tornado Please retry your input was incorrect On that Wednesday it was an F4 tornado On that Thursday it was an F5 tornado Please retry your input was incorrect Please retry your input was incorrect Please retry your input was incorrect 12 x Please retry your input was incorrect Just 12 of many test cases... 29/8/27 Dr Andy Brooks 5

6 Static Analysis The program is assumed to have compiled successfully. Each executable statement is assigned a number. Each statement is represented as a node with its number inside. An arc represents the transition from one statement to another. A sequence of nodes with only one incoming and one outgoing arc can be reduced to a single node. 29/8/27 Dr Andy Brooks 6

7 Figure 1 Sample Fortran program 1 READ(1,3) I, J, K 3 FORMAT(3I4) 3 integers, field width 4 2 IF (I.EQ.) GO TO 3 3 A = SQRT(I**2 + J**2) 4 B = I+J 5 GO TO A = SQRT(J**2 + K**2) 7 B = J+K 8 4 WRITE(2,4) A, B 4 FORMAT(2F8.3) 2 reals, field width 8, 3 decimal 9 STOP 29/8/27 Dr Andy Brooks 7

8 29/8/27 Dr Andy Brooks program graph reduced program graph matrix representation of the reduced graph CC=2

9 extra Drawing program graphs Sequence If While Switch Non-branching sequence of statements 29/8/27 Dr Andy Brooks 9

10 extra Cyclomatic Complexity (CC) The cyclomatic complexity of a program graph may be calculated in 3 ways. 1. Number of regions no. bounded areas + the unbounded area surrounding graph 2. Number of edges - number of nodes Number of predicate nodes + 1 switch is not a predicate node 29/8/27 Dr Andy Brooks 1

11 extra Cyclomatic Complexity (CC) The value of cyclomatic complexity defines the number of independent paths and provides an upper bound for the number of tests that must be conducted to ensure all statements have been executed at least once. An independent path must traverse at least one edge that has not been traversed before. Basis path testing involves writing test cases to execute all the independent paths. 29/8/27 Dr Andy Brooks 11

12 extra Cyclomatic Complexity is 5. At least 5 test cases are needed to ensure statement coverage and that every predicate has been evaluated true and false at least once. Code with high complexity (high CC) may get targeted for additional tests. But note that there are disagreements over the theoretical foundation and usefulness of the CC metric. MAXV is the maximum cyclomatic complexity value for any single method within a class. 29/8/27 Dr Andy Brooks 12

13 path selection criteria/control flow criteria criteria/viðmiðanir execute each statement at least once This criteria is poor and various classes of errors are undetected. If X=X+Y is erroneously written X=X-Y and the test case sets Y to, no error will be detected. If X=X*2 is erroneously written X=X+2 and the test case sets X to 2, no error will be detected. 29/8/27 Dr Andy Brooks 13

14 path selection criteria/control flow criteria execute each statement at least once F P T S1 If the test data causes P to be true then correct results are output. all statements executed erroneous transfer of control S2 But the erroneous transfer of control when P is false will be undetected. 29/8/27 Dr Andy Brooks 14

15 path selection criteria/control flow criteria exhaustive path testing In exhaustive path testing, each path in the program is executed at least once. However, even for small programs, the number of paths can be extremely large. Exhaustive testing is an impossible task. Even if exhaustive testing were possible, a program can still contain errors: There is no guarantee the program meets its specification. A program may be incorrect because of missing paths. If A-B is written instead of ABS(A-B), this error may not be detected by a test. 29/8/27 Dr Andy Brooks 15

16 path selection criteria/control flow criteria exhaustive path testing 2 iterations A The number of unique logical paths between A and B is: N 1 14 Executing a path every 5 minutes would take around one billion years. Executing a path every second would take around three million years. B 29/8/27 Dr Andy Brooks 16

17 path selection criteria/control flow criteria decision coverage (branch testing) Each branch is executed at least once. This means each decision has a true and false outcome at least once. This means each statement is executed at least once. For multi-way decisions (case, switch), the test cases exercise all possible outcomes. 29/8/27 Dr Andy Brooks 17

18 path selection criteria/control flow criteria essential branches criteria In normal branch testing, all branches are treated equally. This means that there are many branches which are executed by many test data. In essential branch testing, redundant test cases are removed by considering only essential branches. essential essential 29/8/27 Dr Andy Brooks 18

19 path selection criteria/control flow criteria condition coverage criterion each condition in a decision takes on all possible outcomes at least once But condition coverage may not satisfy branch coverage. Assume IF (A and B) is being tested. One test case could be: A is true, B is false Another test case could be: A is false, B is true Both conditions A and B take on true and false. But these two test cases do not cause the THEN clause of the IF to be executed. 29/8/27 Dr Andy Brooks 19

20 path selection criteria/control flow criteria decision/condition coverage criterion Each condition in a decision takes on all possible outcomes at least once. Each decision has a true and false outcome at least once. But: if an OR is found to be true early, subsequent conditions need not be evaluated. if an AND is found to be false early, subsequent conditions need not be evaluated. so errors in logical expressions are not necessarily made visible. 29/8/27 Dr Andy Brooks 2

21 path selection criteria/control flow criteria multiple-condition coverage criterion Test cases are written to cover all possible combinations of condition outcomes. Multiple-condition coverage also satisfies decision coverage, condition coverage, and decision/condition coverage. DO I = 1 TO M WHILE N... END 1. I < M and N is true 2. I < M and N is false 3. I > M and N is true 4. I > M and N is false If (A<1) & (B > 2) Draw a truth table for & and create test data for each outcome. A B Outcome 1 1 T 1 F 1 F F extra 29/8/27 21

22 extra Just another condition testing strategy If Grade == A test with: Grade < A Grade == A Grade > A 29/8/27 Dr Andy Brooks 22

23 extra Loop testing strategy Skip/avoid the loop Go round once Go round twice Go round m times where m < n, the maximum number of iterations n-1, n, n+1 29/8/27 Dr Andy Brooks 23

24 Infeasible paths Testing can be problematic if infeasible paths are present. In traditional approaches, paths were chosen whether or not they were feasible. if (age < 21) System.out.println ("Youth is wonderful. Enjoy."); else if (age < 21) System.out.println ("No test will ever get here."); dead code Logical contradiction. 29/8/27 Dr Andy Brooks 24

25 Infeasible paths if (x < ) System.out.println ( S1 ); else System.out.println ( S2 ); if (x > 2) System.out.println ( S3 ); else System.out.println ( S4 ); We can never have: S1 S3 If we know x is always > 5 then we can never have S1 S1 S2 S3 S4 S4 29/8/27 Dr Andy Brooks 25

26 path selection criteria/data flow criteria 3.2. Data flow criteria During program execution a statement may: define a variable (assign a value) reference a variable (access a value) undefine a variable (variable destroyed) loop control variables are undefined once the loop terminates Four types of error are possible: A variable is undefined and then referenced. A variable is defined and then undefined. A variable is defined and then defined again. A variable is defined yet never referenced. A compiler might detect some of these. Which? By following the path of a variable, these four types of error can be detected. 29/8/27 Dr Andy Brooks 26

27 path selection criteria/data flow criteria 3.2. Data flow criteria (Fig. 5) u U d r U undefined D defined R referenced A abnormal u D d,u A d,r,u r d R Data flow errors are: ur du dd 29/8/27 Dr Andy Brooks 27

28 path selection criteria/data flow criteria 3.2. Data flow criteria 1. For each variable, select a path which contains a definition use of the variable, then either a computational or predicate use of that variable. 2. For each variable, select all the paths which include both a definition of the variable and a predicate use of the variable. 3. For each variable, select all the paths which include both a definition of the variable and a computation use of the variable. 4. For each variable, select all the paths which define the variable, then make computation and predicate use of it. This family of test criteria fall between branch testing and every path testing in terms of difficulty of fulfillment. 29/8/27 Dr Andy Brooks 28

29 path selection criteria/data flow criteria 3.2. Data flow criteria y= f(x1,x2) computational-use of x1 and x2 definition of y IF X=Y THEN GOTO M predicate use of X and Y 29/8/27 Dr Andy Brooks 29

30 4. Test case generation Work out the test inputs required for each selected test path. Performing this task manually can take a lot of time. 29/8/27 Dr Andy Brooks 3

31 4. Test case generation Alternatively, generate test data iteratively or randomly and obtain feedback on which paths have been executed by instrumenting the software under test. Continue to generate test data until all the selected test paths have been executed at least once. Information about the number of times a test path has been executed can be useful. test paths with high frequency may be regarded as sufficiently tested 29/8/27 Dr Andy Brooks 31

32 6. Dynamic Analysis Run the program with a set of test cases which satisfies one ore more criteria. Compare expected output with actual output. Correct any identified faults. Removing identified faults does not mean the program is free of faults. 29/8/27 Dr Andy Brooks 32

33 7. Reliability and testing strategies adequacy A set T of test data for a program P is reliable if it reveals that P contains an error whenever P is incorrect. But we cannot construct such reliable test data. Empirically, Howden (1976) found that path testing was almost reliable for about 65% of program errors across 11 programs. But the programs involved were very simple in which each path could be tested. And not all error types were considered. 29/8/27 Dr Andy Brooks 33

34 8. Conclusion Structural testing comprises four phases: static analysis phase flowgraph developed path selection phase control flow criteria data flow criteria test case generation phase possibly using random test data and instrumented software under test dynamic analysis phase compare expected and actual output 29/8/27 Dr Andy Brooks 34

35 8. Conclusion But although we can find a systematic and automated method of testing, yet the testing theory has failed to yield any meaningful comparisons of methods. 29/8/27 Dr Andy Brooks 35

36 8. Conclusion What we shall try to do is to try different combination of methods to extract empirically the optimal combination. 29/8/27 Dr Andy Brooks 36