Structural Testing. White Box Testing & Control Flow Analysis

Transcription

1 Structural Testing White Box Testing & Control Flow Analysis

2 Functional vs. Structural Functional Have I built the right product? Tests derived from the program specification Internal Structure ignored Structural Have I built the product right? Tests derived from structure Aim is to completely exercise the code

3 Internal Structure Routes from start to finish. n is the max. iterations and x is the number of branches within loop i 1 n 1 x i

4 Internal Structure categorising one exam result into one of five categories for 10 students. How many paths?

5 Internal Structure Approx million paths! Again, we have a massive explosion!

6 What Is Test Assessment? Once a test set T, a collection of test inputs, has been developed, we ask: How good is T? It is the measurement of the goodness of T which is known as test assessment. Test assessment is carried out based on one or more criteria.

7 Test Assessment Test assessment provides the following information: A metric, also known as the adequacy score or coverage, usually between 0 and 1. A list of all the weaknesses found in T, which when removed, will raise the score to 1. The weaknesses depend on the criteria used for assessment.

8 Test Assessment (continued) This is continued until all weaknesses are overcome, i.e. the adequacy criterion is satisfied (coverage=1). In some instances it may not be possible to satisfy the adequacy criteria for one or more of the following reasons: Lack of sufficient manpower Weaknesses that cannot be removed because they are infeasible. The cost of removing the weaknesses is not justified. By improving T by removing its weaknesses, one usually tests the program more thoroughly than it has been tested so far.

9 Test Assessment Procedure 0 Develop T 1 Select an adequacy criterion C Yes Measure adequacy of T w.r.t. C. Is T adequate? No Improve T Yes 5 6 More testing is warranted? Done No

10 Principle Underlying Test Assessment There is a uniform principle that underlies test assessment throughout the testing process. This principle is known as the coverage principle.

11 Control flow graph Control flow graph (CFG) of a program is a representation of the flow of execution within the program. It is useful in program analysis specifically during test assessment and improvement.

12 Formal Definition of A CFG CFG G is :: G=(N,A) where N: set of nodes and A: set of arcs There is a unique entry node en in N. There is a unique exit node ex in N. A node represents a single statement or a block of statements. A block is a single-entry-single-exit sequence of instructions that are always executed in a sequence without any diversion of path except at the end of the block.

13 Control flow graph-continued Every statement in a block, except possibly the first one, has exactly one predecessor. Similarly, every statement in the block, except possibly the last one, has exactly one successor. An arc a in A is a pair (n,m) of nodes from N which represent transfer of control from node n to node m.

14 Control flow graph-continued A path of length k in G is an ordered sequence of arcs, from A such that: a 1, a 2,.. a k The first node in the path, a 1 is en The last node in the path, a 2 is ex For any two adjacent arcs a i = (n,m) and a i+1 = (p,q), m=p. A path is considered executable or feasible if there exists a test case which causes this path to be traversed during program execution, otherwise the path is unexecutable or infeasible.

15 Control Graph Example FindMean (FILE ScoreFile) { float SumOfScores = 0.0; int NumberOfScores = 0; float Mean=0.0; float Score; Read(ScoreFile, Score); } while (! EOF(ScoreFile)) { if (Score > 0.0 ) { SumOfScores = SumOfScores + Score; NumberOfScores++; } Read(ScoreFile, Score); } if (NumberOfScores > 0) { Mean = SumOfScores / NumberOfScores; printf( The mean score is %f\n, Mean); } else printf ( No scores found in file\n );

16 FindMean (FILE ScoreFile) { float SumOfScores = 0.0; int NumberOfScores = 0; 1 float Mean=0.0; float Score; Read(ScoreFile, Score); 2 while (! EOF(ScoreFile)) { 3 if (Score > 0.0 ) { SumOfScores = SumOfScores + Score; NumberOfScores++; } 4 6 } Read(ScoreFile, Score); 5 } /* Compute the mean and print the result */ if (NumberOfScores > 0) { Mean = SumOfScores / NumberOfScores; 7 printf( The mean score is %f\n, Mean); } else 8 printf ( No scores found in file\n );

17 Start F 1 2 T 4 T 3 F 5 6 T F 7 8 Exit

18 Finding The Test Cases Start (Data set must be empty) c 4 F (Positive score) d T 1 2 T 3 a (Covered by any data) b (Data set must contain at least one value) F e (Negative score) (Total < 0.0) h 5 6 T F 7 8 j Exit i (Total > 0.0) k g (Reached if either d or e)

19 Compound statements Line n int a, b; Line n int a; Line n+1 int b;

20 Compound Statements Line n for (i=0; i<n; i++) Line n-1 i = 0; Line n while (i<n) { Line n+1. Line n+m i++;}

21 Structure-Based Test Assessment Based on the CFG of a program several test adequacy criteria can be defined. Some are: Statement coverage criterion Branch coverage criterion Condition coverage criterion Path coverage criterion

22 Statement coverage The coverage domain consists of all statements in the program. Restated, in terms of the control flow graph, it is the set of all nodes in a graph G. A test T satisfies the statement coverage criterion if upon execution of the program P on each element of T, each statement of P has been executed at least once. Restated in terms of G, T is adequate w.r.t. the statement coverage criterion if each node in N is on at least one of the paths traversed when P is executed on each element of T.

23 Statement coverage-weakness Consider the following program: A simple absolute value function int abs(int x) { if (x>=0) x = 0 - x; return x; } STOP: Everyone draw a Control Graph.

24 Statement coverage :: Weakness Suppose that T= {(x=0)}. Clearly, T satisfies the statement coverage criterion. But is the program correct and is the error revealed by T, which is adequate w.r.t. the statement coverage criterion? What do you suggest we do to improve T?

25 Branch (or Edge) Coverage In G there may be nodes which correspond to conditions in P. Such nodes, also called condition nodes, contain branches in P. Each such node is considered covered if during some execution of P, the condition evaluates to true and false; these executions of P need not be the same.

26 Branch coverage The coverage domain consists of all branches in G. Restated, in terms of the control flow graph, it is the set of all arcs exiting the condition nodes. A test T satisfies the branch coverage criterion if upon execution of P on each element of T, each branch of P has been executed at least once.

27 Branch Coverage Class exercise: Identify all condition nodes in the flow graph you have drawn earlier. Does T= {(x=0)} satisfy the branch coverage criterion? If not, then improve it so that it does.

28 Branch Coverage :: Weakness Consider the following program that is supposed to check if the input data item is in the range 0 to 100, inclusive: int check(int x) { int check; if ((x >= 0 )&& (x <= 200)) check=true; else check=false; return check; }

29 Condition Coverage Condition nodes in G might have compound conditions. For example, in the check program the condition node contains the condition: ((x>=0 ) && (x<=200)) This is a compound condition which consists of the elementary conditions x>=0 and x<=200. STOP: Draw a Control Graph for this new problem

30 Condition Coverage A compound condition is considered covered if all of its constituent elementary conditions evaluate to true and false, respectively, during some execution of P. A test set T is adequate w.r.t. condition coverage if all conditions in P are covered when P is executed on elements of T.

31 Condition Coverage Class exercise: Improve T from the previous exercise so that it is adequate w.r.t. the condition coverage criterion for the check function and does not reveal the error. Is this possible?

32 Branch / Conditional Coverage Weakness int getvalue(int x, int y, int z){ int result = 0; if (x!=0) y= 5; else z= z-x; if (z > 1) z=z/x; else z=y; } return result;

33 Path Coverage A path through a program is a sequence of arcs such that the entry node of the program CFG is the first node on the path and the exit node is the last one on the path. A test set T is considered adequate w.r.t. the path coverage criterion if all paths in P are executed at least once upon execution on each element of T. STOP: Everyone draw the new control graph.

34 Class exercise: Path Coverage Construct T for getvalue such that T is adequate w.r.t. the path coverage criterion and does not reveal the error. Is the above possible?

35 Path Coverage Weaknesses The number of paths in a program is usually very large; very, very large!

36 Path Coverage Weaknesses It is an infinite or a prohibitively large number of paths that prevent the use of this criterion in practice. Suppose that a test set T covers all paths. Will it guarantee that all errors in P are revealed? Is obtaining 100% path coverage equivalent to exhaustive testing?

37 Variants of path coverage As path coverage is usually impossible to attain, other heuristics have been proposed. Loop coverage: Make sure that each loop is executed 0, 1, and 2 times. Try several combinations of if and switch statements. The combinations must be derived from requirements.

38 Hierarchy in Control flow criteria Path coverage Condition coverage Branch coverage Statement coverage