SFWR ENG 3S03: Software Testing

Transcription

1 (Slide 1 of 52) Dr. Ridha Khedri Department of Computing and Software, McMaster University Canada L8S 4L7, Hamilton, Ontario Acknowledgments: Material based on [?]

2 Techniques (Slide 2 of 52) Empirical Principles Complete Coverage Principle 5 Statement Coverage Edge Coverage Other Testing Criteria

3 Techniques (Slide 3 of 52) In the case of software testing, the usual analogies between traditional engineering fields and software engineering fail to provide useful suggestions You wish to test the bridge s ability to sustain weight If you have verified that the bridge can sustain 1,000 tons it will sustain any weight less than or equal to 1,000 tons (under the same circumstances) The same criterion cannot be applied to software

4 Techniques procedure binary-search key: in element; table: in elementtable; found: out Boolean is begin bottom := tablefirst; top := tablelast; while bottom ă top loop if (bottom + top) rem 2 0 then middle := (bottom + top - 1) / 2; else middle := (bottom + top) / 2; end if; if key ď table (middle) then top := middle; else bottom := middle + 1; end if; end loop; found := key = table (middle); end binary-search; (Slide 4 of 52)

5 Techniques (Slide 5 of 52) The features of most engineering constructions have a continuity property Small differences in operating conditions will not result in dramatically different behaviors Such a continuity property is totally lacking in software In software engineering, testing should be performed as systematically as possible by stating clearly what result one expects to obtain (Need requirements) Often in practice, testing is performed in an unstructured way without applying any criterion

6 Techniques (Slide 6 of 52) if the results delivered by the system are different from the expected ones in just one case, this unequivocally shows that the system is incorrect; by contrast, a correct behavior of the system on a finite number of cases does not guarantee correctness in the general case. Dijkstra (in Dahl et al. [1972]) We could have built a program that behaves properly for even integer numbers but not odd numbers Clearly, any number of tests with even input values will fail to show the error

7 Techniques (Slide 7 of 52) We should simply keep in mind that no absolute certainty can be gained from a pure testing activity Unit testing should be considered only one of the means to analyze the behavior of a system should be integrated with other verification techniques (+ confidence) Unit testing should be based on sound and systematic techniques So that, after testing, we may have a better understanding of the product s reliability

8 Techniques (Slide 8 of 52) A natural approach such as using randomly generated test cases turns out to be inappropriate in many situations Consider the following program fragment: Suppose that the programmer incorrectly wrote z := 2 instead of z:= 22 x y ùñ The error is immediately discovered If we use a random number generator ùñ it is very unlikely that x y will be tested read (x); read (y); if x = y then z := 2; else z := 0; end if; write (z);

9 Techniques (Slide 9 of 52) Testing should help locate errors, not just detect their presence The result of testing should not be viewed as simply providing a Boolean answer to the question of whether the software works properly or not Tests should be organized in a way that helps isolate errors (This information can be used in debugging) Testing should be repeatable (sounds fairly obvious) At sea level, pure water will boil at 100 degrees centigrade (212 degrees Fahrenheit).

10 Techniques (Slide 10 of 52) Hard-to-repeat experiments are not infrequent, and they are responsible for the major difficulties surrounding the debugging activity A typical example is the case of uninitialized variables /* x is not initialized */ if x = 0 then write ( abnormal ); else write ( normal ); end if; The issue of repeatability becomes even more critical with concurrent software

11 Techniques (Slide 11 of 52) Testing should be accurate (increase the reliability of testing) The accuracy of the testing activity depends on the level of precision - and maybe even formality - of the software specifications Requirements 1: As a consequence of input stimulus x, the system should produce output y within t milliseconds. Requirements 2: As a consequence of input stimulus x, the system should produce output y within t milliseconds, no matter which other events happen in the system during this time.

12 Techniques (Slide 12 of 52) If a formula expresses a property of software outputs in a mathematical way, it will be easier to check (possibly 1 ď i ă n : apiq ď api ` 1q q We have an effective way to verify whether the program has produced a valid array a Since it may be unclear whether the result of an execution is indeed correct, verifying it against a formal specification can enhance the reliability of the testing activity

13 Techniques (Slide 13 of 52) we introduce the testing terminology in a precise manner The terminology, however, is not standardized we show the limitations of testing from a mathematical point of view Let P be a program, D def dom P R def ran P

14 Techniques (Slide 14 of 52) We will assume that P behaves as a function - possibly partial - with domain D and range R Let O R denote the requirement on output values of P as stated in P s specification Definition For a given d P D, P is said to be correct for d if Ppdq satisfies O R.

15 Techniques (Slide 15 of 52) Definition P is correct d P D : P correct for d q. The presence of an error or defect is demonstrated by showing that Dpd d P D : P is not correct for d q

16 Techniques (Slide 16 of 52) Definition A test case is an element d of D. A test set T is a finite set of test cases, that is, a finite subset of D. P is correct for T if it is correct for all elements in T. In such a case, we also say that T is successful for P.

17 Techniques (Slide 17 of 52) A test set T is said to be ideal if, whenever P is incorrect, there exists a pd P T q such that P is incorrect for d. P is correct _ Dpd d P T : P is incorrect for d q ùñ T is ideal ùñ T is NOT ideal P is NOT correct d P T : P is correct for d q

18 Techniques (Slide 18 of 52) In other words, an ideal test set always shows the existence of an error in a program, if an error exists. Obviouslly, for a correct program, any test set is ideal. Also, if T is an ideal test set and T is successful for P, then P is correct. Definition A test selection criterion C is a subset of PpDq, where PpDq denotes the set of all finite subsets of D.

19 Techniques (Slide 19 of 52) Example for a program whose D dom P N C could require that test sets contain at least three elements, one of them being negative, one positive, and one T T P C : Dpx, y, z x, y, z P T : x ă 0 ^ y ą 0 ^ z 0 q Definition A test set T satisfies C if it belongs to C.

20 Techniques (Slide 20 of 52) C may be described by a suitable formula that must be satisfied by all elements of any of the subsets of D that belong to it C tt p u where p is the characteristic predicate of C T P C ðñ p Definition A selection criterion C is consistent T 1, T 2 T 1, T 2 P C : T 1 is successful ðñ T 2 is successful

21 Techniques (Slide 21 of 52) If one is provided with a consistent criterion, there is no theoretical reason to make a specific choice of a test set among those satisfying C We will later see, though, that there are practical reasons for preferring one test set over another Definition A criterion C is complete if, whenever P is incorrect, there is an unsuccessful test set that satisfies C. Remark Any test set T satisfying a consistent and complete criterion C could be used to decide P s correctness.

22 Techniques (Slide 22 of 52) Exercise P is a program to sort sequences of integers P works properly only if the length of the sequence to be sorted is even Give a complete and consistent criterion for P Definition We say that a testing criterion C 1 is finer than C 2 if, for any program P, for every test set T 1 satisfying C 1, there exists a subset of T 1, say T 2, that satisfies C 2. Unfortunately, none of the above definitions is effective

23 Techniques (Slide 23 of 52) (1) Consider the correctness of a program P Suppose you may formally specify the functional requirements as a first-order formula FRpd, uq, having d and u as free variables, and let Ppdq be the function associated with the program the correctness of P may be formally stated u d P Domain(P) ^ u P Range(P) : u Ppdq ùñ FRpd, uq q This statement, however, cannot be decided, since doing so would mean we would have to decide the truth of any first-order formula (which is an undecidable problem)

24 Techniques (Slide 24 of 52) (2) In some cases, it could even be impossible to decide whether a value d is in a test set T or not, depending on how T is defined - another well-known undecidable problem Conclusion: a test is ideal, It is impossible to decide whether a criterion is consistent or complete, etc. These are in the long list of major program properties such as correctness, termination, and equivalence that are undecidable We need some practical possible criteria for the selection of test sets

25 Techniques Empirical Principles (Slide 25 of 52) A testing criterion attempts to group elements of the input domain into classes The elements of a given class are expected to behave in exactly the same way This way, we can choose a single test case as representative of each class Definition If the classes D i are such that Ypi : D i q D, we say that the testing criterion satisfies the complete coverage principle. Complete Coverage Principle We will see that many well-known practical testing criteria satisfy this principle

26 Techniques Empirical Principles (Slide 26 of 52) Example n ă 0 n 0 0 ă n ă ď n ď 200 n ą 200 facpnq error 1 n facpn 1q approximatepnq error It is quite natural to divide the input domain into the classes tn ă 0u, tn 0u, t0 ă n ă 20u, t20 ď n ď 200u, and tn ą 200u Complete Coverage Principle Then, use test sets containing one element from each class

27 Techniques Empirical Principles (Slide 27 of 52) Complete Coverage Principle If we divide the input domain into disjoint classes D i such j i j : D i X D j H q then there is no particular reason to choose one element over another as a class representative If the classes are not disjoint, however, we have the possibility of choosing a representative to minimize the number of test cases Complete Coverage Principle If D i X D j H, a test case in D i X D j exercises both D i and D j

28 Techniques Empirical Principles Complete Coverage Principle Example: Consider the following testing criterion: the program must be tested with classes D 1, D 2, and D 3 of values for x, where D 1 td i d i mod 2 0 u Since D 2 td i D 3 td i d i ă 0 u d i mod 2 0 u D 1 X D 2 H and D 2 X D 3 H, We can choose just two test cases to exercise the program and still satisfy the complete coverage principle The test set { x = 48; x = -37 } would be acceptable (Slide 28 of 52) Complete Coverage Principle

29 Techniques Empirical Principles Complete Coverage Principle The complete coverage principle is subject to many interpretations, including several trivial ones We could insist that each class consist of a single element We could group all input data into just one class Thus, we may evaluate how good a testing criterion is based on how significant the representatives of the classes obtained by its decomposition are (Slide 29 of 52) Complete Coverage Principle

30 Techniques Empirical Principles (Slide 30 of 52) Complete Coverage Principle The application of the principle to produce significant tests is quite a challenging task Ideally, the complete coverage principle should help approximate consistent and complete criteria in the following way If the input domain D is decomposed into several subsets D i, Complete Coverage Principle then a test set may be chosen such that it contains at least one representative from each D i

31 Techniques Empirical Principles Complete Coverage Principle Example if x ą y then max := x; else max := x; end if; Any test set that causes the execution of both branches would most likely show the error (Slide 31 of 52) Complete Coverage Principle no matter which data are selected from the classes tpx, yq x ą y u and tpx, yq x ď y u

32 Techniques Empirical Principles (Slide 32 of 52) Complete Coverage Principle A careful selection of test cases, with the goal of satisfying the complete coverage principle, can in some sense approximate consistent and complete criteria We can increase the reliability of such criteria by choosing more than one element from each class If we feel that a particular partition is rather consistent and complete, but that there could be some exceptions, we could improve the method by generating several cases - maybe randomly - from the same class Complete Coverage Principle It turns out that this general principle can be applied in many different - and sometimes complementary - ways

33 Techniques (Slide 33 of 52) Testing in the small addresses the testing of individual modules There are two main approaches: White-box testing Black-box testing Testing a piece of software as a black box means operating the software without relying on any knowledge of the way it has been designed and coded Statement Coverage Edge Coverage Other Testing Criteria Test sets are developed, and their results evaluated, solely on the basis of the specification

34 Techniques (Slide 34 of 52) Testing software as a white box means using information about the internal structure of the software and maybe even ignoring its specification Both strategies are useful and somewhat complementary White-box testing tests what the program does black-box testing tests what it is supposed to do Statement Coverage Edge Coverage Other Testing Criteria They can both increase the level of confidence in the reliability of the component

35 Techniques (Slide 35 of 52) White-box testing is also called unit structural testing Consider the following well-known algorithm of Euclid: begin read (x); read (y); while x y loop if x ą y then x := x - y; else y := y - x; end if; end loop; gcd := x; end; The while condition suggests two possible cases to be exercised by tests: x y and x y Supplying as input data x 3, y 3, and x 4, y 5, we obtain a first kind of completeness with respect to exercising the main loop Statement Coverage Edge Coverage Other Testing Criteria

36 Techniques (Slide 36 of 52) Consider the following well-known algorithm of Euclid: begin read (x); read (y); while x y loop if x ą y then x := x - y; else y := y - x; end if; end loop; gcd := x; end; We could refine the condition x y by exercising both the cases x ą y and x ă y The test set tă x 3, y 3 ą, ă x 4, y 3 ą, ă x 3, y 4 ąu, the statements are executed at least once This is called the statement coverage criterion Statement Coverage Edge Coverage Other Testing Criteria

37 Techniques (Slide 37 of 52) Statement Coverage Intuitively, the statement coverage criterion is based on the observation that an error cannot be discovered if the parts of the program containing the error and generating the failure are not executed; thus, we should strive for complete coverage of statements The criterion, however, has some obvious weaknesses It is clear that executing some statement once and observing that it behaves properly is no guarantee that it is correct In block-structured languages, where statements may be part of more complex statements, it is not even clear what we mean by a statement Statement Coverage Edge Coverage Other Testing Criteria

38 Techniques (Slide 38 of 52) Statement Coverage A natural assumption is to assume as an elementary statement any statement that is derived from the ăstatementą nonterminal without producing any recursive occurrence of the same nonterminal Thus, in a simple, conventional block-structured language, elementary statements are assignment statements, I/O statements, and procedure calls Statement Coverage Edge Coverage Other Testing Criteria

39 Techniques (Slide 39 of 52) Statement Coverage With this assumption, the criterion can be stated as follows Statement coverage criterion: Select a test set T such that, by executing P for each d in T, each elementary statement of P is executed at least once. In general, the same input datum causes the execution of many statements Thus, we are left with the problem of trying to minimize the number of test cases and still ensure the execution of all statements Statement Coverage Edge Coverage Other Testing Criteria

40 Techniques (Slide 40 of 52) Statement Coverage Consider the following program fragment: read (x); read (y); if x ą 0 then /* I 1 */ write ( 1 ); /* W 1 */ else write ( 2 ); /* W 2 */ end if; if y ą 0 then /* I 2 */ write ( 3 ); /* W 3 */ else write ( 4 ); /* W 4 */ end if; Statement Coverage Edge Coverage Other Testing Criteria

41 Techniques (Slide 41 of 52) Statement Coverage Let D i denote the class of input values that cause execution of W i, (1 ď i ď 4) D 1 tx x ą 0 u, D 2 tx x ď 0 u, D 3 ty y ą 0 u, D 4 ty y ď 0 u If a representative is chosen from each of the classes, we are guaranteed that all of W 1, W 2, W 3, and W 4 are executed at least once Therefore, we might choose the following test set: Statement Coverage Edge Coverage Other Testing Criteria tă x 2, y 3 ą, ă x 13, y 51 ą, ă x 97, y 17 ą, ă x 1, y 1 ąu

42 Techniques (Slide 42 of 52) Statement Coverage This test set is not minimal with respect to the criterion, since every input datum belongs to two classes Thus, we may reduce the number of test cases to two For example, we can select the following representatives: t ă x 13, y 51 ą, ă x 2, y 3 ą u Statement Coverage Edge Coverage Other Testing Criteria

43 Techniques Edge Coverage We consider the graphical representation of the program control flow The complete coverage principle may also be applied to the control flow graph For any program fragment P its control flow graph G p is built inductively in the following way: (1) For each I/O, assignment, or procedure call statement, the following graph is built Entry into the statement (Slide 43 of 52) Statement Coverage Edge Coverage Other Testing Criteria Exit from the statement

44 Techniques (Slide 44 of 52) Edge Coverage (2) Let S 1 and S 2 denote two statements, and G 1 and G 2, denote their corresponding graphs. Then with the statement if cond then S 1 ; else S 2 ; end if; we associate the following graph. Statement Coverage Edge Coverage Other Testing Criteria G1 G2

45 Techniques (Slide 45 of 52) Edge Coverage (3) The following graph is associated with the statement if cond then S 1 ; end if; G1 Statement Coverage Edge Coverage Other Testing Criteria

46 Techniques (Slide 46 of 52) Edge Coverage (4) The next graph is associated with the statement while cond loop S 1 ; end loop; G1 Statement Coverage Edge Coverage Other Testing Criteria

47 Techniques (Slide 47 of 52) Edge Coverage (5) The following graph is associated with the statement S 1 ; S 2 ; G1 Once we have associated a control flow graph with a program fragment, we can state the following criterion: G2 Statement Coverage Edge Coverage Other Testing Criteria Edge coverage criterion: Select a test set T such that, by executing P for each d in T, each edge of P s control flow graph is traversed at least once.

48 Techniques (Slide 48 of 52) Other Testing Criteria Condition Coverage The edge coverage criterion can be further strengthened, to make it more likely to expose errors Condition coverage criterion: Select a test set T such that, by executing P for each element in T, each edge of P s control flow graph is traversed, and all possible values of the constituents of compound conditions are exercised at least once. Statement Coverage Edge Coverage Other Testing Criteria

49 Techniques Other Testing Criteria Path coverage criterion The following example, shows that the edge coverage criterion may fail to identify significant test cases. if x 0 then y : 5; else z : z x; end if; if z ą 1 then z : z{x; else z : 0; end if; (Slide 49 of 52) Statement Coverage Edge Coverage Other Testing Criteria

50 Techniques Other Testing Criteria the test set tă x 0, z 1 ą, ă x 1, z 3 ąu causes the execution of all edges but fails to show the risk of a division by zero. Such a risk would become evident by the test set tă x 0, z 3 ą, ă x 1, z 1 ąu, Path coverage criterion: Select a test set T such that, by executing P for each d in T, all paths leading from the initial to the final node of P s control flow graph are traversed. (Slide 50 of 52) Statement Coverage Edge Coverage Other Testing Criteria

51 Techniques (Slide 51 of 52) Other Testing Criteria we could suggest the following empirical guideline for testing loops: Look for conditions that execute loops (1) zero times (or a minimum number of times in special cases such as repeat-until loops) (2) a maximum number of items (3) an average number of times (according to some statistical criterion) when dealing with complex combinations of alternatives (several if or case statements), one should try to combine paths as much as possible Statement Coverage Edge Coverage Other Testing Criteria The major source of complexity is nested loops

52 References I (Slide 52 of 52) Statement Coverage Edge Coverage Other Testing Criteria