Verification Overview Testing Theory and Principles Testing in Practice. Verification. Miaoqing Huang University of Arkansas 1 / 80

Transcription

1 1 / 80 Verification Miaoqing Huang University of Arkansas

2 Outline 1 Verification Overview 2 Testing Theory and Principles Theoretical Foundations of Testing Empirical Testing Principles 3 Testing in Practice White-box Testing Statement coverage Edge coverage Condition coverage Path coverage Black-box Testing Testing driven by logic specification Syntax-driven testing Decision Table-based Testing Cause-effect Graph Technique Testing in the Large Module Testing Integration Testing 2 / 80

3 3 / 80 Need for verification Designers are fallible even if they are skilled and follow sound principles You are human!!! Everything must be verified, every required quality, process and products Even the verification process itself needs to be verified You have confidence after verification

4 4 / 80 Properties of verification May not be binary (OK, not OK) severity of defect is important some defects may be tolerated May be subjective or objective e.g., usability, maintainability Even implicit qualities should be verified e.g., robustness

5 5 / 80 Approaches to verification Experiment with behavior of product sample behaviors via testing goal is to find counterexamples dynamic technique Analyze product to deduce its adequacy analytic study of properties static technique

6 6 / 80 Lack of continuity in software testing Testing sample behaviors by examining test cases Impossible to extrapolate behavior of software from a finite set of test cases No continuity of behavior it can exhibit correct behavior in many cases, but may still be incorrect in some cases

7 7 / 80 Verification in traditional engineering Example of bridge design One test assures infinite correct situations

8 8 / 80 Goals for testing To show the presence of bugs, but never to show their absence If the results delivered by the system are different from the expected ones in just one case, this unequivocally shows that the system is incorrect By contrast, a correct behavior of the system on a finite number of cases does not guarantee the correctness in the general case Still, we need to do testing driven by sound and systematic principles

9 9 / 80 Goals for testing To show the presence of bugs, but never to show their absence If the results delivered by the system are different from the expected ones in just one case, this unequivocally shows that the system is incorrect By contrast, a correct behavior of the system on a finite number of cases does not guarantee the correctness in the general case Still, we need to do testing driven by sound and systematic principles read (x); read(y); if x==y then z=2; //should be z=22 else z=0; end if; write(z);

10 10 / 80 Goals for testing To show the presence of bugs, but never to show their absence If the results delivered by the system are different from the expected ones in just one case, this unequivocally shows that the system is incorrect By contrast, a correct behavior of the system on a finite number of cases does not guarantee the correctness in the general case Still, we need to do testing driven by sound and systematic principles read (x); read(y); if x==y then z=2; //should be z=22 else z=0; end if; write(z);

11 11 / 80 Goals for testing Help isolate errors facilitate debugging print out the location where the error takes place Be repeatable repeating the same experiment, we should get the same results if (x==0) then // x is not initialized write( abnormal ); else write( normal ); end if;

12 Outline 1 Verification Overview 2 Testing Theory and Principles Theoretical Foundations of Testing Empirical Testing Principles 3 Testing in Practice White-box Testing Statement coverage Edge coverage Condition coverage Path coverage Black-box Testing Testing driven by logic specification Syntax-driven testing Decision Table-based Testing Cause-effect Graph Technique Testing in the Large Module Testing Integration Testing 12 / 80

13 13 / 80 Definition I Definition P: the program D: the input domain R: the output domain OR: the requirement on output values of P as stated in P s specification P(d) is correct <d, P(d)> OR P is correct if all P(d)s are correct

14 14 / 80 Definition II FAILURE P(d) is not correct ERROR (DEFECT) Anything that may cause a failure Typing mistake Incomplete branches in if-else statement FAULT Incorrect intermediate state entered by program during runtime

15 15 / 80 Definition II FAILURE P(d) is not correct ERROR (DEFECT) Anything that may cause a failure Typing mistake Incomplete branches in if-else statement FAULT Incorrect intermediate state entered by program during runtime Their relations A FAILURE definitely demonstrates the existence of an ERROR An ERROR does not necessarily create a FAILURE A FAILURE occurs only if a FAULT happens during execution A FAULT occurs only if the program contains an ERROR The goal of testing

16 Definition II FAILURE P(d) is not correct ERROR (DEFECT) Anything that may cause a failure Typing mistake Incomplete branches in if-else statement FAULT Incorrect intermediate state entered by program during runtime Their relations A FAILURE definitely demonstrates the existence of an ERROR An ERROR does not necessarily create a FAILURE A FAILURE occurs only if a FAULT happens during execution A FAULT occurs only if the program contains an ERROR The goal of testing Select appropriate test cases to increase the likelihood that errors cause failures 16 / 80

17 17 / 80 Definition III Test case d An element of D, i.e., d D Test case d is successful if P(d) is correct Test set T A finite subset of D, i.e., T D Test set T is successful if P(d) is correct for all d T Ideal test set T Whenever P is incorrect, there exists a d T such that P(d) is incorrect

18 18 / 80 Test Selection Criterion I A test selection criterion C is a subset of 2 D 2 D denotes the set of all finite subsets of D Each element in C is a test set A test set T satisfies C if T C Example: for a program whose domain D is the set of integers C could require that test sets contain at least three elements: one negative, one zero, and one positive

19 19 / 80 Test Selection Criterion I A test selection criterion C is a subset of 2 D 2 D denotes the set of all finite subsets of D Each element in C is a test set A test set T satisfies C if T C Example: for a program whose domain D is the set of integers C could require that test sets contain at least three elements: one negative, one zero, and one positive C = {<x 1, x 2,... x n> n 3 and exists i, j, k (x i < 0, x j = 0, x k > 0)} <-5, 0, 22> <-10, 2, 8, 33, 0, -19> <1, 3, 99> x C is consistent for any pairs T 1, T 2 satisfying C, T 1 is successful iff T 2 is successful

20 20 / 80 Test Selection Criterion I A test selection criterion C is a subset of 2 D 2 D denotes the set of all finite subsets of D Each element in C is a test set A test set T satisfies C if T C Example: for a program whose domain D is the set of integers C could require that test sets contain at least three elements: one negative, one zero, and one positive C = {<x 1, x 2,... x n> n 3 and exists i, j, k (x i < 0, x j = 0, x k > 0)} <-5, 0, 22> <-10, 2, 8, 33, 0, -19> <1, 3, 99> x C is consistent for any pairs T 1, T 2 satisfying C, T 1 is successful iff T 2 is successful Theoretically, there is no need to make a specific choice of a test set among those satisfying C if C is consistent

21 21 / 80 Test Selection Criterion II C is complete Whenever P is incorrect, there is a test set T C that is not successful C is complete and consistent Identify an ideal test set Allow correctness to be proved! C 1 is finer than C 2 For any program P, for any T 1 C 1 there is a subset T 2 T 1, which satisfies C 2 C 1 is more restrictive than C 2 Example C 1 = {<x 1, x 2,... x n> n 3 and exists i, j, k, m, p(x i < 0, x j = 0, x k > 0, x m even and x p odd)} C 2 = {<x 1, x 2, x 3 > (x 1 < 0, x 2 = 0, x 3 > 0)} T 1 = T 2 = {-3, 0, 6} T 1 = {-3, 0, 5, 8}, T 2 = {-3, 0, 5}

22 22 / 80 Property of the criterion definition None is effective No algorithms are exist to state if a program, test set, or criterion has that property In particular, there is no algorithm to derive a test set that would prove program correctness There is no constructive criterion that is consistent and complete

23 Outline 1 Verification Overview 2 Testing Theory and Principles Theoretical Foundations of Testing Empirical Testing Principles 3 Testing in Practice White-box Testing Statement coverage Edge coverage Condition coverage Path coverage Black-box Testing Testing driven by logic specification Syntax-driven testing Decision Table-based Testing Cause-effect Graph Technique Testing in the Large Module Testing Integration Testing 23 / 80

24 24 / 80 Empirical Testing Principles We want to test a system exhaustively, but it is impossible Compromise between the impossible and the inadequate We try to find strategy to select significant test cases Significant = having high potential of uncovering presence of error Running a sufficient number of significant test cases instead of a large number of test cases

25 25 / 80 Empirical Testing Principles We want to test a system exhaustively, but it is impossible Compromise between the impossible and the inadequate We try to find strategy to select significant test cases Significant = having high potential of uncovering presence of error Running a sufficient number of significant test cases instead of a large number of test cases if x>y then max = x; else max = x; end if; 1 {x=3, y=2}, {x=2, y=3} 2 {x=3, y=2}, {x=4, y=3}, {x=5, y=1}

26 26 / 80 Complete coverage principle An intuitive introduction 1 Given a system, find all the possible test cases in the domain

27 27 / 80 Complete coverage principle An intuitive introduction 1 Given a system, find all the possible test cases in the domain 2 Analyze these test cases and figure out their properties or behaviors

28 28 / 80 Complete coverage principle An intuitive introduction 1 Given a system, find all the possible test cases in the domain 2 Analyze these test cases and figure out their properties or behaviors 3 Group test cases that behave in approximately the same way into the sub-domain

29 Complete coverage principle An intuitive introduction 1 Given a system, find all the possible test cases in the domain 2 Analyze these test cases and figure out their properties or behaviors 3 Group test cases that behave in approximately the same way into the sub-domain 4 Select one test case from each sub-domain to compose a test set 29 / 80

30 Complete coverage principle An example Example Compute the factorial of any number If n < 0, then an appropriate error message must be printed. If 0 n < 20, then the exact value of n! must be printed. If 20 n 200, then an approximate value of n! must be printed in floating-point format (e.g., using some approximate method of numerical calculus). The admission error is 0.1% of the exact value. Finally, if n > 200, the input can be rejected by printing an appropriate error message. {n < 0},{0 n < 20},{20 n 200},{n > 200} {-10,5,175,290} The assertion that the program will behave correctly for any other value is a reasonable expectation 30 / 80

31 31 / 80 Complete coverage principle The theory Try to group elements of D into sub-domains D 1, D 2,..., D n where any element of each D i is likely to have similar behavior D = D 1 D 2 D n Satisfy the complete coverage principle Select one test case as a representative of the sub-domain from each sub-domain

32 32 / 80 Complete coverage principle The theory Try to group elements of D into sub-domains D 1, D 2,..., D n where any element of each D i is likely to have similar behavior D = D 1 D 2 D n Satisfy the complete coverage principle Select one test case as a representative of the sub-domain from each sub-domain If D i D j, a test case in D i D j exercises both D i and D j D 1 = {d i d i mod 2 = 0} D 2 = {d i d i 0} D 3 = {d i d i mod 2 0} D 1 D 2 and D 2 D 3 {-48, -37}

33 33 / 80 Testing in small and Testing in large Testing in small Address the testing of individual modules Testing in large Address the issue of decomposing and organizing the testing activity according to the modular structure of complex programs

34 34 / 80 Two approaches White-box testing Test what the program does Structural testing Implementation-based testing Partition the test domain based on the module s internal code Example, the {x > y} and {x y} case Black-box testing Test what the program is supposed to do Functional testing Specification-based testing Partition the test domain based on the module s specification Example, test the program to compute the factorial n!

35 Outline 1 Verification Overview 2 Testing Theory and Principles Theoretical Foundations of Testing Empirical Testing Principles 3 Testing in Practice White-box Testing Statement coverage Edge coverage Condition coverage Path coverage Black-box Testing Testing driven by logic specification Syntax-driven testing Decision Table-based Testing Cause-effect Graph Technique Testing in the Large Module Testing Integration Testing 35 / 80

36 36 / 80 Structural Coverage Testing Derive test cases from program code Only test what have been implemented Adequacy criteria If significant parts of program structure are not tested, testing is inadequate Testing strategies Statement coverage Edge coverage Condition coverage Path coverage

37 37 / 80 Statement coverage Statement-coverage Criterion Select a test set T such that every elementary statement in P is executed at least once by some d in T Elementary statement A statement that does not produce any recursive occurrence of itself Example: assignment statements, I/O statements, procedure calls Try to minimize the number of test cases still preserving the desired coverage

38 38 / 80 Example read(x); read(y); if x > 0 then write( 1 ); else write( 2 ); end if; if y > 0 then write( 3 ); else write( 4 ); end if; Let W 1, W 2, W 3, W 4 represent the 4 write statements Let D i denote the class of input values that causes execution of W i, i = 1,..., 4. D 1 = {x > 0} D 2 = {x 0} D 3 = {y > 0} D 4 = {y 0} {<x = 2, y = 3>, <x = - 13, y = 51>, <x = 97, y = 17>, <x = - 1, y = - 1>} covers all statements

39 39 / 80 Example read(x); read(y); if x > 0 then write( 1 ); else write( 2 ); end if; if y > 0 then write( 3 ); else write( 4 ); end if; Let W 1, W 2, W 3, W 4 represent the 4 write statements Let D i denote the class of input values that causes execution of W i, i = 1,..., 4. D 1 = {x > 0} D 2 = {x 0} D 3 = {y > 0} D 4 = {y 0} {<x = 2, y = 3>, <x = - 13, y = 51>, <x = 97, y = 17>, <x = - 1, y = - 1>} covers all statements {<x = - 13, y = 51>, <x = 2, y = - 3>} is minimal

40 40 / 80 Edge coverage Edge-coverage Criterion Select a test set T such that every edge (branch) of the control flow is exercised at least once by some d in T Notation in control graph Edges represent statements Nodes at the ends of an edge represent entry into and exit from of the statement Use node to connect statements

41 41 / 80 Control graph construction rules A single Statement Statement Block if-then-else if-then while Sequential statements

42 42 / 80 Simplification A sequence of edges can be collapsed into just one edge k-1 k 1 k

43 43 / 80 Example: Euclid s algorithm read(x); read(y); while x y loop if x > y then x = x - y; else y = y - x; end if; end loop; gcd = x;

44 44 / 80 Condition coverage Motivation found = false; counter = 1; while (not found) and counter < number_of_items loop if table (counter) == desired_element then found = true; end if; counter = counter + 1; end loop; if found then write ( the desired element is in the table ); else write ( the desired element is not in the table ); end if;

45 45 / 80 Condition coverage Motivation found = false; counter = 1; while (not found) and counter < number_of_items loop if table (counter) == desired_element then found = true; end if; counter = counter + 1; end loop; if found then write ( the desired element is in the table ); else write ( the desired element is not in the table ); end if; Only covering all edges is not going to discover the error

46 46 / 80 Condition coverage Condition-coverage Criterion Select a test set T such that every edge of P s control flow is traversed and all possible values of the constituents of compound conditions are exercised at least once Compound Boolean expression C = c 1 and c 2 or c 3... It is finer than edge coverage

47 47 / 80 Condition coverage Example if c1 and c2 then st; else sf; end if; if c1 then if c2 then st; else sf; end if; else sf; end if;

48 48 / 80 Path coverage Motivation if x 0 then y = 5; else z = z - x; end if; if z > 1 then z = z / x; else z = 0; end if; {<x = 0, z = 1>, <x = 1, z = 3>} covers all edges Fail to expose the risk of a division by zero

49 49 / 80 Path coverage Motivation if x 0 then y = 5; else z = z - x; end if; if z > 1 then z = z / x; else z = 0; end if; {<x = 0, z = 1>, <x = 1, z = 3>} covers all edges Fail to expose the risk of a division by zero {<x = 0, z = 3>, <x = 1, z = 1>} would work

50 50 / 80 Path coverage Path-coverage Criterion Select a test set T that traverses all paths from the initial to the final node of P s control flow It is finer than edge coveragr Number of paths may be too large, or even infinite (see while loops) Empirical guideline for testing loops Zero times or a minimum number of times A maximum number of times An average number of times

51 Outline 1 Verification Overview 2 Testing Theory and Principles Theoretical Foundations of Testing Empirical Testing Principles 3 Testing in Practice White-box Testing Statement coverage Edge coverage Condition coverage Path coverage Black-box Testing Testing driven by logic specification Syntax-driven testing Decision Table-based Testing Cause-effect Graph Technique Testing in the Large Module Testing Integration Testing 51 / 80

52 52 / 80 An intuitive example Specification The program receives input as a record describing an invoice The invoice must be inserted into a file of invoices sorted by date If other invoices exist in the file with the same date, then the invoice should be inserted after the last one Some consistency checks must be performed The program should verify whether the customer is already in a corresponding file of customers Whether the customer s data in the two files match, etc.

53 53 / 80 An intuitive example Specification The program receives input as a record describing an invoice The invoice must be inserted into a file of invoices sorted by date If other invoices exist in the file with the same date, then the invoice should be inserted after the last one Some consistency checks must be performed The program should verify whether the customer is already in a corresponding file of customers Whether the customer s data in the two files match, etc. Possible test cases (informal test procedure) An invoice whose date is the current date An invoice whose date is before the current date An invoice whose date is the same as some existing invoice An invoice whose date does not exist in any previous invoice Several incorrect invoices, checking different types of inconsistencies

54 54 / 80 Strategies Informal Decompose the specifications into a set of classes Give at least one test case for each class Formal Testing driven by logic specification Syntax-driven testing The input is formally described by a grammar Decision table-based testing A specification is described by a decision table Cause-effect graphs

55 55 / 80 An example Specification The program receives input as a record describing an invoice The invoice must be inserted into a file of invoices that is sorted by date If other invoices exist in the file with the same date, then the invoice should be inserted after the last one Some consistency checks must be performed The program should verify whether the customer is already in a corresponding file of customers Whether the customer s data in the two files match, etc.

56 56 / 80 Definition Let x be a variable of type invoice, (i.e., tuple <x.customer, x.amount, x.data,...>, where customer, amount are given Let current_date denote a variable of type date storing the value of the day when the insert operation is executed Let f be a variable of type invoice_file (i.e., a sequence of elements of type invoice). f(i) denotes the ith element of f. Given a generic record x and a file f of such records (with no repeated elements), let pos be a function that returns the position i of x in f. pos(x,f)=i f(i)=x sorted_by_date(f) forall i,j in Integers (i<j implies f(i).date f(j).date)

57 57 / 80 The logic specification for all x in Invoices, f in Invoice_Files {sorted_by_date(f) and not exist j,k (j k and f(j)=f(k)}

58 58 / 80 The logic specification for all x in Invoices, f in Invoice_Files {sorted_by_date(f) and not exist j,k (j k and f(j)=f(k)} insert(x,f)

59 59 / 80 The logic specification for all x in Invoices, f in Invoice_Files {sorted_by_date(f) and not exist j,k (j k and f(j)=f(k)} insert(x,f) {sorted_by_date(f) and for all k,z (old_f(k) = z implies exists j (f(j)=z)) and for all k,z (f(k)=z and z x) implies exists j (old_f(j)=z) and exists j (f(j).date=x.date and f(j) x) implies j<pos(x,f) and result (x.customer belongs_to custome_file and...) and warning (x belongs_to old_f or x.data > current_date or...) } The postcondition as the conjunction of several clauses, each characterized by some condition.

60 60 / 80 The conditions TRUE implies sorted_by_data(f) and for all k,z (old_f(k) = z implies exists j (f(j)=z)) and for all k,z (f(k)=z and z x) implies exists j (old_f(j)=z)

61 61 / 80 The conditions TRUE implies sorted_by_data(f) and for all k,z (old_f(k) = z implies exists j (f(j)=z)) and for all k,z (f(k)=z and z x) implies exists j (old_f(j)=z) and (x.customer belongs_to custome_file and...) implies result

62 62 / 80 The conditions TRUE implies sorted_by_data(f) and for all k,z (old_f(k) = z implies exists j (f(j)=z)) and for all k,z (f(k)=z and z x) implies exists j (old_f(j)=z) and (x.customer belongs_to custome_file and...) implies result and not (x.customer belongs_to custome_file and...) implies not result

63 63 / 80 The conditions TRUE implies sorted_by_data(f) and for all k,z (old_f(k) = z implies exists j (f(j)=z)) and for all k,z (f(k)=z and z x) implies exists j (old_f(j)=z) and (x.customer belongs_to custome_file and...) implies result and not (x.customer belongs_to custome_file and...) implies not result and x belongs_to old_f implies warning

64 64 / 80 The conditions TRUE implies sorted_by_data(f) and for all k,z (old_f(k) = z implies exists j (f(j)=z)) and for all k,z (f(k)=z and z x) implies exists j (old_f(j)=z) and (x.customer belongs_to custome_file and...) implies result and not (x.customer belongs_to custome_file and...) implies not result and x belongs_to old_f implies warning and x.data > current_date implies warning and...

65 65 / 80 Apply condition-coverage criterion to produce the test set Any test case can be used to verify that the file, which is produced, contains all and only previous invoices plus the new one and is sorted We need at least one test case with an invoice whose field customer exists in the customer_file, and one test case with an invoice whose field customer does not exist in such file

66 66 / 80 Use syntax-driven testing to verify a compiler An interpreter of simple arithmetic expression (in BNF grammar) <expression> ::= <expression>+<term> <expression>-<term> <term> <term> ::= <term> <factor> <term>/<factor> <factor> <factor> ::= ident (<expression>)

67 67 / 80 Use syntax-driven testing to verify a compiler An interpreter of simple arithmetic expression (in BNF grammar) <expression> ::= <expression>+<term> <expression>-<term> <term> <term> ::= <term> <factor> <term>/<factor> <factor> <factor> ::= ident (<expression>) Possible test cases <expression> = <expression>+<term> = <expression>+<term> <factor> = <term>+<term> <factor> = <factor>+<factor> <factor> = ident+ident ident

68 68 / 80 Decision table-based testing Example A word processor Three different formats plain text (p) boldface (b) italics (i) Five different commands make text plain (P) make text boldface (B) make text italics (I) emphasize (E) E can be either B or I superemphasize (SE) SE can be B, I or both (B+I)

69 69 / 80 The decision table Rows represent conditions Columns represent rules (i.e., results) Apply complete-coverage criterion to exercise each column

70 70 / 80 The decision table Rows represent conditions Columns represent rules (i.e., results) Apply complete-coverage criterion to exercise each column The number of column may go exponentially n conditions may generate 2 n results Need to select a significant subset of all possible input classes

71 71 / 80 Cause-effect graph technique Formally structure complex input-output specification using cause-effect graphs Steps 1 Transform inputs and outputs into Boolean values 2 Build the Boolean functions between inputs and outputs

72 72 / 80 Cause-effect graph technique Formally structure complex input-output specification using cause-effect graphs Steps 1 Transform inputs and outputs into Boolean values 2 Build the Boolean functions between inputs and outputs Example Transform inputs to a 10-tuple of Boolean {P, B, I, E, E=B, E=I, SE, SE=B, SE=I, SE=B+I} <false,true,false,false,false,false,false,false,false,false> represents command B Transform outputs of a 3-tuple of Boolean {p, b, i} <false,true,false> denotes a boldface text output

73 73 / 80 The cause-effect graph for boldface text output

74 74 / 80 Put constraints among inputs and outputs Inputs are not independent in many cases, same as outputs The output text can not be both plain and boldface at the same time

75 75 / 80 The cause-effect graph with constraints m m m m

76 Outline 1 Verification Overview 2 Testing Theory and Principles Theoretical Foundations of Testing Empirical Testing Principles 3 Testing in Practice White-box Testing Statement coverage Edge coverage Condition coverage Path coverage Black-box Testing Testing driven by logic specification Syntax-driven testing Decision Table-based Testing Cause-effect Graph Technique Testing in the Large Module Testing Integration Testing 76 / 80

77 77 / 80 Testing in the large Module testing testing a single module Integration testing integration of modules and subsystems System testing testing the entire system Acceptance testing performed by the customer

78 78 / 80 Build scaffolding STUB Call Module Under Test Call DRIVER Access to nonlocal variables Scaffolding needed to create the environment in which the module should be tested stubs modules used by the module under test driver module activating the module under test

79 79 / 80 Build scaffolding STUB Call Module Under Test Call DRIVER Access to nonlocal variables Scaffolding needed to create the environment in which the module should be tested stubs modules used by the module under test driver module activating the module under test How to build the scaffolding Develop simplified calling modules and called modules

80 80 / 80 Integration testing Big-bang approach first test individual modules in isolation then test integrated system Incremental approach modules are progressively integrated and tested can proceed both top-down and bottom-up according to the USES relation Top-down only stubs are needed Bottom-up only drivers are needed