Logic-based test coverage

Size: px

Start display at page:

Download "Logic-based test coverage"

Ernest Boyd
5 years ago
Views:

1 Logic-based test coverage!!! Basic approach Clauses and predicates Basic coverage criteria: CC, PC, CoC Structural logic coverage of source code Logic coverage of specifications Active clause coverage criteria (GACC, CACC, RACC)! Verificação e Validação de Software! Departamento de Informática! Faculdade de Ciências da Universidade de Lisboa!! Eduardo Marques, edrdo@di.fc.ul.pt

2 Logic Coverage Logic coverage approach Test criteria are based on logical expressions found in the specification or code of the SUT. Predicate An expression that evaluates to a boolean value (true or false) and may contain boolean-valued expressions connected by logical operators. Clause A boolean expression that does not make use of logical operators. Example: x > 10 (f(y) = 30 z ) is a predicate x > 10, f(y) = 30, and z are clauses and are logical operators. 2

3 Logical operators a b (and) a b (or) a (negation) a b (implication) a b (equivalence: a if and only if b ) a b (exclusive or/choice, also known as xor ) 3

4 Coverage criteria (PC, CC, CoC) Let P be a set of predicates and C the set of clauses that appear in P. For each p P let C p = { c C c occurs in p } Predicate coverage (PC) For every predicate p P, TR contains two requirements: (1) p evaluates to true, and (2) p evaluates to false. Clause coverage (CC) For every clause c C, TR contains two requirements: (1) c evaluates to true and (2) c evaluates to false. Combinatorial test coverage (CoC) Fo every p P TR contains test requirements for each possible combination of truth values of clauses in C p 4

5 Example For p1 = x > y (x = z-1 x > z ) the clauses are: a: x > y b: x = z-1 c: x > z For p2 = z > 0 z > x+y the clauses are d: z > 0 e: z > x+y For P = { p1, p2 } we have TR(PC) = { p1, p1, p2, p2 } TR(CC) = { a, a, b, b, c, c, d, d, e, e } TR(CoC) = { a b c, a b c, a b c, a b c, a b c, a b c, a b c, a b c, d e, d e, d e, d e } Exercise 1: find combinations of values for x, y, z that will satisfy the test requirements of PC, CC and CoC in turn. Observe that there are infeasible requirements for CoC. 5

6 Subsumption relations PC does not subsume CC, nor CC subsumes PC; for instance consider P = { a b } T1 = { (a = true, b = false), (a = false, b = false) } covers TR(PC) but not TR(CC) [does not cover case where b is true] T2 = { (a = true, b = false), (a = false, b = true) } covers TR(CC) but not TR(PC) CoC subsumes CC and PC of course (it covers all possible combinations) but it easily becomes impractical... 2 n test requirements are required per predicate with n independent clauses 6

7 Structural logical coverage for source code Aim: to identify test requirements (and then test cases for them) according to predicates and clauses found in source code. Clauses and predicates: identified in connection with logical operators defined by the programming language at stake. Reachability predicates: need to be derived to ensure proper evaluation of each predicate (in line with program flow) / derive test inputs. 7

8 Logical operators in source code logical expression Java expression a b a b a && b a b a! a a b a b a? b : true; a == b a b a!= b 8 Note: the & ^ ~ operators also correspond to logical operators [what are the differences between a && b and a & b, a b and a b? ]

9 Example: predicates/clauses in source code public static int daysinmonth(int m, int y) {! if (m <= 0 m > 12)! throw new IllegalArgumentException("Invalid month: " + m);! if (m == 2) {! if (y % 400 == 0 (y % 4 == 0 && y % 100!= 0))! return 29;! else! return 28;! }! } if (m <= 7) {! if (m % 2 == 1)! return 31;! return 30;! }! if (m % 2 == 0)! return 31;! return 30;! Predicates and clauses p1: c1 c2 ; c1: m <= 0, c2: m > 12! p2: c3; c3: m == 2;! p3: c4 (c5 && c6) ; c4: y % 400 == 0, c5: y % 4 == 0, c6: y % 100!= 0;! p4: c7; c7: m <= 7;! p5: c8; c8: m % 2 == 1;! p6: c9; c9: m % 2 == 0 9

10 Exercise 2: TR(CC), TR(PC), TR(CoC)? public static int daysinmonth(int m, int y) {! if (m <= 0 m > 12)! throw new IllegalArgumentException("Invalid month: " + m);! if (m == 2) {! if (y % 400 == 0 (y % 4 == 0 && y % 100!= 0))! return 29;! else! return 28;! }! } if (m <= 7) {! if (m % 2 == 1)! return 31;! return 30;! }! if (m % 2 == 0)! return 31;! return 30;! Predicates and clauses p1: c1 c2 ; c1: m <= 0, c2: m > 12! p2: c3; c3: m == 2;! p3: c4 (c5 && c6) ; c4: y % 400 == 0, c5: y % 4 == 0, c6: y % 100!= 0;! p4: c7; c7: m <= 7;! p5: c8; c8: m % 2 == 1;! p6: c9; c9: m % 2 == 0 10

11 Exercise 2 (solution) Predicates and clauses p1: c1 c2 ; c1: m <= 0, c2: m > 12! p2: c3; c3: m == 2;! p3: c4 (c5 && c6) ; c4: y % 400 == 0, c5: y % 4 == 0, c6: y % 100!= 0;! p4: c7; c7: m <= 7;! p5: c8; c8: m % 2 == 1;! p6: c9; c9: m % 2 == 0 TR(PC) = { p1, p1, p2, p2,, p6, p6 } TR(CC) = { c1, c1, c2, c2,, c9, c9 } TR(CoC) = { c1 c2, c1 c2, c1 c2, c1 c2, c4 c5 c6,, c4 c5 c6, same as CC for c3, c7,c8,c9 } 11

12 Reachability predicates public static int daysinmonth(int m, int y) {! if (m <= 0 m > 12) // p1! throw new IllegalArgumentException("Invalid month: " + m);! if (m == 2) { // p2! if (y % 400 == 0 (y % 4 == 0 && y % 100!= 0)) // p3! return 29;! Reachability predicates - r(p) 12 } else! return 28;! }! if (m <= 7) { // p4! if (m % 2 == 1) // p5! return 31;! return 30;! }! if (m % 2 == 0) // p6! return 31;! return 30;! r(p1) = true (always reached)! r(p2) = p1 = m > 0 m <= 12! r(p3) = r(p2) p2 = (m > 0 m <= 12 m == 2) = (m == 2)! r(p4) = r(p2) p2 = (m > 0 m <= 12 m!= 2)! r(p5) = r(p4) p4 = (m > 0 m <= 12 m!= 2) m <= 7! r(p6) = r(p4) p4 = (m > 0 m <= 12 m!= 2) m > 7 A reachability predicate r(p) expresses the logical conditions for p to be evaluated.

13 Exercise 3 public static int daysinmonth(int m, int y) {! if (m <= 0 m > 12) // p1! throw new IllegalArgumentException("Invalid month: " + m);! if (m == 2) { // p2! if (y % 400 == 0 (y % 4 == 0 && y % 100!= 0)) // p3! return 29;! } else! return 28;! }! if (m <= 7) { // p4! if (m % 2 == 1) // p5! return 31;! return 30;! }! if (m % 2 == 0) // p6! return 31;! return 30;! Reachability predicates - r(p) r(p1) = true (always reached)! r(p2) = p1 = m >= 0 m < 12! r(p3) = r(p2) p2 = (m > 0 m < 12 m == 2) = (m == 2)! r(p4) = r(p2) p2! r(p5) = r(p4) p4 = (m > 0 m < 12 m!= 2) m <= 7! r(p6) = r(p4) p4 = (m > 0 m < 12 m!= 2) m > 7 13 Identify test cases that satisfy PC, CC, CoC (in this order). Are there infeasible requirements?

14 Logic coverage of specifications Specifications of a software may also be subject to logical test coverage. For instance these may take the form of: Contracts: informal (e.g., Javadoc) or formal (e.g., JML) FSM abstractions 14

15 Example - a JML contract for Time.tick() public Time(int h, int m) { }! public int gethours() { }! public int getminutes() { }!! public requires getminutes() < ensures getminutes() == \old(getminutes()) + ensures gethours() == public requires getminutes() == 59 && gethours() < ensures getminutes() == ensures gethours() == \old(gethours()) + public requires getminutes() == 59 && gethours() == ensures getminutes() == ensures gethours() == public void tick() { } JML pre-conditions define the predicates of interest! 3 test cases will satisfy PC (and CC too). Identify them. 15

16 Example 2 - BoundedQueue.enqueue() /*@! public normal_behavior! requires!isfull();! ensures size() == \old(size()) + 1;! ensures elementat(\old(size())) == data;! ensures (\forall int i; i >= 0 && i < \old(size())! ==> elementat(i) == \old(elementat(i)));! also! public exceptional_behavior! requires isfull();! signals_only IllegalStateException;! */! void enqueue(e data) throws IllegalStateException;! We need to cover predicate /pre-condition isfull() See complete specification online and implementing class (BoundedQueue / BoundedArrayQueue). 16

17 Predicate determination PC and CC do not subsume one another and CoC may easily become unpractical or lead to too many infeasible requirements. How to ensure that clause-level test criteria also do have an effect on the predicate at stake? We will introduce the notion of determination: the conditions under which a clause solely determines the outcome of a predicate. For p = a ( b c ) the determination predicates are: d(a) = b c a determines p when b c! d(b) = a c b determines p when a c! d(c) = a b c determines p when a b 17

18 Determination (more formally) Determination predicate Let p P and c C p. We say that c determines p if there is a logical assignment (determination predicate) d(c) to all other clauses s.t. changing the value of c changes the value of p. Major and minor clauses [terminology] For a given c we are finding d(c) we call c the major clause and all other clauses the minor clauses. Finding the determination predicate d(c) = p [T / c ] p [F / c ] where p [X / c ] stands for p with every occurrence of the major clause c replaced by X [See example in the next slide] 18

19 Deriving determination predicates Finding the determination predicate d(c) = p [T / c ] p [F / c ] where p [X / c ] stands for p with every occurrence of the major clause c replaced by X Example 1 - taking p = a (b c)! d(a) = p [T / a ] p [F / a ] = (b c) F = b c! d(b) = p [T / b ] p [F / b ] = a (a c) = a c! d(c) = p [T / c ] p [F / c ] = a (a b) = a b! Example 2 - taking p = a (b c)! d(a) = p [T / a ] p [F / a ] = T (b c) = (b c) = b c! d(b) = p [T / b ] p [F / b ] = (a c ) a = a c! d(c) = p [T / c ] p [F / c ] = (a b ) a = a b 19

20 General Active Clause Coverage GACC criterion - For p P and c C p include two requirements in TR: (1) c d(c) (2) c d(c) Example: 2 predicates involving 5 clauses giving us 10 test requirements P= { p1, p2 } p1 = a ( b c ) p2 = x y TR(GACC) = { a d(a), a d(a), b d(b), b d(b), c d(c), c d(c), x d(x), x d(x), y d(y), y d(y) } d(a) = b c, d(b) = a c, d(c) = a b d(x) = y d(y) = x 20

21 GACC and subsumption of CC/PC By definition GACC subsumes CC... but not PC (though this may happen in many practical cases of interest) Example: for p = a b we have d(a) = T and d(b) = T... so TR(GACC) = { a, a, b, b } [Obs.: equivalent to TR(CC)] T = { [a = T, b=t ], [a=f, b=f] } satisfies GACC for p = a b but not PC. Both assignments to a and b yield p = T. T = { [a = T, b=f ], [a=f, b=t] } would also satisfy GACC but not PC. Both assignments to a and b yield p = F. Does GACC subsume PC? Not necessarily 21

22 Correlated Active Clause Coverage CACC criterion - For p P and c C p include two requirements in TR: (1) c d(c) (p = X) (2) c d(c) (p = X) where X { T, F}, that is, p must evaluate to T in one case and F in the other. By definition CACC subsumes GACC [thus CC] but also PC. In the previous slide s example: p = a b where d(a) = T and d(b) = T TR(CACC) = { a (p=x), a (p= X), b (p=y), b (p= Y) } T = { [a = T, b=t ], PC). [a=f, b=t], [a = T, b=f ] } satisfies CACC (and 22

23 GACC vs CACC (example) t a b c p = a (b c) Satisfies 1 T T T T a d(a) b d(b) c d(c) 2 T T F F b d(b) c d(c) 3 T F T F b d(b) c d(c) 4 T F F T a d(a) b d(b) c d(c) 5 F T T F a d(a) 6 F T F F 7 F F T F Determination!! d(a) = b c!! d(b) = a!! d(c) = a 8 F F F F a d(a) GACC is satisfied by { (1), (4), (5) } for instance. This choice of assignments will not cover the CACC requirements for b and c for the case where p must be false. CACC: can be satisfied by { (1), (2), (3), (5) }. Exercise: perform a similar analysis for p = a (b c). 23

24 Restricted Active Clause Coverage RACC criterion - For p P and c C p include two requirements in TR: (1) c p c (p = X) (2) c p c (p = X) where X { T, F}, that is, p must evaluate to T in one case and F in the other (as in CACC) but additionally: the minor clause assignments must be the same in both cases. Obs.: RACC subsumes CACC by definition. RACC imposes more uniform tests, but is also more likely to imply infeasible requirements. 24

25 CACC vs RACC (example) t a b c p = a (b c) Satisfies 1 T T T T a d(a) 2 T T F T a d(a) d(a) = b c 3 T F T T a d(a) 4 T F F F 5 F T T F a d(a) 6 F T F F a d(a) 7 F F T F a d(a) 8 F F F F 25 CACC coverage for a - 9 possible choices: (1), (2) or (3) combined with one of (5), (6), or (7). RACC coverage for a - only 3 possible choices: (1) combined with (5); (2) and (6); (3) and (7).

26 Example: isleapyear() public static boolean isleapyear(int y) {! return y % 400 == 0 (y % 4 == 0 && y % 100!= 0);! } a: y % 400 == 0 b: y % 4 == 0 c: y % 100!= 0! p: a (b c)!! d(a) = b c d(b) = a c d(c) = a b TR(GACC)={ (1) a ( b c), (2) a ( b c),! (3) b a c, (4) b a c,! c a b /* duplicate of (3) */, (5) c a b } y exp. value logical clause! values covered GACC requirements t true a b c (1) a ( b c) t false a b c t false a b c (2) a ( b c)! (4) b a c! (2) a ( b c)! (5) c a b 26 t true a b c (3) b a c

27 isleapyear() analysis public static boolean isleapyear(int y) {! return y % 400 == 0 (y % 4 == 0 && y % 100!= 0);! } y exp. value logical clause! values covered GACC requirements t true a b c (1) a ( b c) t false a b c t false a b c (2) a ( b c)! (4) b a c! (2) a ( b c)! (5) c a b t true a b c (3) b a c GACC coverage implies CACC coverage in this case RACC is also satisfied: (t1,t4) pair for a; (t2,t4) for b; (t3,t4) for c {t1, t2} satisfies CC and PC {t1, t3} satisfies PC but not CC CoC would lead to the following infeasible requirements 27 a b c, a b c, a b c, a b c

28 Exercise 4 public static TClass triangletype(int a, int b, int c) {! if (a <= 0 b <= 0 c <= 0) /* p1 */! return INVALID;! if (a >= b + c b >= a+c c >= a+b) /* p2 */! return INVALID;! } int count = 0;! if (a == b) /* p3 */! count++;! if (a == c) /* p4 */! count++;! if (b == c) /* p5 */! count++;! if (count == 0) /* p6 */! return SCALENE;! if (count == 1) /* p7 */! return ISOSCELES;! return EQUILATERAL;! Identify: 1) the reachability predicates; 2) TR(CC) and TR(PC) 3) test cases that satisfy a) PC b) CC 4) determination predicates for the clauses of p1 and p2 5) TR(CACC) 28 6) test cases that satisfy a) CACC b) RACC [are there infeasible requirements?]

Logic-based test coverage

Eduardo Marques, Vasco Thudichum Vasconcelos Logic-based test coverage Basic approach Clauses and predicates Basic coverage criteria: CC, PC, CoC Structural logic coverage of source code Logic coverage