Introduction to Software Testing Chapter 3, Sec# 1 & 2 Logic Coverage

Transcription

1 Introduction to Software Testing Chapter 3, Sec# 1 & 2 Logic Coverage Paul Ammann & Jeff Offutt waretest/

2 Ch. 3 : Logic Coverage Four Structures for Modeling Software Graphs Logic Input Space Syntax Applied to Applied to Source FSMs Applied to Specs DNF Source Specs Source Models Design Use cases Integ Input Introduction to Software Testing (Ch 3) Ammann & Offutt 2

3 Covering Logic Expressions (3.1) Logic expressions show up in many situations Covering logic expressions is required by the US Federal Aviation Administration for safety critical software (FAA) Logical expressions can come from many sources Decisions in programs FSMs and statecharts Requirements Tests are intended to choose some subset of the total number of truth assignments to the expressions Introduction to Software Testing (Ch 3) Ammann & Offutt 3

4 Logic Predicates and Clauses A predicate is an expression that evaluates to a boolean value Predicates can contain boolean variables non-boolean variables that contain >, <, ==, >=, <=,!= boolean function calls Internal structure is created by logical operators : the negation operator : the and operator : the or operator : the implication operator : the exclusive or operator : the equivalence operator A clause is a predicate with no logical operators ((a < b) C) ((a >= b) p(x)) Introduction to Software Testing (Ch 3) Ammann & Offutt 4

5 Review: Truth Table for Logical Operators p q p q p q p q p q p q T T T T F T T T F F T T F F F T F T T T F F F F F F T T Introduction to Software Testing (Ch 3) Ammann & Offutt 5

6 Review: the Laws of Logic 1. Commutative Laws p q q p p q q p 2. Associative Laws (p q) r p (q r) (p q) r p (q r) 3. Distributive Laws p (q r) (p q) (p r) p (q r) (p q) ( p r) 4. Idempotent Laws p p p p p p 5. Identity Laws p F F p F p p T p p T T 6. Involution Law ( p) p 7. De Morgan s Laws (p q) ( p) ( q) (sometimes written p NOR q) (p q) ( p) ( q) (sometimes written p NAND q) 8. Complement Laws p p F p p T T F F T Introduction to Software Testing (Ch 3) Ammann & Offutt 6

7 Examples P = (a < b) f (z) D (m >= n*o) P is a predicate P has four clauses: 1. (a < b) relational expression 2. f (z) boolean-valued function 3. D boolean variable 4. (m >= n*o) relational expression Most predicates have few clauses It would be nice to quantify that claim!!! Sources of predicates Decisions in programs Guards in finite state machines Decisions in UML activity graphs Requirements, both formal and informal SQL queries (i.e where conditions) Introduction to Software Testing (Ch 3) Ammann & Offutt 7

8 Translating from English I am interested in SWE 637 and CS 652 course = swe637 OR course = cs652 Humans have trouble translating from English to Logic If you leave before 6:30 AM, take Braddock to 495, if you leave after 7:00 AM, take Prosperity to 50, then 50 to 495 time < 6:30 path = Braddock time > 7:00 path = Prosperity Hmm this is incomplete! time < 6:30 path = Braddock time >= 6:30 path = Prosperity Introduction to Software Testing (Ch 3) Ammann & Offutt 8

9 Source Code if ((gear == 5 gear == 6) && speed >= 100) print( highway ) else print( some other road ); P : (g = 5 g = 6) s >= 100 Introduction to Software Testing (Ch 3) Ammann & Offutt 9

10 Testing and Covering Predicates (3.2) We use predicates in testing as follows : Developing a model of the software as one or more predicates Requiring tests to satisfy some combination of clauses Abbreviations: P is the set of predicates p is a single predicate in P C is the set of clauses in P C p is the set of clauses in predicate p c is a single clause in C Introduction to Software Testing (Ch 3) Ammann & Offutt 10

11 Predicate and Clause Coverage The first (and simplest) two criteria require that each predicate and each clause be evaluated to both true and false Predicate Coverage (PC) : For each p in P, TR contains two requirements: p evaluates to true, and p evaluates to false. When predicates come from conditions on edges, this is equivalent to edge coverage PC does not evaluate all the clauses, so Clause Coverage (CC) : For each c in C, TR contains two requirements: c evaluates to true, and c evaluates to false. Introduction to Software Testing (Ch 3) Ammann & Offutt 11

12 Predicate Coverage Example ((a < b) D) (m >= n*o) predicate coverage Predicate = true a = 5, b = 10, D = true, m = 1, n = 1, o = 1 = (5 < 10) true (1 >= 1*1) = true true TRUE = true Predicate = false a = 10, b = 5, D = false, m = 1, n = 1, o = 1 =(10< 5) false (1>=1*1) = false false TRUE = false Introduction to Software Testing (Ch 3) Ammann & Offutt 12

13 Clause Coverage Example ((a < b) D) (m >= n*o) Clause coverage (a < b) = true (a < b) = false D = true D = false a = 5, b = 10 a = 10, b = 5 D = true D = false m >= n*o = true m = 1, n = 1, o = 1 m >= n*o = false m = 1, n = 2, o = 2 false cases true cases Two tests 1) a = 5, b = 10, D = true, m = 1, n = 1, o = 1 2) a = 10, b = 5, D = false, m = 1, n = 2, o = 2 Introduction to Software Testing (Ch 3) Ammann & Offutt 13

14 Problems with PC and CC a b p = a b 1 T T T 2 T F T 3 F T T 4 F F F Examples: T 23 : Satisfies CC but not PC, because p is never false T 24 : Satisfies PC but not CC, because b is never true T 14 : Satisfies both CC and PC coverage So, neither PC nor CC subsumes the other in all cases. PC does not fully exercise all the clauses, especially in the presence of short circuit evaluation CC does not always ensure PC That is, we can satisfy CC without causing the predicate to be both true and false This is definitely not what we want! The simplest solution is to test all combinations Introduction to Software Testing (Ch 3) Ammann & Offutt 14

15 Combinatorial Coverage CoC requires every possible combination Sometimes called Multiple Condition Coverage Combinatorial Coverage (CoC) : For each p in P, TR has test requirements for the clauses in C p to evaluate to each possible combination of truth values. a < b D m >= n*o P = ((a < b) D) (m >= n*o) 1 T T T T 2 T T F F 3 T F T T 4 T F F F 5 F T T T 6 F T F F 7 F F T F 8 F F F F Introduction to Software Testing (Ch 3) Ammann & Offutt 15

16 Combinatorial Coverage This is simple, neat, clean, and comprehensive But quite expensive! Requires 2 N tests, where N is the number of clauses Impractical for predicates with more than 3 or 4 clauses The literature has lots of suggestions The general idea is simple: Test each clause independently from the other clauses Getting the details right is hard What exactly does independently mean? The book presents this idea as making clauses active Introduction to Software Testing (Ch 3) Ammann & Offutt 16

17 Active Clauses Clause Coverage has a weakness : The values do not always make a difference Consider the first test for clause coverage, which caused each clause to be true: (5 < 10) true (1 >= 1*1) Only the last clause counts! To really test the results of a clause, the clause should be the determining factor in the value of the predicate Determination : A clause c i in predicate p, called the major clause that determines p iff the values of the remaining minor clauses c j are such that changing c i changes the value of p This is considered to make the clause active Introduction to Software Testing (Ch 3) Ammann & Offutt 17

18 Determining Predicates Determination, the conditions under which a clause influences the outcome of a predicate If you flip the clause, and the predicate changes value, then the clause determines the predicate. The major clause, c i, is the clause on which we are focusing. All of the other clauses c j, j!= i, are minor clauses. Typically, to satisfy a given criterion, each clause is treated in turn (One at a time, One by one) as a major clause. From the testing perspective, we would like to test each clause under circumstances where the clause determines the predicate Introduction to Software Testing (Ch 3) Ammann & Offutt 18

19 Determining Predicates p = a b if b = true, p is always true. So, if b = false, a determines p. if a = false, b determines p. p = a b if b = false, p is always false. So, if b = true, a determines p. if a = true, b determines p. Note: if we do not vary B under circumstances where B determines P, then we have no evidence that B is used correctly. Goal : Find tests for each clause when the clause determines the value of the predicate This is formalized in several criteria that have subtle, but very important, differences Introduction to Software Testing (Ch 3) Ammann & Offutt 19

20 Making Clauses Determine a Predicate Finding values for minor clauses c j is easy for simple predicates But how to find values for more complicated predicates? Definitional approach: p c=true is predicate p with every occurrence of c replaced by true p c=false is predicate p with every occurrence of c replaced by false To find values for the minor clauses, connect p c=true and p c=false with Exclusive OR p c = p c=true p c=false After solving, p c describes exactly the values needed for c to determine p Introduction to Software Testing (Ch 3) Ammann & Offutt 20

21 Examples p = a b p a = p a=true p a=false = (true b) (false b) = true b = b p = a b p a = p a=true p a=false = (true b) (false b) = b false = b p = a (b c) p a = p a=true p a=false = (true (b c)) (false (b c)) = true (b c) = (b c) = b c NOT b NOT c means either b or c can be false RACC requires the same choice for both values of a, CACC does not Introduction to Software Testing (Ch 3) Ammann & Offutt 21

22 A More Subtle Example p = ( a b ) ( a b) p a = p a =true p a=false = ((true b) (true b)) ((false b) (false b)) = (b b) false = true false = true p = ( a b ) ( a b) p b = p b =true p b=false = ((a true) (a true)) ((a false) (a false)) = (a false) (false a) = a a = false a always determines the value of this predicate b never determines the value b is irrelevant! Introduction to Software Testing (Ch 3) Ammann & Offutt 22

23 Repeated Variables The definitions in this chapter yield the same tests no matter how the predicate is expressed (a b) (c b) == (a c) b (a b) (b c) (a c) Only has 8 possible tests, not 64 Use the simplest form of the predicate, and ignore contradictory truth table assignments Introduction to Software Testing (Ch 3) Ammann & Offutt 23

24 Active Clause Coverage Active Clause Coverage (ACC) : For each p in P and each major clause c i in C p, choose minor clauses c j, j!= i, so that c i determines p. TR has two requirements for each c i : c i evaluates to true and c i evaluates to false. p = a b 1) a = true, b = false 2) a = false, b = false 3) a = false, b = true 4) a = false, b = false a is major clause; c i = a Duplicate b is major clause; c i = b This is a form of MCDC (Modified Condition/Decision Coverage), which is required by the FAA for safety critical software Ambiguity : Do the minor clauses have to have the same values when the major clause is true and false? Introduction to Software Testing (Ch 3) Ammann & Offutt 24

25 Resolving the Ambiguity p = a ( b c ) Major clause : a (or c i = a) a = true, b = false, c = true Is this allowed? a = false, b = false, c = false c = false Is the minor clauses c j need to have the same values when the major clause c i is true as when c i is false??? This question caused confusion among testers for years Considering this carefully leads to three separate criteria : 1. Minor clauses do not need to be the same (GACC) 2. Minor clauses do need to be (must be) the same (RACC) 3. Minor clauses force the predicate to become both true and false (CACC) Introduction to Software Testing (Ch 3) Ammann & Offutt 25

26 General Active Clause Coverage General Active Clause Coverage (GACC) : For each p in P and each major clause c i in C p, choose minor clauses c j, j!= i, so that c i determines p. TR has two requirements for each c i : c i evaluates to true and c i evaluates to false. Condition: The values chosen for the minor clauses c j do not need to be the same when c i is true as when c i is false, that is, c j (c i = true) = c j (c i = false) for all c j OR c j (c i = true)!= c j (c i = false) for all c j This is complicated! It is possible to satisfy GACC without satisfying predicate coverage (PC). i.e. p = a b We really want to cause predicates to be both true and false! Introduction to Software Testing (Ch 3) Ammann & Offutt 26

27 Restricted Active Clause Coverage Restricted Active Clause Coverage (RACC) : For each p in P and each major clause c i in C p, choose minor clauses c j, j!= i, so that c i determines p. TR has two requirements for each c i : c i evaluates to true and c i evaluates to false. Condition: The values chosen for the minor clauses c j must be the same when c i is true as when c i is false, that is, it is required that c j (c i = true) = c j (c i = false) for all c j. This has been a common interpretation by aviation developers RACC often leads to infeasible test requirements There is no logical reason for such a restriction Introduction to Software Testing (Ch 3) Ammann & Offutt 27

28 Correlated Active Clause Coverage Correlated Active Clause Coverage (CACC) : For each p in P and each major clause c i in C p, choose minor clauses c j, j!= i, so that c i determines p. TR has two requirements for each c i : c i evaluates to true and c i evaluates to false. Condition: The values chosen for the minor clauses c j must cause p to be true for one value of the major clause c i and false for the other, that is, it is required that p(c i = true)!= p(c i = false). A more recent interpretation Implicitly allows minor clauses to have different values Explicitly satisfies (subsumes) predicate coverage (PC) Introduction to Software Testing (Ch 3) Ammann & Offutt 28

29 Simple Example #1: GACC, RACC, CACC a b p = a b 1 T T T 2 T F F 3 F T F 4 F F T GACC: Remember: The values chosen for the minor clauses c j do not need to be the same when c i is true as when c i is false GACC for c i = a : T 14 = {TT, FF}; GACC for c i = b : T 14 = {TT, FF}, GACC = T 14 T 14 = {TT, FF}; This satisfies CC but not PC NOTE: For simple predicates, RACC and CACC might be the same. But, they sure differ for complex (bigger) predicates RACC: Remember: The values chosen for the minor clauses c j must be the same when c i is true as when c i is false RACC for c i = a : T 13 = {TT, FT} RACC for c i = b : T 12 = {TT, TF} RACC = T 13 T 12 = {TT, FT, TF}; This satisfies CC & PC CACC: Remember: The values chosen for the minor clauses c j must cause p to be true for one value of the major clause c i and false for the other CACC for c i = a : T 13 = {TT, FT}; CACC for c i = b : T 12 = {TT, TF} CACC = T 13 T 12 = {TT, FT, TF}; This satisfies CC & PC Introduction to Software Testing (Ch 3) Ammann & Offutt 29

30 Example #2 : CACC vs. RACC a b c p = a (b c) a b c p = a (b c) 1 T T T T 1 T T T T 2 T T F T 5 F T T F 3 T F T T 2 T T F T 5 F T T F 6 F T F F 6 F T F F 3 T F T T 7 F F T F 7 F F T F major clause major clause CACC can be satisfied by choosing any of rows 1, 2, 3 AND any of rows 5, 6, 7 a total of nine pairs TR = any pair of {1,2,3} X {5,6,7} RACC can only be satisfied by one of the three pairs above TR = any pair of { (1,5), (2,6), (3,7) } Introduction to Software Testing (Ch 3) Ammann & Offutt 30

31 Inactive Clause Coverage The active clause coverage (ACC) criteria ensures that major clauses do affect the predicates Inactive clause coverage (ICC) takes the opposite approach major clauses do not affect the predicates Inactive Clause Coverage (ICC) : For each p in P and each major clause c i in C p, choose minor clauses c j, j!= i, so that c i does not determine p. TR has four requirements for each c i : (1) c i evaluates to true with p true, (2) c i evaluates to false with p true, (3) c i evaluates to true with p false, and (4) c i evaluates to false with p false. Introduction to Software Testing (Ch 3) Ammann & Offutt 31

32 General and Restricted ICC ICC is unlike ACC, the notion of correlation is not relevant c i does not determine p, so cannot correlate with p, So there is no CICC Predicate coverage is always guaranteed General Inactive Clause Coverage (GICC) : For each p in P and each major clause c i in C p, choose minor clauses c j, j!= i, so that c i does not determine p. Condition: The values chosen for the minor clauses c j do not need to be the same when c i is true as when c i is false. In other words, The values chosen for the minor clauses c j may vary amongst the four cases. Restricted Inactive Clause Coverage (RICC) : For each p in P and each major clause c i in C p, choose minor clauses c j, j!= i, so that c i does not determine p. Condition: The values chosen for the minor clauses c j must be the same in cases (1) and (2), and the values chosen for the minor clauses c j must also be the same in cases (3) and (4). Introduction to Software Testing (Ch 3) Ammann & Offutt 32

33 Logic Coverage Criteria Subsumption Combinatorial Clause Coverage CoC Restricted Active Clause Coverage RACC Restricted Inactive Clause Coverage RICC Correlated Active Clause Coverage CACC General Inactive Clause Coverage GICC General Active Clause Coverage GACC Clause Coverage CC Predicate Coverage PC Introduction to Software Testing (Ch 3) Ammann & Offutt 33

34 Infeasibility Infeasibility is often a problem because clauses are sometimes related to one another. choosing the truth value for one clause may affect the truth value for another clause while ( i < n && a[i]!= 0 ) { do something to a[i] } Solution: simply ignore infeasible requirements, which usually does not affect the quality of the tests A butter solution: whenever it is possible, consider the counterparts of the requirements in a subsumed coverage criterion For example, if RACC coverage with respect to clause a in predicate p is infeasible (due to additional constraints between the clauses), but CACC coverage is feasible, then it makes sense to replace the infeasible RACC test requirements with the feasible CACC test requirements. Introduction to Software Testing (Ch 3) Ammann & Offutt 34

35 Infeasible Test Requirements Consider the predicate: (a > b b > c) c > a (a > b) = true, (b > c) = true, (c > a) = true is infeasible As with graph-based criteria, infeasible test requirements have to be recognized and ignored Recognizing infeasible test requirements is hard, and in general, undecidable Software testing is inexact (roughly) engineering, not science Introduction to Software Testing (Ch 3) Ammann & Offutt 35

36 Logic Coverage Summary Predicates are often very simple in practice, most have less than 3 clauses In fact, most predicates only have one clause! With only clause, PC is enough With 2 or 3 clauses, CoC is practical Advantages of ACC and ICC criteria significant for large predicates CoC is impractical for predicates with many clauses Control software often has many complicated predicates, with lots of clauses Question, why don t complexity metrics count the number of clauses in predicates?? Introduction to Software Testing (Ch 3) Ammann & Offutt 36