COMP 3705 Advanced Software Engineering: Software Testing Prof. Matt Rutherford For Next Week Readings For Midterm: Chapter 1 and Sections 2.1-2.6, 3.1 3.5 New material: Chapter 4 Homework #5 http://mjrutherford.org/teaching/2009/winter/comp3705/homeworks 1
Today Inactive Clause Coverage Logic Coverage of Code, Specifications and FSMs Review Administrative Items Handwritten Homework Handwritten homework is OK, but you need to take extra care to make sure it is neat and well-organized Suggestion: draw graphs / figures by hand (neatly!!) and type the solutions Common sense: do not make it hard for the grader to give you partial credit 2
Administrative Items Office Hours: Nobody takes advantage of the before class office hours from 4-5pm. From now on, if you want to meet me before class, please make an appointment Questions on Homework #4? 3
GACC What is beneficial about GACC? What is the big problem with GACC? Example: p = a == b pa = true pb = true GACC a b p pa pb 1 t t t t t 2 t f f t t 3 f t f t t 4 f f t t t Can be satisfied by {1,2} x {3,4} If (1,4) is used, p is always True If (2,3) is used, p is always False GACC does not guarantee predicate coverage 4
CACC Does Guarantee PC CACC guarantees Predicate Coverage For previous example, only (1,3) and (2,4) satisfy CACC CACC and RACC What is the difference between CACC and RACC? Technically: RACC imposes more constraints on the truth values of minor clauses There are two requirements for each clause: when it is the major clause: one requirement for when it is true and another requirement for when it is false RACC requires that the truth values for minor clauses are consistent between the two requirements 5
CACC vs. RACC RACC imposes more constraints than CACC In practice? There are fewer possible truth table row combinations that satisfy RACC RACC leads to more infeasible test requirements that CACC Huh? CACC and RACC pa = b OR c a b c a (b c) 1 T T T T 2 T T F T 3 T F T T 5 F T T F 6 F T F F 7 F F T F major clause a b c a (b c) 1 T T T T 5 F T T F 2 T T F T 6 F T F F 3 T F T T 7 F F T F major clause CACC can be satisfied by choosing any of rows 1, 2, 3 AND any of rows 5, 6, 7 a total of nine pairs RACC can only be satisfied by one of the three pairs above Ammann & Offutt 12 6
RACC Leads to Infeasible Requirements If clauses are independent, there is no problem But, clauses are often dependent Example Constraints: 1 A valve must be open in Operational mode and closed in all other modes 2 Mode cannot be in both Operational and Standby at the same time Clauses: a = The valve is closed b = The system mode is Operational c = The system mode is Standby Constraints: NOT a == b NOT(b AND c) 7
Example (1) NOT a == b (2) NOT(b AND c) a b c a AND (b OR c) constraint violations 1 T T T T 1,2 2 T T F T 1 3 T F T T 4 T F F F 5 F T T F 2 6 F T F F 7 F F T F 1 8 F F F F 1 CACC: out of {1,2,3} x {5,6,7} only (3,6) is feasible RACC: out of (1,5), (2,6), (3,7) none are feasible Inactive Clause Coverage The active clause coverage criteria ensure that major clauses do affect the predicates Inactive clause coverage takes the opposite approach major clauses do not affect the predicates Ammann & Offutt 16 8
General and Restricted ICC Unlike ACC, the notion of correlation is not relevant ci does not determine p, so cannot correlate with p Predicate coverage is always guaranteed Ammann & Offutt 17 Example: Section 3.2, #2 (g), (h) p = a OR (b AND c) pa = NOT b OR NOT c pb = NOT a AND c pc = NOT a AND b 9
Section 3.2, #2 Truth Table a b c p pa pb pc 1 T T T T 2 T T F T T 3 T F T T T 4 T F F T T 5 F T T T T T 6 F T F T T 7 F F T T T 8 F F F T Exercise: 3.2 #7 (c), (g), (h) p = (a OR b) AND (c OR d) Part (b) Solution: pa = NOT b AND (c OR d) pb = NOT a AND (c OR d) pc = NOT d AND (a OR b) pd = NOT c AND (a OR b) Do parts (c), (g), (h) 10
Logic Coverage Criteria Subsumption Combinatorial Clause Coverage COC Restricted Active Clause Coverage RACC Restricted Inactive Clause Coverage RICC Correlated Active Clause Coverage CACC General Inactive Clause Coverage GICC General Active Clause Coverage GACC Clause Coverage CC Predicate Coverage PC Ammann & Offutt 21 Logic Expressions from Source Predicates are derived from decision statements in programs In programs, most predicates have less than four clauses Wise programmers actively strive to keep predicates simple When a predicate only has one clause, COC, ACC, ICC, and CC all collapse to predicate coverage (PC) Applying logic criteria to program source is hard because of reachability and controllability: Reachability : Before applying the criteria on a predicate at a particular statement, we have to get to that statement Controllability : We have to find input values that indirectly assign values to the variables in the predicates Variables in the predicates that are not inputs to the program are called internal variables These issues are illustrated through an example in the following slides Ammann & Offutt 22 11
Triang (pg 1 of 5) 1 // Jeff Offutt -- Java version Feb 2003 2 // The old standby: classify triangles 3 // Figures 3.2 and 3.3 in the book. 4 import java.io.*; 5 class trityp 6 { 7 private static String[] tritypes = { "", // Ignore 0. 8 "scalene", "isosceles", "equilateral", "not a valid triangle"}; 9 private static String instructions = "This is the ancient TriTyp program.\nenter three integers that represent the lengths of the sides of a triangle.\nthe triangle will be categorized as either scalene, isosceles, equilateral\nor invalid.\n"; 10 11 public static void main (String[] argv) 12 { // Driver program for trityp 13 int A, B, C; 14 int T; Ammann & Offutt 23 Triang (pg 2 of 5) 16 System.out.println (instructions); 17 System.out.println ("Enter side 1: "); 18 A = getn(); 19 System.out.println ("Enter side 2: "); 20 B = getn(); 21 System.out.println ("Enter side 3: "); 22 C = getn(); 23 T = Triang (A, B, C); 24 25 System.out.println ("Result is: " + tritypes [T]); 26 } 27 28 // ==================================== Ammann & Offutt 24 12
Triang (pg 3 of 5) 29 // The main triangle classification method 30 private static int Triang (int Side1, int Side2, int Side3) 31 { 32 int tri_out; 33 34 // tri_out is output from the routine: 35 // Triang = 1 if triangle is scalene 36 // Triang = 2 if triangle is isosceles 37 // Triang = 3 if triangle is equilateral 38 // Triang = 4 if not a triangle 39 40 // After a quick confirmation that it s a legal 41 // triangle, detect any sides of equal length 42 if (Side1 <= 0 Side2 <= 0 Side3 <= 0) 43 { 44 tri_out = 4; 45 return (tri_out); 46 } Ammann & Offutt 25 Triang (pg 4 of 5) 48 tri_out = 0; 49 if (Side1 == Side2) 50 tri_out = tri_out + 1; 51 if (Side1 == Side3) 52 tri_out = tri_out + 2; 53 if (Side2 == Side3) 54 tri_out = tri_out + 3; 55 if (tri_out == 0) 56 { // Confirm it s a legal triangle before declaring 57 // it to be scalene 58 59 if (Side1+Side2 <= Side3 Side2+Side3 <= Side1 60 Side1+Side3 <= Side2) 61 tri_out = 4; 62 else 63 tri_out = 1; 64 return (tri_out); 65 } Ammann & Offutt 26 13
Triang (pg 5 of 5) 67 /* Confirm it s a legal triangle before declaring */ 68 /* it to be isosceles or equilateral */ 69 70 if (tri_out > 3) 71 tri_out = 3; 72 else if (tri_out == 1 && Side1+Side2 > Side3) 73 tri_out = 2; 74 else if (tri_out == 2 && Side1+Side3 > Side2) 75 tri_out = 2; 76 else if (tri_out == 3 && Side2+Side3 > Side1) 77 tri_out = 2; 78 else 79 tri_out = 4; 80 return (tri_out); 81 } // end Triang Ammann & Offutt 27 Ten Triang Predicates 42: (Side1 <= 0 Side2 <= 0 Side3 <= 0) 49: (Side1 == Side2) 51: (Side1 == Side3) 53: (Side2 == Side3) 55: (triout == 0) 59: (Side1+Side2 <= Side3 Side2+Side3 <= Side1 Side1+Side3 <= Side2) 70: (triout > 3) 72: (triout == 1 && Side1+Side2 > Side3) 74: (triout == 2 && Side1+Side3 > Side2) 76: (triout == 3 && Side2+Side3 > Side1) Ammann & Offutt 28 14
Reachability for Triang Predicates 42: True 49: P1 = s1>0 && s2>0 && s3>0 51: P1 53: P1 55: P1 59: P1 && triout = 0 62: P1 && triout = 0 && (s1+s2 > s3) && (s2+s3 > s1) && (s1+s3 > s2) 70: P1 && triout!= 0 72: P1 && triout!= 0 && triout <= 3 74: P1 && triout!= 0 && triout <= 3 && (triout!=1 s1+s2<=s3) 76: P1 && triout!= 0 && triout <= 3 && (triout!=1 s1+s2<=s3) && (triout!=2 s1+s3<=s2) Need to solve for the internal variable triout 78: P1 && triout!= 0 && triout <= 3 && (triout!=1 s1+s2<=s3) && (triout!=2 s1+s3 <= s2) && (triout!=3 s2+s3 <= s1) Ammann & Offutt 29 Program Transformation Issues if ((a && b) c) { S1; } else { S2; } d = a && b; e = d c; if (e) { S1; } else { S2; } Transform (2)? Transform (1)? if (a) { if (b) S1; else { if (c) { S1; else { S2; } } else { if (c) { S1; else { S2; } Ammann & Offutt 30 15
Problems with Transformed Programs Maintenance is certainly harder with Transform (1) Not recommended! Coverage on Transform (1) PC on transform does not imply CACC on original CACC on original does not imply PC on transform Coverage on Transform (2) Structure used by logic criteria is lost Hence CACC on transform 2 only requires 3 tests a b c (a b) c CACC PC CACC(2) T T T T X T T F T X X T F T T X X X T F F F X X F T T T X F T F F X X F F T T F F F F X Summary : Logic Coverage for Source Code Predicates appear in decision statements if, while, for, etc. Most predicates have less than four clauses But some applications have predicates with many clauses The hard part of applying logic criteria to source is resolving the internal variables Non-local variables (class, global, etc.) are also input variables if they are used If an input variable is changed within a method, it is treated as an internal variable thereafter To maximize effect of logic coverage criteria: Avoid transformations that hide predicate structure Ammann & Offutt 32 16
Exercise: Section 3.3 #3 Specifications in Software Specifications can be formal or informal Formal specs are usually expressed mathematically Informal specs are usually expressed in natural language Lots of formal languages and informal styles are available Most specification languages include explicit logical expressions, so it is very easy to apply logic coverage criteria Implicit logical expressions in natural-language specifications should be re-written as explicit logical expressions as part of test design You will often find mistakes One of the most common is preconditions Ammann & Offutt 34 17
Preconditions Programmers often include preconditions for their methods The preconditions are often expressed in comments in method headers Preconditions can be in javadoc, requires, pre, Example Saving addresses // name must not be empty // state must be valid // zip must be 5 numeric digits // street must not be empty // city must not be empty Rewriting to logical expression name!= state in statelist zip >= 00000 zip <= 99999 street!= city!= Ammann & Offutt 35 Covering Finite State Machines FSMs are graphs nodes represent state edges represent transitions among states Transitions often have logical expressions as guards or triggers As we said: Find a logical expression and cover it Ammann & Offutt 36 18
Example Subway Train secondplatform = right Left Doors Open All Doors Open emergencystop overrideopen doorsclear (all three transitions) secondplatform= left Right Doors Open trainspeed = 0 platform=left (instation (emergencystop overrideopen)) All Doors Closed trainspeed = 0 platform=right (instation (emergencystop overrideopen)) Ammann & Offutt 37 Determination of the Predicate trainspeed = 0 : platform = left (instation (emergencystop overrideopen)) platform = left : trainspeed = 0 (instation (emergencystop overrideopen)) instation : trainspeed = 0 platform = left ( emergencystop overrideopen) emergencystop : trainspeed = 0 platform = left ( instation overrideopen) overrideopen : trainspeed = 0 platform = left ( instation emergencystop) Ammann & Offutt 38 19
Test Truth Assignments (CACC) trainspeed=0 platform=left instation emergencystop overrideopen trainspeed = 0 T t t t t trainspeed!= 0 F t t T t platform = left duplicate t T t t t platform!= left T F t t t instation t t T f f instation t t F f f emergencystop t t f T t emergencystop t t f F t overrideopen duplicate t t f t T overrideopen t t f t F Note : other choices are possible Ammann & Offutt 39 Complicating Issues Some buttons must be pressed simultaneously to have effect so timing must be tested Reachability : The tests must reach the state where the transition starts (the prefix) Exit : Some tests must continue executing to an end state Expected output : The expected output is the state that the transition reaches for true values, or same state for false values Accidental transitions : Sometimes a false value for one transition happens to be a true value for another The alternate expected output must be recognized Ammann & Offutt 40 20
Exercise: Section 3.5, #3 Midterm Next Monday Less emphasis on the detailed working of problems More emphasis on concepts, tradeoffs and relationships between concepts 21
Review: High-Level Why do we need test adequacy criteria anyway!?! What is subsumption? What is infeasibility? Review: Graph Coverage Basics: Node, Edge, Edge-Pair Coverage Prime Path Coverage Touring versus Best-Effort Touring (Sidetrips and Detours) Data-Flow All-Definitions All-Uses (all DU Pairs) All-DU-Paths 22
Review: Graph Coverage Source code to Control Flow Graphs (CFGs) Source code to Call Graphs Structural Coverage of Call Graphs Data-Flow Coverage of Call Graphs (last def/first-use) Finite State Machines Activity Diagrams Review: Call-Graph Data Flow Last-Definition / First-Use pairs When making a call When returning Non-local variables 23
Review: Logic Coverage Basic Boolean Logic Active Clause Coverage: GACC, CACC, RACC High-level tradeoffs between the three Inactive Clause Coverage: GICC, RICC Logic Coverage from Source Code Predicates from if-statements and loop conditions Issues: Reachability Internal variables 24