MTAT Software Engineering

Transcription

1 MTAT Software Engineering Lecture 11: Verification and Validation (Part 2) Dietmar Pfahl Fall

2 Exam Registration Status as of Nov 19 Exam 1: Friday, 08-Jan-2015, 14:15-16:15, room Capacity limit: 90 students Registered: 47 Free seats: 43 Exam 2: Friday, 15-Jan-2015, 14:15-16:15, room Capacity limit: 90 students Registered: 9 Free seats: 81 You must register: First-come-first-serve (FIFO) principle

3 Schedule of Lectures Week 01: Introduction to SE Week 02: Requirements Engineering I Week 03: Requirements Engineering II Week 04: Analysis Week 05: Development Infrastructure I Week 06: Development Infrastructure II Week 07: Architecture and Design Week 08: No lecture Week 09: Refactoring Week 10: Verification and Validation I Week 11: Industry Lecture (Testing) Week 12: Verification and Validation II Week 13: Agile/Lean Methods Week 14: Industry Lecture (Agile) Week 15: Measurement / Course wrap-up, review and exam preparation Week 16: no lecture

4 Structure of Lectures 09 & 11 Fundamental Definitions and Concepts Test Strategies Black-Box White-Box Assessing the Quality of Test Suites Test Coverage Mutation Testing Static Analysis Quality Prediction

5 How Do We Test This? 34 switches = 2 34 = 1.7 x possible inputs = 1.7 x tests If only 3-way interactions, need only 33 tests For 4-way interactions, need only 85 tests

6 Combinatorial Testing What is it? Methods for systematically testing t-way interaction effects of input (or configuration parameter) values. Why do it? The interaction of specific combinations of input values may trigger failures that won t be triggered if testing input (or configuration parameter) values only in isolation.

7 Two Scopes of Combinatorial Testing Test Configurations Test Inputs Size Topp Addr Phone Sm Custom Val Val Sm Preset Inv Inv Test inputs Med Custom Inv Val Med Preset Val Inv Lg Custom Val Inv Lg Preset Inv Val Test case OS CPU Protocol 1 Windows Intel IPv4 2 Windows AMD IPv6 3 Linux Intel IPv6 4 Linux AMD IPv4 Pizza Ordering System under test (SUT)

8 Combinatorial Testing Example 2 Platform configuration parameters: OS: Windows XP, Apple OS X, Red Hat Linux Browser: Internet Explorer, Firefox Protocol: IPv4, IPv6 CPU: Intel, AMD DBMS: MySQL, Sybase, Oracle Total number of combinations: 3*2*2*2*3 = 72 Do we need 72 test cases?

9 Pair-Wise Testing (2-Way Interaction) Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pairwise interaction) # of parameter pairs: 5 2 5! 10 2!(5 2)! # of 2-way interactions N=57: N Test OS Browser Protocol CPU DBMS 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X IE IPv4 Intel Sybase 6 OS X Firefox IPv4 Intel Oracle 7 RHL IE IPv6 AMD MySQL 8 RHL Firefox IPv4 Intel Sybase 9 RHL Firefox IPv4 AMD Oracle 10 OS X Firefox IPv6 AMD Oracle

10 Pair-Wise Testing (2-Way Interaction) # of parameter pairs: 5 2 5! 10 2!(5 2)! # of 2-way interactions N=57: N 3 2 a:b 3 x 2 = 6 a:c 3 x 2 = 6 a:d 3 x 2 = 6 a:e 3 x 3 = 9 b:c 2 x 2 = 4 b:d 2 x 2 = 4 b:e 2 x 3 = 6 c:d 2 x 2 = 4 c:e 2 x 3 = 6 d:e 2 x 3 = Test OS (a) Browser (b) Protocol (c) CPU (d) DBMS (e) 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X IE IPv4 Intel Sybase 6 OS X Firefox IPv4 Intel Oracle 7 RHL IE IPv6 AMD MySQL 8 RHL Firefox IPv4 Intel Sybase 9 RHL Firefox IPv4 AMD Oracle 10 OS X Firefox IPv6 AMD Oracle

11 Pair-Wise Testing (2-Way Interaction) Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pairwise interaction) Parameter pair 1: 3 x 2 = 6 value combinations Test OS Browser Protocol CPU DBMS 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X IE IPv4 Intel Sybase 6 OS X Firefox IPv4 Intel Oracle 7 RHL IE IPv6 AMD MySQL 8 RHL Firefox IPv4 Intel Sybase 9 RHL Firefox IPv4 AMD Oracle 10 OS X Firefox IPv6 AMD Oracle

12 Pair-Wise Testing (2-Way Interaction) Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pairwise interaction) Pair 2: 2 x 2 = 4 value combinations Test OS Browser Protocol CPU DBMS 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X IE IPv4 Intel Sybase 6 OS X Firefox IPv4 Intel Oracle 7 RHL IE IPv6 AMD MySQL 8 RHL Firefox IPv4 Intel Sybase 9 RHL Firefox IPv4 AMD Oracle 10 OS X Firefox IPv6 AMD Oracle

13 Pair-Wise Testing (2-Way Interaction) Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pairwise interaction) Pair 3: 3 x 3 = 9 value combinations Test OS Browser Protocol CPU DBMS 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X IE IPv4 Intel Sybase 6 OS X Firefox IPv4 Intel Oracle 7 RHL IE IPv6 AMD MySQL 8 RHL Firefox IPv4 Intel Sybase 9 RHL Firefox IPv4 AMD Oracle 10 OS X Firefox IPv6 AMD Oracle

14 Is Testing 2-Way Interactions Enough? Analyses of failure-triggering conditions showed this: Medical device (dark blue) NASA distrib. DB (light blue) Browser (green) Web server (magenta) Network security (orange) TCAS* module (purple) * Traffic Collision Avoiding System

15 ACTS Tool (NIST & UT Arlington)

16 Combinatorial Testing Links

17 One problem is left Creating test (input) data is the (relatively) easy part! How do we check that the code worked correctly on the test input?

18 How to (Automatically) Generate Test Oracles? Creating test (input) data is the (relatively) easy part! How do we check that the code worked correctly on the test input? Using existing test sets (with known oracles) easy if test sets exist Crash testing code to ensure it does not crash for randomly generated test input ( fuzz testing ) easy but of limited value Embedded assertions incorporate assertions in code to check critical states at different points in the code will they cover all possible incorrect states? Model-checking using a mathematical model of the system under test and a model checker to generate expected results for each input expensive but tractable

19 State-Transition Testing Input (Event) State-Transition Graph Action Create a set of test cases that triggers each state-transition at least once

20 State Table State Input (Event) Wait for Card (S1) Wait for PIN (S2) Card inserted S2 (Ask for PIN) - - Invalid PIN - S2 (Beep) - Valid PIN - S3 (Ask amount) - Cancel - S1 (Return card) Next (S3)

21 State Table 4 Test Cases: - Input Card inserted in S1 - Input Invalid PIN in S2 - Input Valid PIN in S2 - Input Cancel in S2 State Input (Event) Wait for Card (S1) Wait for PIN (S2) Card inserted S2 (Ask for PIN) - - Invalid PIN - S2 (Beep) - Valid PIN - S3 (Ask amount) - Cancel - S1 (Return card) Next (S3)

22 Error Guessing Exploratory testing, happy testing,... Always worth including Can trigger failures that systematic techniques miss Consider Past failures Intuition Experience Brain storming What is the craziest thing we can do? Lists in literature

23 Exploratory Testing Inventors: Cem Kaner, James Bach (1990s) Definition: Exploratory testing is simultaneous learning, test design, and test execution. Elements / Variants Charter: defines mission (and sometimes tactics to use) Example: Check UI against Windows interface standards Session-based test management: Defects + Notes + Interviews of the testers

24 Testing Strategies requirements output input events Black Box Testing White Box Testing

25 White-Box Testing If-then-else There are many possible paths! 5 20 (~10 14 ) different paths loop < 20x Selective Testing

26 Selective Testing a selected path Selective: Control flow testing Data flow testing

27 Control Flow Graph (CFG) MTAT / Lecture 03 / Dietmar Pfahl 2015

28 Control Flow Graph (CFG) empty Blocks (=Nodes): 4 Edges: 4 MTAT / Lecture 03 / Dietmar Pfahl 2015

29 Control Flow Graph (CFG) e 1 e 2 s 1 s 2 e 3 e 4 s 3 d 1 s 5 e 5 e 6 s 4 s 6 e 7 e 8 Nodes: 8 Edges: 8 empty Blocks: 4 Edges: 4 MTAT / Lecture 03 / Dietmar Pfahl 2015

30 CFG Construction Example s 1 c 2 d 3 d 1 d 2 d s 6 3 d c 7 d 5 c 14 3 d 8 d 9 c 1 d 10 c4 d 4 s 4 d 5 d s 12 2 d 13 d 11 If (c1) then { if (c2) then {s1} s2 while (c3) do {s3} } else { if (c4) then { repeat {s4} until (c5) } }

31 CFG Construction Example CFG(t) c 1 d 1 d 2 CFG(f) If (c1) then { if (c2) then {s1} s2 If (c1) then { CFG(c1=true) while (c3) do {s3} } } else { else { if (c4) then { CFG(c1=false) repeat {s4} until (c5) } } } }

32 CFG Construction Example CFG(if) c 1 d 1 d 2 c4 If (c1) then { if (c2) then {s1} s2 If (c1) then { CFG(if) s2 d 4 d 10 while (c3) do {s3} } CFG(while) } CFG (while) s 2 d 6 d 9 CFG (repeat) d 14 d 11 else { if (c4) then { repeat {s4} until (c5) } } else { if (c4) then { CFG(repeat) } }

33 CFG Construction Example s 1 c 2 d 3 d 1 d 2 d s 6 3 d c 7 d 5 c 14 3 d 8 d 9 c 1 d 10 c4 d 4 s 4 d 5 d s 12 2 d 13 d 11 If (c1) then { if (c2) then {s1} s2 while (c3) do {s3} } else { if (c4) then { repeat {s4} until (c5) } }

34 Overview of Control Flow Criteria s 1 c 2 d 3 d 1 d 2 d s 6 3 d c 7 d 5 c 14 3 d 8 d 9 c 1 d 10 c4 d 4 s 4 d 5 d s 12 2 d 13 d 11 Statement (or Block) Coverage all nodes Decision (or Branch) Coverage all edges Condition Coverage Condition/Decision Coverage Multiple Condition Coverage Modified Condition Decision Coverage (MC/DC) Linearly Independent Paths Simple Paths Visit-Each Loop All Paths

35 Life Insurance Example boolean AccClient(int age; gtype gender) S 1: 2: 3: 4: 5: if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); d 1 = c 1 && c d 2 = c 3 && c 4 E 5

36 Statement Coverage Criterion: Execute each statement at least once Tools can be used to monitor execution ( coverage) Possible concern: Dead code Example: assume that due to some previous calculations, AccClient can only be invoked with parameter value gender = female

37 Statement Coverage /1 boolean AccClient(int age; gtype gender) S 1: 2: 3: 4: 5: if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); d 1 = c 1 && c d 2 = c 3 && c 4 Test: AccClient(83, female)->accept AccClient(83, male)->reject E 5 0 %

38 Statement Coverage /2 boolean AccClient(int age; gtype gender) S 1: 2: 3: 4: 5: if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); d 1 = c 1 && c d 2 = c 3 && c 4 Test: AccClient(83, female)->accept AccClient(83, male)->reject E 5 40 %

39 Statement Coverage /3 boolean AccClient(int age; gtype gender) S 1: 2: 3: 4: 5: if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); d 1 = c 1 && c d 2 = c 3 && c 4 Test: AccClient(83, female)->accept AccClient(83, male)->reject E 5 80 %

40 Statement Coverage /4 boolean AccClient(int age; gtype gender) S 1: 2: 3: 4: 5: if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); d 1 = c 1 && c d 2 = c 3 && c 4 Test: AccClient(83, female)->accept AccClient(83, male)->reject E 5 AccClient(25, male)->accept 100 %

41 Decision (Branch) Coverage /1 boolean AccClient(int age; gtype gender) S 1: 2: 3: 4: 5: if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); d 1 = c 1 && c d 2 = c 3 && c 4 Test: AccClient(83, female)->accept AccClient(83, male)->reject E 5 0 %

42 Decision (Branch) Coverage /2 1: 2: 3: 4: 5: boolean AccClient(int age; gtype gender) if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); Test: AccClient(83, female)->accept AccClient(83, male)->reject AccClient(25, male)->accept 25 % T d 1 = c 1 && c 2 2 E 4 1 S d 2 = c 3 && c 4 3 5

43 Decision (Branch) Coverage /3 1: 2: 3: 4: 5: boolean AccClient(int age; gtype gender) if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); Test: AccClient(83, female)->accept AccClient(83, male)->reject AccClient(25, male)->accept 75 % T/F d 1 = c 1 && c 2 2 E 4 1 S F d 2 = c 3 && c 4 3 5

44 Decision (Branch) Coverage /4 1: 2: 3: 4: 5: boolean AccClient(int age; gtype gender) if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); Test: AccClient(83, female)->accept AccClient(83, male)->reject AccClient(25, male)->accept 100 % T/F d 1 = c 1 && c 2 2 E 4 1 S T/F d 2 = c 3 && c 4 3 5

45 Decision (Branch) Coverage /? 1: 2: 3: 4: 5: 6: 7: 8: 9: boolean AccClient(int age; gtype gender) if (gender = female){ if (age < 85) return(true); return(false);} if (gender = male){ if (age < 80) return(true); return(false);} return(false); Dead code! Test: T d 2 = c 2 AccClient(83, female)->accept AccClient(83, male)->reject AccClient(25, male)->accept 3 2 T/F d 1 = c 1 4 E 1 T/F d 4 = c % 8 S T d 3 = c 3 5 9

46 Decision (Branch) Coverage /? 1: 2: 3: 4: 5: 6: 7: 8: 9: boolean AccClient(int age; gtype gender) if (gender = female){ if (age < 85) return(true); return(false);} if (gender = male){ if (age < 80) return(true); return(false);} return(false); Test: d 2 = c 2 AccClient(83, female)->accept AccClient(83, male)->reject AccClient(25, male)->accept 3 2 d 1 = c 1 1 d 4 = c E 100 % S d 3 = c 3 5 9

47 Condition Coverage Test all conditions (in all predicate nodes) Each condition must evaluate at least once (or: once to true and once to false ) A (simple) condition may contain: Relational operators Arithmetic expressions... If (A<10 and B>250) then A predicate may contain several (simple) conditions connected via Boolean operators

48 Condition Coverage /1 boolean AccClient(int age; gtype gender) S 1: 2: 3: 4: 5: if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); d 1 = c 1 && c d 2 = c 3 && c 4 Test: AccClient(83, female)->accept AccClient(83, male)->reject E 5 0 %

49 Condition Coverage /2 1: 2: 3: 4: 5: boolean AccClient(int age; gtype gender) if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); Test: AccClient(83, female)->accept AccClient(83, male)->reject T T T d 1 = c 1 && c 2 2 E 4 1 S d 2 = c 3 && c % or 25 %

50 Condition Coverage /3 1: 2: 3: 4: 5: boolean AccClient(int age; gtype gender) if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); Test: AccClient(83, female)->accept AccClient(83, male)->reject T/F T/F T d 1 = c 1 && c 2 2 E 4 1 S F T F d 2 = c 3 && c % or 62.5 %

51 Condition Coverage /4 1: 2: 3: 4: 5: boolean AccClient(int age; gtype gender) if (gender = female && age < 85) return(true); if (gender = male && age < 80) return(true); return(false); Test: AccClient(83, female)->accept AccClient(83, male)->reject AccClient(25, male)->accept T/F T/F T d 1 = c 1 && c 2 2 E 100 % or 75 % 4 1 S F/T T F/T d 2 = c 3 && c 4 3 5

52 Advanced Condition Coverage Condition/Decision Coverage (C/DC) as DC plus: every (simple) condition in each decision is tested in each possible outcome Modified Condition/Decision coverage (MC/DC) as above plus, every (simple) condition shown to independently affect a decision outcome (by varying that condition only) a (simple) condition independently affects a decision when, by flipping that condition and holding all the others fixed, the decision changes this criterion was created at Boeing and is required for aviation software according to RCTA/DO-178B Multiple-Condition Coverage (M-CC) all possible combinations of (simple) conditions within each decision taken

53 CC, DC, C/DC, M-CC, MC/DC Examples If (A<10 and B>250) then Condition: (TF) A = 2; B = 200 (D: False) [(FT) A = 12; B = 300 (D: False)] Decision: (TT) A = 2; B = 300 (D: True) (FT) A = 12; B = 300 (D: False) Condition/Decision: (TT) A = 2; B = 300 (D: True) (FF) A = 12; B = 200 (D: False) Multiple Condition: (TT) A = 2; B = 300 (D: True) (FT) A = 12; B = 300 (D: False) (TF) A = 2; B = 200 (D: False) (FF) A = 12; B = 200 (D: False) Modified Condition/Decision: (TT) A = 2; B = 300 (D: True) (FT) A = 12; B = 300 (D: False) (TF) A = 2; B = 200 (D: False)

54 Independent Path Coverage McCabe cyclomatic complexity estimates number of test cases needed The number of independent paths needed to cover all simple paths at least once in a program Visualize by drawing a CFG CC = #(edges) #(nodes) + 2 CC = #(decisions) + 1 while-loop if-then-else case-of

55 Independent Paths Coverage Example Independent Paths Coverage Requires that a minimum set of linearly independent paths through the control flow-graph be executed This test strategy is the rationale for McCabe s cyclomatic number (McCabe 1976) which is equal to the number of test cases required to satisfy the strategy. Cyclomatic Complexity = = 6

56 Independent Paths Coverage Example Edges: Path1: Path2: Path3: Path4: Path5: Path6:

57 Independent Paths Coverage Example Edges: Why no need to cover Path7??? Path7: Because it equals Path1+Path2-Path3!!! Path1: Path2: P1+P2: Path3: P3:

58 Control-Flow Coverage Relationships Subsumption: a criterion C1 subsumes another criterion C2, if any test set {T} that satisfies C1 also satisfies C2 Multiple Condition MC/DC C/DC Complete Path Linearly Indep. Path Condition Decision Statement

59 Data Flow Testing Identifies paths in the program that go from the assignment of a value to a variable to the use of such variable, to make sure that the variable is properly used. X:=14;.. Y:= X-3;

60 Data Flow Testing Definitions Def assigned or changed Uses utilized (not changed) C-use (Computation) e.g. right-hand side of an assignment, an index of an array, parameter of a function. P-use (Predicate) branching the execution flow, e.g. in an if statement, while statement, for statement. Example: All def-use paths (DU) requires that each DU chain is covered at least once Why interesting? E.g., consider: def-def-def (and no use) or use (without def)

61 Data Flow Testing Example [1] bool AccClient(agetype age; gndrtype gender) [2] bool accept [3] if(gender=female) [4] accept := age < 85; [5] else [6] accept := age < 80; [7] return accept Considering age, there are two DU paths: (a)[1]-[4] (b)[1]-[6] Test case data: AccClient(*, female)-> * AccClient(*, male)-> *

62 Data Flow Testing Example [1] bool AccClient(agetype age; gndrtype gender) [2] bool accept [3] if(gender=female) [4] accept := age < 85; [5] else [6] accept := age < 80; [7] return accept Considering gender, there is one DU path: (a)[1]-[3] Test case data: AccClient(*, *)-> *

63 Data Flow Criteria All c-uses All defs All p-uses Weaker All c-uses, some p-uses All p-uses, some c-uses All uses # tests All def-use paths Stronger

64 OO-Testing Approaches (c) 2008 Mauro Pezzè & Michal Young MTAT / Lecture 02 / Dietmar Pfahl 2015

65 Class Testing and State push(s, elem1) Show_top(s) -> elem1 pop(s, x) -> x=elem1 pop(s, x) -> x=? The result of a method call depends on the state of the stack MTAT / Lecture 05 / Dietmar Pfahl 2015

66 Stack Example How many test cases needed to cover all feasible branches? MTAT / Lecture 05 / Dietmar Pfahl 2015

67 Stack Example How many test cases needed to cover all feasible branches? 4 decisions MTAT / Lecture 05 / Dietmar Pfahl 2015

68 Stack Example How many test cases needed to cover all feasible branches? 4 decisions MTAT / Lecture 05 / Dietmar Pfahl 2015

69 Stack Example 2 test cases are enough: MTAT / Lecture 05 / Dietmar Pfahl 2015

70 Stack Example D1 D2 D3 2 test cases are enough: D4 MTAT / Lecture 05 / Dietmar Pfahl 2015

71 Stack Example D1 D2 D3 2 test cases are enough: D3: false D4 MTAT / Lecture 05 / Dietmar Pfahl 2015

72 Stack Example D1 D2 D3 D4 2 test cases are enough: D1: false D2: true D1: false D2: true D1: false D2: true D1: true D4: true... false D3: true MTAT / Lecture 05 / Dietmar Pfahl 2015

73 Stack Example CFGs true true if if false false true if false true for false MTAT / Lecture 05 / Dietmar Pfahl 2015

74 Stack Example CFG (Class) pop() Stack() true if false Stack push(x) true if false resize() true if false true for false MTAT / Lecture 05 / Dietmar Pfahl 2015

75 State Diagram for Stack Example non_exist Stack() empty (size = 0) pop() error c [size < length - 1 ] push(x) push(x) not_empty (0 < size < length) [size = 1] pop() [size > 1] pop() [size = length - 1] push(x) / resize() push(x): size++ / pop(): size resize(): length := length * 2 MTAT / Lecture 05 / Dietmar Pfahl 2015

76 Structure of Lectures 09 & 11 Fundamental Definitions and Concepts Test Strategies Black-Box White-Box Assessing the Quality of Test Suites Test Coverage Mutation Testing Static Analysis Quality Prediction

77 Code Coverage (Test Coverage) Measures the extent to which certain code items related to a defined test adequacy criterion have been executed (covered) by running a set of test cases (= test suites) Goal: Define test suites such that they cover as many (disjoint) code items as possible

78 Code Coverage Measure Example Statement Coverage (CV s ) Portion of the statements tested by at least one test case. S CV t s 100% S p S : number of statements tested S t p : total number of statements

79 Code Coverage Analysis Code coverage analysis is the process of: Finding areas of a program not exercised by a set of test cases Creating additional test cases to increase coverage Determining a quantitative measure of code coverage, which is (believed to be) a predictor of code quality Code coverage analyzers automate this process Additional aspect of code coverage analysis: Identifying redundant test cases that do not increase coverage

80 Main Classes of Test Adequacy Criteria Control Flow Criteria: Statement, decision (branch), condition, and path coverage are examples of control flow criteria They rely solely on syntactic characteristics of the program (ignoring the semantics of the program computation) Data Flow Criteria: Require the execution of path segments that connect parts of the code that are intimately connected by the flow of data Other Criteria: Requirements covered,...

81 Structure of Lectures 09 & 11 Fundamental Definitions and Concepts Test Strategies Black-Box White-Box Assessing the Quality of Test Suites Test Coverage Mutation Testing Static Analysis Quality Prediction

82 Static Analysis Document Review (manual) Different types Static Code Analysis (automatic) Structural properties / metrics

83 Static Code Analysis Example 1 Bug finding is about pointing out programming mistakes: bad practice, coding errors, unexpected behavior. One interesting example of bugs that static analysis can find is null pointer dereference. int i=0; String s = null; If (i>0) { s = "positive"; } If (s.contains("pos")) { System.out.println(s); } This code will compile but at runtime a null pointer exception will be thrown, the String s being null and calling the method contains. Static analysis tools, for instance FindBugs can find this bug and report it.

84 Static Code Analysis Example 2 Bug finding is about pointing out programming mistakes: bad practice, coding errors, unexpected behavior. One interesting example of bugs that static analysis can find is security vulnerabilities. Static analysis can be applied to find security flaws in code. With dataflow analysis, it is possible to follow the propagation of input data, and thus detect possible code injection. Example: public static void main(string args[]) { File f = new File(args[0]); f.open(); //... } This program opens a file with an argument entered in command line. The fact that we use this argument to open a file is just an example, the important fact is that we use directly an input without validation, which constitutes a serious security vulnerability. Dataflow analysis can detect this kind of flaw by finding the source of data inputs, following their propagation until their use in a sensitive instruction (like creating a File object). Source:

85 Reviews - Terminology Static testing testing without software execution Review meeting to evaluate software artifact Inspection formally defined review Walkthrough author guided review

86 Reviews complement testing

87 Relative Cost of Faults Maintenance 200 Source: Davis, A.M., Software Requirements: analysis and specification (1990)

88 Reading Techniques Ad hoc Checklist-based Defect-based Scenario-based Usage-based Perspective-based

89 Perspective-based Reading User Designer Tester Scenarios Purpose Decrease overlap (redundancy) Improve effectiveness

90 Structure of Lectures 09 & 11 Fundamental Definitions and Concepts Test Strategies Black-Box White-Box Assessing the Quality of Test Suites Test Coverage Mutation Testing Static Analysis Quality Prediction

91 Quality Prediction Quality def.: Undetected #Defects Based on product and process properties Quality = Function(Code Size Complexity) Quality = Function(Code Changes) Quality = Function(Detected #Defects) Quality = Function(Test Effort) Based on detected defects Capture-Recapture Models Reliability Growth models

92 Capture-Recapture Defect Estimation

93 Capture-Recapture Defect Estimation

94 Capture-Recapture Defect Estimation Situation: Two inspectors are assigned to inspect the same product d 1 : #defects detected by Inspector 1 d 2 : #defects detected by Inspector 2 d 12 : #defects by both inspectors N t : total #defects (detected and undetected) N r : remaining #defects (undetected) N t d 1 d d 12 2 N r N t ( d d d )

95 Capture-Recapture Example Situation: Two inspectors are assigned to inspect the same product d 1 : 50 defects detected by Inspector 1 d 2 : 40 defects detected by Inspector 2 d 12 : 20 defects by both inspectors N t : total defects (detected and undetected) N r : remaining defects (undetected) d d N t N 100 ( ) 30 d r 12

96 Advanced Capture-Recapture Models Four basic models used for inspections Degree of freedom Prerequisites for all models All reviewers work independently of each other It is not allowed to inject or remove faults during inspection

97 Advanced Capture-Recapture Models Model Probability of defect being found is equal across... Defect Reviewer Estimator M0 Yes Yes Maximum-likelihood Mt Yes No Maximum-likelihood Chao s estimator Mh No Yes Jackknife Chao s estimator Mth No No Chao s estimator

98 Mt Model Maximum-likelihood: Mt = total marked animals (=faults) at the start of the t'th sampling interval Ct = total number of animals (=faults) sampled during interval t Rt = number of recaptures in the sample Ct An approximation of the maximum likelihood estimate of population size (N) is: SUM(Ct*Mt)/SUM(Rt) First resampling: M1=50 (first inspector) C1=40 (second inspector) R1=20 N=40*50/20=100 Second resampling: M2=70 (first and second inspector) C2=40 (third inspector) R2=30 N=(40*50+40*70)/(20+30)=4800/50=96 Third resampling: M3=80 C3=30 (fourth inspector) R3=30 N=( *80)/( )=7200/80=90

99 Reliability Growth Models To predict the probability of future failure occurrence based on past (observed) failure occurrence Can be used to estimate the number of residual (remaining) faults or the time until the next failure occurs the remaining test time until a reliability objective is achieved or Application typically during system test

100 Reliability Growth Models (RGMs) 100% 95% Cumulative #Failures (m) m( t ) 0e (estimated n 0 ) t 0 0 t n 0 dt Purpose: Stop testing when a certain percentage (90%, 95%, 99%, 99.9%, ) of estimated total number of failures has been reached a certain failure rate has been reached Test Intensity (t) (CPU time, test effort, test days, calendar days, )

101 Model Selection Many different RGMs have been proposed (>100) To choose a reliability model, perform the following steps: 1. Collect failure data 2. Examine data (failure data vs. test time/effort) 3. Select a set of candidate models 4. Estimate model parameters for each candidate model Least squares method Maximum likelihood method 5. Customize model using the estimated parameters 6. Compare models with goodness-of-fit test and select the best 7. Make reliability predictions with selected model(s)

102 Next Lecture Date/Time: Friday, 27-Nov, 10:15-12:00 Topic: Agile / Lean Methods For you to do: Work on Lab Task 6 Attend assessment session for Lab Task 5