MTAT Software Engineering

Transcription

1 MTAT Software Engineering Lecture 10: Verification and Validation Dietmar Pfahl Fall

2 Schedule of Lectures Week 01: Introduction to SE Week 02: Requirements Engineering I Week 03: Requirements Engineering II Week 04: Analysis Week 05: Development Infrastructure I Week 06: Development Infrastructure II Week 07: Architecture and Design Week 08: Refactoring Week 09: Quality Management Week 10: Verification and Validation (incl. SW Quality) Week 11: Agile/Lean Methods Week 12: Measurement Week 13: Process Improvement Week 14: Course wrap-up, review and exam preparation Week 15: no lecture Week 16: no lecture

3 Structure of Lecture 10 Fundamental Definitions and Concepts Creating Test Cases Manually Automatically Assessing the Quality of Test Suites Test Coverage Mutation Testing Static Analysis Quality Prediction

4 Quality attributes ISO 9126

5 What is Software Testing? The process of evaluating a program or a system Verifying that the product is right Validating that the right product is developed Confirm quality vs. Find defects (Testing paradox)

6 Definition: Error Fault Failure Failure is an event, fault is a state of the software caused by an error Error mistake made by human (e.g., programmer) Fault anomaly in the software Failure inability to perform its required functions Defect? Bug? Debugging / Fault localization localizing, repairing, re-testing.

7 Origins and Impact of Faults Defect sources Lack of education Poor communication Oversight Transcription Immature process Errors Fault model Impact of software artifacts Faults Failures Impact from user s view Poor quality software User dissatisfaction

8 Types of Defects Requirements defects Design defects Coding defects Testing defects

9 Definition: Test Case A Test Case consists of: A set of inputs + expected outputs Execution conditions Example of execution condition : When pressing the save button of a word processer, what happens depends on what you did previously Test Suite = set of test cases

10 Definition: Test Oracle Test Oracle = a mechanism used for determining whether a test has passed or failed It provides the expected result for a test, e.g. from Specification document, mathematical formula, program model, other program, person, Test Verdict = the actual judgment after a test case terminates pass/fail/warning/don t know Sometimes it is difficult/impossible to find an oracle Non-functional quality aspects Scientific computing

11 Classification: Types of Testing acceptance, regression,... Level of detail (scope) system integration module (component) unit (class, method) ISO 9126 portability maintainability white box black box Accessibility efficiency usability reliability functionality Characteristics

12

13 Structure of Lecture 10 Fundamental Definitions and Concepts Creating Test Cases Manually Automatically Assessing the Quality of Test Suites Test Coverage Mutation Testing Static Analysis Quality Prediction

14 Test Complexity Example: 40 variables, 2 levels -> combinations 3 tests/second -> 3000 years of test! 3 tests/millisecond -> 3 years of test!

15 Why Test Case Design Techniques? Exhaustive testing (use of all possible inputs and conditions) is impractical Must use a subset of all possible test cases Must have high probability of detecting faults Need processes that help us selecting test cases Different people equal probability to detect faults Effective testing detect more faults Focus attention on specific types of faults Know you re testing the right thing Efficient testing detect faults with less effort Avoid duplication Systematic techniques are measurable and repeatable

16 Basic Testing Strategies

17 Black-Box vs. White-Box External/user view: Check conformance with specification -> function coverage Abstraction from details: Source code not needed Scales up: Different techniques at different levels of granularity USE Internal/developer view: Allows tester to be confident about code coverage Based on control or data flow: Easier debugging Does not scale up: Most useful at unit & integration testing levels, as well as regression testing BOTH!

18 Black-Box vs. White-Box System Specification Implementation Missing functionality: Cannot be revealed by whitebox techniques Unexpected functionality: Cannot be revealed by blackbox techniques

19 Black-Box Testing Methods Equivalence Class Partitioning (ECP) Boundary Value Analysis (BVA) Combinatorial Testing Exploratory Testing ( error guessing ) Usability Testing...

20 Equivalence Class Partitioning Split input/output into classes which the software handles equivalently Select test cases to represent each class x x x x

21 Example Insurance System Specification Statement: System shall reject over-age insurance applicants Specification Item: Reject male insurance applicants, if over the age of 80 years on day of application Reject female insurance applicants, if over the age of 85 years on day of application

22 Example (cont.) Input: Gender & Age Output: accept/reject Classes C1: Input: Males over 80 C2: Input: Males 80 or under C3: Input: Females over 85 C4: Input: Females 85 or under C5: Output: accept C6: Output: reject Test Cases TC1: male, 83, reject TC2: male, 56, accept TC3: female, 101, reject TC4: female, 83, accept TC5: female, -3,?? TC6: fmale, 14,??

23 ECP Guidelines [Myers 79/04] If input condition: is a range, e.g., x = [0, 9] is an ordered list of values, e.g., owner = <1, 2, 3, 4> one valid and two invalid classes are defined is a set, e.g., vehicle = [car, motorcycle, truck] is a must be condition (boolean) one valid and one invalid class are defined is anything else partition further

24 Boundary Value Analysis Adds to the equivalence partitioning method Select test cases to represent each side of the class boundaries x x x x x x

25 Insurance Example Input: Gender & Age Output: accept/reject Classes C1: Males over 80 C2: Males 80 or under C3: Females over 85 C4: Females 85 or under C5: Output accept. C6: Output reject. Test Cases TC1: male, 81, reject TC2: male, 80, accept TC3: female, 86, reject TC4: female, 85, accept TC5: female, -1,?? TC6: female, 0, accept TC7: fmale, -1,?? TC8: fmale, 0,??

26 Boundary Value Analysis Guidelines Range a..b a, b, just above a, just below b List of values: max, min, just below min, just above max Boundaries of externally visible data structures shall be checked (e.g. ordered sets, arrays) Output bounds should be checked

27 Combinatorial Designs ECP and BVA define test cases per class Combinations of equivalence classes need to be handled Combinatorial explosion needs to be handled x x x x x x x x

28 Cause-Effect Graphing 1 2 and/or 3 3 occurs if both/one of 1 and 2 are present occurs if 1 occurs 2 occurs if 1 does not occur

29 Cause-Effect Graphing 1. The tester must decompose the specification into lower level units 2. For each specification unit, the tester needs to identify causes and effects. A cause is a distinct input condition (or equivalence class of input conditions) An effect is an output condition (or equivalence class of output conditions) or a system transformation A table of causes and effects helps the tester to record the necessary details. Then relationships between the causes and effects should be determined, ideally in the form of a set of rules. 3. From the set of rules, the boolean cause-effect graph is constructed, which in turn is converted into a decision table. 4. Test cases are then generated from this decision table.

30 Cause-Effect Graphing female Age Gender <=80 <80 <=85 >85 male female Age <= 80 and Accept/Reject accept reject 80 < Age <= 85 Age > 85 and accept reject male One advantage of this method is that development of the rules and the graph from the specification allows for a thorough inspection (logically, semantically) of the specification.

31 Decision Table Format Case Female Male <= 80 >80 & <= 85 > 85 Accept Reject TC TC TC TC A test-input would then be generated from this table, E.g.: TC1: male, 83 reject TC2: male, 75 accept (could also be female) TC3: female, 88 reject (could also be male) TC4: female, 83 accept

32 Combinatorial Testing Example 1 Many variables Many values per variable Need to abstract values (equivalence classes, boundary values) Plan: flt, flt+hotel, flt+hotel+car From: CONUS, HI, AK, Europe, Asia To: CONUS, HI, AK, Europe, Asia Compare: yes, no Date-type: exact, 1to3, flex Depart: today, tomorrow, 1yr, Sun, Mon Return: today, tomorrow, 1yr, Sun, Mon Adults: 1, 2, 3, 4, 5, 6 Minors: 0, 1, 2, 3, 4, 5 Seniors: 0, 1, 2, 3, 4, 5

33 How Do We Test This? 34 switches = 2 34 = 1.7 x possible inputs = 1.7 x tests If only 3-way interactions, need only 33 tests For 4-way interactions, need only 85 tests

34 Combinatorial Testing What is it? Methods for systematically testing t-way interaction effects of input (or configuration parameter) values. Why do it? The interaction of specific combinations of input values may trigger failures that won t be triggered if testing input (or configuration parameter) values only in isolation.

35 Two Scopes of Combinatorial Testing Test Configurations Test Inputs Size Topp Addr Phone Sm Custom Val Val Sm Preset Inv Inv Test inputs Med Custom Inv Val Med Preset Val Inv Lg Custom Val Inv Lg Preset Inv Val Test case OS CPU Protocol 1 Windows Intel IPv4 2 Windows AMD IPv6 3 Linux Intel IPv6 4 Linux AMD IPv4 Pizza Delivery System under test

36 Combinatorial Testing Example 2 Platform configuration parameters: OS: Windows XP, Apple OS X, Red Hat Linux Browser: Internet Explorer, Firefox Protocol: IPv4, IPv6 CPU: Intel, AMD DBMS: MySQL, Sybase, Oracle Total number of combinations: 3*2*2*2*3 = 72 Do we need 72 test cases?

37 Pair-Wise Testing (2-Way Interaction) # of pairs: 5 2 5! 10 2!(5 2)! # of 2-way interactions N=57: N Test a:b 3 x 2 = 6 a:c 3 x 2 = 6 a:d 3 x 2 = 6 a:e 3 x 3 = 9 b:c 2 x 2 = 4 b:d 2 x 2 = 4 b:e 2 x 3 = 6 c:d 2 x 2 = 4 c:e 2 x 3 = 6 d:e 2 x 3 = OS (a) Browser (b) Protocol (c) CPU (d) DBMS (e) 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X IE IPv4 Intel Sybase 6 OS X Firefox IPv4 Intel Oracle 7 RHL IE IPv6 AMD MySQL 8 RHL Firefox IPv4 Intel Sybase 9 RHL Firefox IPv4 AMD Oracle 10 OS X Firefox IPv6 AMD Oracle

38 Pair-Wise Testing (2-Way Interaction) Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pairwise interaction) Pair 1: 3 x 2 = 6 Combinations Test OS Browser Protocol CPU DBMS 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X IE IPv4 Intel Sybase 6 OS X Firefox IPv4 Intel Oracle 7 RHL IE IPv6 AMD MySQL 8 RHL Firefox IPv4 Intel Sybase 9 RHL Firefox IPv4 AMD Oracle 10 OS X Firefox IPv6 AMD Oracle

39 Pair-Wise Testing (2-Way Interaction) Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pairwise interaction) Pair 2: 2 x 2 = 4 Combinations Test OS Browser Protocol CPU DBMS 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X IE IPv4 Intel Sybase 6 OS X Firefox IPv4 Intel Oracle 7 RHL IE IPv6 AMD MySQL 8 RHL Firefox IPv4 Intel Sybase 9 RHL Firefox IPv4 AMD Oracle 10 OS X Firefox IPv6 AMD Oracle

40 Pair-Wise Testing (2-Way Interaction) Only 10 tests needed, if we want to test all interactions of one parameter with one other parameter (pairwise interaction) Pair 3: 3 x 3 = 9 Combinations Test OS Browser Protocol CPU DBMS 1 XP IE IPv4 Intel MySQL 2 XP Firefox IPv6 AMD Sybase 3 XP IE IPv6 Intel Oracle 4 OS X Firefox IPv4 AMD MySQL 5 OS X IE IPv4 Intel Sybase 6 OS X Firefox IPv4 Intel Oracle 7 RHL IE IPv6 AMD MySQL 8 RHL Firefox IPv4 Intel Sybase 9 RHL Firefox IPv4 AMD Oracle 10 OS X Firefox IPv6 AMD Oracle

41 Is Testing 2-Way Interactions Enough? Analyses of failure-triggering conditions showed this: Medical device (dark blue) NASA distrib. DB (light blue) Browser (green) Web server (magenta) Network security (orange) TCAS* module (purple) * Traffic Collision Avoiding System

42 Is Testing 2-Way Interactions Enough? Several studies have shown that Pair-wise Testing triggers between 50% and 90% of all failures. More strengths than 6-way interaction has hardly ever shown to find more defects. Why is that good?

43 Error Guessing Exploratory testing, happy testing,... Always worth including Can trigger failures that systematic techniques miss Consider Past failures Intuition Experience Brain storming What is the craziest thing we can do? Lists in literature

44 Exploratory Testing Inventors: Cem Kaner, James Bach (1990s) Definition: Exploratory testing is simultaneous learning, test design, and test execution. Elements / Variants Charter: defines mission (and sometimes tactics to use) Example: Check UI against Windows interface standards Session-based test management: Defects + Notes + Interviews of the testers

45 Usability Testing Characteristics Understandability Easy to understand? Ease of learning Easy to learn? Operability Matches purpose & environment of operation? Ergonomics: color, font, sound,... Communicativeness In accordance with psychological characteristics of user? Environments Free form tasks Procedure scripts Paper screens Mock-ups Field trial

46 Usability Test Types + Environment Rubin s Types of Usability Tests (Rubin, 1994, p ) Exploratory test early product development Assessment test most typical, either early or midway in the product development Validation test verification of product s usability Comparison test compare two or more designs; can be used with other three types of tests

47 Testing Strategies requirements output input events Black Box Testing White Box Testing

48 White-Box Testing If-then-else There are many possible paths! 5 20 (~10 14 ) different paths loop < 20x Selective Testing

49 Selective Testing a selected path Selective: Control flow testing Data flow testing

50 Control Flow Graph (CFG)

51 Control Flow Graph (CFG) empty Blocks (= Nodes): 4 Edges: 4

52 Control Flow Example s 1 c 2 d 3 d 1 d 2 d s 6 3 d c 7 d 5 c 14 3 d 8 d 9 c 1 d 10 c4 d 4 s 4 d 5 d s 12 2 d 13 d 11 If (c1) then { if (c2) then {s1} s2 while (c3) do {s3} } else { if (c4) then { repeat {s4} until (c5) } }

53 Control Flow Example CFG(t) c 1 d 1 d 2 CFG(f) If (c1) then { if (c2) then {s1} s2 If (c1) then { CFG(c1=true) while (c3) do {s3} } } else { else { if (c4) then { CFG(c1=false) repeat {s4} until (c5) } } } }

54 Control Flow Example CFG(if) c 1 d 1 d 2 c4 If (c1) then { if (c2) then {s1} s2 If (c1) then { CFG(if) s2 d 4 d 10 while (c3) do {s3} } CFG(while) } CFG (while) s 2 d 6 d 9 CFG (repeat) d 14 d 11 else { if (c4) then { repeat {s4} until (c5) } } else { if (c4) then { CFG(repeat) } }

55 Control Flow Example s 1 c 2 d 3 d 1 d 2 d s 6 3 d c 7 d 5 c 14 3 d 8 d 9 c 1 d 10 c4 d 4 s 4 d 5 d s 12 2 d 13 d 11 If (c1) then { if (c2) then {s1} s2 while (c3) do {s3} } else { if (c4) then { repeat {s4} until (c5) } }

56 Overview of Control Flow Criteria s 1 c 2 d 3 d 1 d 2 d s 6 3 d c 7 d 5 c 14 3 d 8 d 9 c 1 d 10 c4 d 4 s 4 d 5 d s 12 2 d 13 d 11 Statement (or Block) Coverage all nodes Decision (or Branch) Coverage all edges Condition Coverage Condition/Decision Coverage Multiple Condition Coverage Modified Condition Decision Coverage (MC/DC) Linearly Independent Paths Simple Paths Visit-Each Loop All Paths

57 Life Insurance Example bool AccClient(agetype age; gndrtype gender) d1 c1 d5 bool accept if(gender=female) d2 c2 d3 d6 c3 d7 accept := age < 85; s1 s2 else accept := age < 80; d4 s3 d8 return accept

58 Statement Coverage Execute each statement at least once Tools can be used to monitor execution Possible concern: Dead code Example: assume that due to some previous calculations, AccClient can only be invoked with parameter value gender = female

59 Statement Coverage bool AccClient(agetype age; gndrtype gender) bool accept c1 if(gender=female) d1 d5 accept := age < 85; c2 c3 else d2 d3 d6 d7 accept := age < 80; s1 s2 return accept d4 s3 d8

60 Statement Coverage bool AccClient(agetype age; gndrtype gender) AccClient(83, female)->accept bool accept c1 if(gender=female) d1 d5 accept := age < 85; c2 c3 else d2 d3 d6 d7 accept := age < 80; s1 s2 return accept d4 s3 d8

61 Statement Coverage bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) else accept := age < 85; accept := age < 80; return accept AccClient(83, female)->accept AccClient(83, male) ->reject d2 c2 s1 d1 d4 d3 c1 d6 s3 d5 c3 s2 d8 d7

62 Decision (Branch) Coverage /1 bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; else accept := age < 80; return accept true AccClient(83, female) ->accept AccClient(83, male) ->reject d2 c2 s1 d1 d4 d3 c1 d6 s3 d5 c3 s2 d8 d7

63 Decision (Branch) Coverage /2 bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; else accept := age < 80; return accept false AccClient(83, female) ->accept AccClient(83, male) ->reject d2 c2 s1 d1 d4 d3 c1 d6 s3 d5 c3 s2 d8 d7

64 Decision (Branch) Coverage /3 bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; else accept := age < 80; return accept true false AccClient(83, female) ->accept AccClient(83, male) ->reject d2 75% coverage only! c2 s1 d1 d4 d3 c1 d6 s3 d5 c3 s2 d8 d7

65 Condition Coverage Test all conditions (in all predicate nodes) Each condition must evaluate at least once (or: once to true and once to false ) A (simple) condition may contain: Relational operators Arithmetic expressions... If (A<10 and B>250) then A predicate may contain several (simple) conditions connected via Boolean operators

66 Condition Coverage /1 bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; else accept := age < 80; return accept true AccClient(83, female) ->accept AccClient(83, male) ->reject true d2 true c2 s1 d1 d4 d3 c1 d6 s3 d5 c3 s2 d8 d7

67 Condition Coverage /2 bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; else accept := age < 80; return accept false AccClient(83, female) ->accept AccClient(83, male) ->reject d2 c2 s1 false d1 d4 d3 c1 false d6 s3 d5 c3 s2 d8 d7

68 Condition Coverage /3 bool AccClient(agetype age; gndrtype gender) bool accept if(gender=female) accept := age < 85; else accept := age < 80; return accept true false AccClient(83, female) ->accept AccClient(83, male) ->reject d2 true 67% or 100% coverage (depends on definition) c2 s1 true, false d1 d4 d3 c1 false d6 s3 d5 c3 s2 d8 d7

69 Advanced Condition Coverage Condition/Decision Coverage (C/DC) as DC plus: every (simple) condition in each decision is tested in each possible outcome Modified Condition/Decision coverage (MC/DC) as above plus, every (simple) condition shown to independently affect a decision outcome (by varying that condition only) a (simple) condition independently affects a decision when, by flipping that condition and holding all the others fixed, the decision changes this criterion was created at Boeing and is required for aviation software according to RCTA/DO-178B Multiple-Condition Coverage (M-CC) all possible combinations of (simple) conditions within each decision taken

70 CC, DC, C/DC, M-CC, MC/DC Examples If (A<10 and B>250) then Condition: (TF) A = 2; B = 200 (False) [(FT) A = 12; B = 300 (False)] Decision: (TT) A = 2; B = 300 (True) (FT) A = 12; B = 300 (False) Condition/Decision: (TT) A = 2; B = 300 (True) (FF) A = 12; B = 200 (False) Multiple Condition: (TT) A = 2; B = 300 (True) (FT) A = 12; B = 300 (False) (TF) A = 2; B = 200 (False) (FF) A = 12; B = 200 (False) Modified Condition/Decision: (TT) A = 2; B = 300 (True) (FT) A = 12; B = 300 (False) (TF) A = 2; B = 200 (False)

71 Independent Path Coverage McCabe cyclomatic complexity estimates number of test cases needed The number of independent paths needed to cover all simple paths at least once in a program Visualize by drawing a CFG CC = #(edges) #(nodes) + 2 CC = #(decisions) + 1 while-loop if-then-else case-of

72 Independent Paths Coverage Example Independent Paths Coverage Requires that a minimum set of linearly independent paths through the control flow-graph be executed This test strategy is the rationale for McCabe s cyclomatic number (McCabe 1976) which is equal to the number of test cases required to satisfy the strategy. Cyclomatic Complexity = = 6

73 Independent Paths Coverage Example Edges: Path1: Path2: Path3: Path4: Path5: Path6:

74 Independent Paths Coverage Example Edges: Why no need to cover Path7??? Path7: Because it equals Path1+Path2-Path3!!! Path1: Path2: P1+P2: Path3: P3:

75 Control-Flow Coverage Relationships Subsumption: a criterion C1 subsumes another criterion C2, if any test set {T} that satisfies C1 also satisfies C2 Multiple Condition MC/DC C/DC Complete Path Linearly Indep. Path Condition Decision Statement

76 Data Flow Testing Identifies paths in the program that go from the assignment of a value to a variable to the use of such variable, to make sure that the variable is properly used. X:=14;.. Y:= X-3;

77 Data Flow Testing Definitions Def assigned or changed Uses utilized (not changed) C-use (Computation) e.g. right-hand side of an assignment, an index of an array, parameter of a function. P-use (Predicate) branching the execution flow, e.g. in an if statement, while statement, for statement. Example: All def-use paths (DU) requires that each DU chain is covered at least once Why interesting? E.g., consider: def-def-def (and no use) or use (without def)

78 Data Flow Testing Example [1] bool AccClient(agetype age; gndrtype gender) [2] bool accept [3] if(gender=female) [4] accept := age < 85; [5] else [6] accept := age < 80; [7] return accept Considering age, there are two DU paths: (a)[1]-[4] (b)[1]-[6] Test case data: AccClient(*, female)-> * AccClient(*, male)-> *

79 Data Flow Testing Example [1] bool AccClient(agetype age; gndrtype gender) [2] bool accept [3] if(gender=female) [4] accept := age < 85; [5] else [6] accept := age < 80; [7] return accept Considering gender, there is one DU path: (a)[1]-[3] Test case data: AccClient(*, *)-> *

80 Data Flow Criteria All c-uses All defs All p-uses Weaker All c-uses, some p-uses All p-uses, some c-uses All uses # tests All def-use paths Stronger

81 What can be automated? 1. Identify 2. Design Intellectual Performed once 3. Build 4. Execute Repeated Clerical 5. Check

82 Test Automation Effectiveness independent of automation Efficiency can be improved by automation Automation takes 2-10(-30) times the time for first run!

83 What to automate first? Most important tests A set of breadth tests (sample each system area overall) Test for the most important functions Tests that are easiest to automate Tests that will give the quickest payback Test that are run the most often

84 Tests to automate Tests that are run often Regression tests Important but trivial Expensive to perform manually Multi-user tests Endurance/reliability tests Difficult to perform manually... not to automate Tests that are run rarely Tests that are not important will not find severe faults Hard to recognize defects Usability tests Do the colours look nice? Timing critical Complex tests Difficult comparisons

85 Test Automation Not only the automatic execution of test cases Also: Generation of test data (input data) Evaluation of tests (-> test oracle needed) Generation of test cases (including test oracle) Selection of test cases (e.g., in regression testing) Reporting of test results Measurement of test effectiveness

86 Test Automation Approaches For input data generation: Random testing Statistical testing Load / stress testing For test case generation: Symbolic execution Model-based testing For test case selection: Search-based testing Regression testing Combinatorial testing (covering arrays) For test effectiveness measurement: Coverage measurement Mutation testing

87 Test automation promises 1. Efficient regression test 2. Run tests more often 3. Perform difficult tests (e.g. load, outcome check) 4. Better use of resources 5. Consistency and repeatability 6. Reuse of tests 7. Earlier time to market 8. Increased confidence

88 Common problems 1. Unrealistic expectations 2. Poor testing practice Automatic chaos just gives faster chaos 3. Expected effectiveness 4. False sense of security 5. Maintenance of automatic tests 6. Technical problems (e.g. Interoperability) 7. Organizational issues

89 Structure of Lecture 10 Fundamental Definitions and Concepts Creating Test Cases Manually Automatically Assessing the Quality of Test Suites Test Coverage Mutation Testing Static Analysis Quality Prediction

90 Code Coverage (Test Coverage) Measures the extent to which certain code items related to a defined test adequacy criterion have been executed (covered) by running a set of test cases (= test suites) Goal: Define test suites such that they cover as many (disjoint) code items as possible

91 Code Coverage Measure Example Statement Coverage (CV s ) Portion of the statements tested by at least one test case. S CV t s 100% S p S : number of statements tested S t p : total number of statements

92 Code Coverage Analysis Code coverage analysis is the process of: Finding areas of a program not exercised by a set of test cases Creating additional test cases to increase coverage Determining a quantitative measure of code coverage, which is (believed to be) a predictor of code quality Code coverage analyzers automate this process Additional aspect of code coverage analysis: Identifying redundant test cases that do not increase coverage

93 Main Classes of Test Adequacy Criteria Control Flow Criteria: Statement, decision (branch), condition, and path coverage are examples of control flow criteria They rely solely on syntactic characteristics of the program (ignoring the semantics of the program computation) Data Flow Criteria: Require the execution of path segments that connect parts of the code that are intimately connected by the flow of data Other Criteria: Requirements covered,...

94 Mutation Testing

95 Let s count marbles... a lot of marbles Suppose we have a big bowl of marbles How can we estimate how many? I don t want to count every marble individually I have a bag of 100 other marbles of the same size, but a different color What if I mix them?

96 Estimating marbles... I mix 100 black marbles into the bowl Stir well... I draw out 100 marbles at random 20 of them are black How many marbles were in the bowl to begin with? 400

97 Estimating Test Suite Quality Now, instead of a bowl of marbles, I have a program with faults (bugs) I add 100 new faults Assume they are exactly like real faults in every way I make 100 copies of my program, each with one of my 100 new faults I run my test suite on the programs with seeded faults and the tests reveal 20 of the faults (the other 80 program copies do not fail) What can I infer about my test suite?

98 Basic Assumptions We d like to judge effectiveness of a test suite in finding real faults, by measuring how well it finds seeded fake faults. Valid to the extent that the seeded bugs are representative of real bugs Not necessarily identical (e.g., black marbles are not identical to clear marbles); but the differences should not affect the selection E.g., if I mix metal ball bearings into the marbles, and pull them out with a magnet, I don t learn anything about how many marbles were in the bowl

99 Mutation Testing Assumptions Competent programmer hypothesis: Programs are nearly correct Real faults are small variations from the correct program => Mutants are reasonable models of real faulty programs Coupling effect hypothesis: Tests that find simple faults also find more complex faults Even if mutants are not perfect representatives of real faults, a test suite that kills mutants is good at finding real faults too

100 Fault-Based Testing (Mutation Testing) Terminology Mutant new version of the program with a small deviation (=fault) from the original version Killed mutant new version detected by the test suite Live mutant new version not detected by the test suite

101 Mutation Testing A method for evaluation of test suite effectiveness not for designing test cases! 1. Take a program and test data generated for that program 2. Create a number of similar programs (mutants), each differing from the original in a small way 3. The original test data are then run through the mutants 4. If tests detect all changes in mutants, then the mutants are dead and the test suite adequate Otherwise: Create more test cases and iterate 2-4 until a sufficiently high number of mutants is killed

102 Example Mutation Operations Change relational operator (<,>, ) Change logical operator (II, &, ) Change arithmetic operator (*, +, -, ) Change constant name / value Change variable name / initialisation Change (or even delete) statement

103 Example Mutants if (a b) c = a + b; else c = 0; if (a && b) c = a + b; else c = 0; if (a b) c = a + b; else c = 0; if (a b) c = a * b; else c = 0;

104 Types of Mutants Stillborn mutants: Syntactically incorrect killed by compiler, e.g., x = a ++ b Trivial mutants: Killed by almost any test case Equivalent mutant: Always acts in the same behaviour as the original program, e.g., x = a + b and x = a (-b) None of the above are interesting from a mutation testing perspective Those mutants are interesting which behave differently than the original program, and we do not (yet) have test cases to identify them (i.e., to cover those specific changes)

105 Equivalent Mutants if (a == 2 && b == 2) c = a + b; else c = 0; if (a == 2 && b == 2) c = a * b; else c = 0; int index=0; int index=0; while (...) while (...) { {...;...; index++; index++; if (index==10) if (index>=10) break; break; } }

106 Program Example nbrs = new int[range] public int max(int[] a) { int imax := 0; for (int i = 1; i <= range; i++) if a[i] > a[imax] imax:= i; return imax; } Program returns the index of the array element with the maximum value. a[0] a[1] a[2] max TC TC TC

107 Variable Name Mutant nbrs = new int[range] public int max(int[] a) { int imax := 0; for (int i = 1; i <= range; i++) if i > a[imax] imax:= i; return imax; a[0] a[1] a[2] max TC TC TC }

108 Relational Operator Mutant nbrs = new int[range] public int max(int[] a) { int imax := 0; for (int i = 1; i <= range; i++) if a[i] >= a[imax] imax:= i; return imax; a[0] a[1] a[2] max TC TC TC } Need a test case with two identical max entries in a[.], e.g., (1, 3, 3)

109 Variable Operator Mutant nbrs = new int[range] public int max(int[] a) { int imax := 0; for (int i = 0; i < range; i++) if a[i] > a[imax] imax:= i; return imax; a[0] a[1] a[2] max TC TC TC } Need a test case detecting wrong loop counting

110 In real life... Fault-based testing is widely used in semiconductor manufacturing With good fault models of typical manufacturing faults, e.g., stuck-at-one for a transistor But fault-based testing for design errors is more challenging (as in software) Mutation testing is not widely used in software industry But plays a role in software testing research, to compare effectiveness of testing techniques Some use of fault models to design test cases is important and widely practiced ( manual fault-seeding)

111 Structure of Lecture 10 Fundamental Definitions and Concepts Creating Test Cases Manually Automatically Assessing the Quality of Test Suites Test Coverage Mutation Testing Static Analysis Quality Prediction

112 Static Analysis Document Review (manual) Different types Static Code Analysis (automatic) Structural properties / metrics

113 Reviews - Terminology Static testing testing without software execution Review meeting to evaluate software artifact Inspection formally defined review Walkthrough author guided review

114 Why Review? Main objective Detect faults Other objectives Inform Educate Learn from (other s) mistakes Improve! (Undetected) faults may affect software quality negatively during all steps of the development process!

115 Relative Cost of Faults Maintenance 200 Source: Davis, A.M., Software Requirements: analysis and specification (1990)

116 Reviews complement testing

117 Inspections Objective: Detect faults Collect data Communicate information Roles Moderator Reviewers (Inspectors) Presenter Author Elements Formal process Planned, structured meeting Preparation important Team (3 to 6 people) Disadvantages Short-term cost

118 Inspection Process

119 Reading Techniques Ad hoc Checklist-based Defect-based Scenario-based Usage-based Perspective-based

120 Perspective-based Reading User Designer Tester Scenarios Purpose Decrease overlap (redundancy) Improve effectiveness

121 Structure of Lecture 10 Fundamental Definitions and Concepts Creating Test Cases Manually Automatically Assessing the Quality of Test Suites Test Coverage Mutation Testing Static Analysis Quality Prediction

122 Quality Prediction Quality def.: Undetected #Defects Based on product and process properties Quality = Function(Code Size Complexity) Quality = Function(Code Changes) Quality = Function(Detected #Defects) Quality = Function(Test Effort) Based on detected defects Capture-Recapture Models Reliability Growth models

123 Capture-Recapture Defect Estimation

124 Capture-Recapture Defect Estimation

125 Capture-Recapture Defect Estimation Situation: Two inspectors are assigned to inspect the same product d 1 : #defects detected by Inspector 1 d 2 : #defects detected by Inspector 2 d 12 : #defects by both inspectors N t : total #defects (detected and undetected) N r : remaining #defects (undetected) N t d 1 d d 12 2 N r N t ( d d d )

126 Capture-Recapture Example Situation: Two inspectors are assigned to inspect the same product d 1 : 50 defects detected by Inspector 1 d 2 : 40 defects detected by Inspector 2 d 12 : 20 defects by both inspectors N t : total defects (detected and undetected) N r : remaining defects (undetected) d d N t N 100 ( ) 30 d r 12

127 Advanced Capture-Recapture Models Four basic models used for inspections Degree of freedom Prerequisites for all models All reviewers work independently of each other It is not allowed to inject or remove faults during inspection

128 Advanced Capture-Recapture Models Model Probability of defect being found is equal across... Defect Reviewer Estimator M0 Yes Yes Maximum-likelihood Mt Yes No Maximum-likelihood Chao s estimator Mh No Yes Jackknife Chao s estimator Mth No No Chao s estimator

129 Mt Model Maximum-likelihood: Mt = total marked animals (=faults) at the start of the t'th sampling interval Ct = total number of animals (=faults) sampled during interval t Rt = number of recaptures in the sample Ct An approximation of the maximum likelihood estimate of population size (N) is: SUM(Ct*Mt)/SUM(Rt) First resampling: M1=50 (first inspector) C1=40 (second inspector) R1=20 N=40*50/20=100 Second resampling: M2=70 (first and second inspector) C2=40 (third inspector) R2=30 N=(40*50+30*70)/(20+30)=4100/50=82 Third resampling: M3=80 C3=30 (fourth inspector) R3=30 N=( *80)/( )=6500/80=81.xx

130 Reliability Growth Models To predict the probability of future failure occurrence based on past (observed) failure occurrence Can be used to estimate the number of residual (remaining) faults or the time until the next failure occurs the remaining test time until a reliability objective is achieved or Application typically during system test

131 Reliability Growth Models (RGMs) 100% 95% Cumulative #Failures (m) m( t ) 0e (estimated n 0 ) t 0 0 t n 0 dt Purpose: Stop testing when a certain percentage (90%, 95%, 99%, 99.9%, ) of estimated total number of failures has been reached a certain failure rate has been reached Test Intensity (t) (CPU time, test effort, test days, calendar days, )

132 Model Selection Many different RGMs have been proposed (>100) To choose a reliability model, perform the following steps: 1. Collect failure data 2. Examine data (failure data vs. test time/effort) 3. Select a set of candidate models 4. Estimate model parameters for each candidate model Least squares method Maximum likelihood method 5. Customize model using the estimated parameters 6. Compare models with goodness-of-fit test and select the best 7. Make reliability predictions with selected model(s)

133 Structure of Lecture 10 Fundamental Definitions and Concepts Creating Test Cases Manually Automatically Assessing the Quality of Test Suites Test Coverage Mutation Testing Static Analysis Quality Prediction More topics: - Test Management - Test Measurement - Test Tools -...

134 Next Lecture Date/Time: Friday, 15-Nov, 10:15-12:00 Topic: Agile / Lean Methods For you to do: Finish Lab Task 5 and submit solutions in time!