CMPT 473 Software Quality Assurance. Graph Coverage. Nick Sumner
|
|
|
- Roderick Young
- 5 years ago
- Views:
Transcription
1 CMPT 473 Software Quality Assurance Graph Coverage Nick Sumner
2 Recall: Coverage/Adequacy Can't look at all possible inputs. Need to determine if a test suite covers / is adequate for our quality objectives. 2
3 Recall: Coverage/Adequacy Can't look at all possible inputs. Need to determine if a test suite covers / is adequate for our quality objectives. So far: Input & Requirements based 3
4 Recall: Coverage/Adequacy Can't look at all possible inputs. Need to determine if a test suite covers / is adequate for our quality objectives. So far: Input & Requirements based Efficiently sort a provided list in under X seconds void sortefficiently(list list) { if (list.size() < THRESHOLD) { sort1(list); } else { sort2(list); } } 4
5 Recall: Coverage/Adequacy Can't look at all possible inputs. Need to determine if a test suite covers / is adequate for our quality objectives. So far: Input & Requirements based Efficiently sort a provided list in under X seconds void sortefficiently(list list) { if (list.size() < THRESHOLD) { } sort1(list); } else { sort2(list); } How well can input based techniques test this? 5
6 Recall: Coverage/Adequacy Can't look at all possible inputs. Need to determine if a test suite covers / is adequate for our quality objectives. So far: Input & Requirements based Efficiently sort a provided list in under X seconds void sortefficiently(list list) { if (list.size() < THRESHOLD) { } sort1(list); } else { sort2(list); } How well can input based techniques test this? How might we do better? 6
7 White Box / Black Blox Considering only the requirements or input is a black box approach Treats the program like an opaque box No deep knowledge of the program's structure 7
8 White Box / Black Blox Considering only the requirements or input is a black box approach Treats the program like an opaque box No deep knowledge of the program's structure Techniques that use artifacts of the program structure are white box approaches They can 'see into' the program's implementation 8
9 White Box Testing What is a simple approach that solves our problem here? void sortefficiently(list list) { if (list.size() < THRESHOLD) { sort1(list); } else { sort2(list); } } 9
10 White Box Testing What is a simple approach that solves our problem here? Statement Coverage How many of the statements did the suite test? void sortefficiently(list list) { if (list.size() < THRESHOLD) { sort1(list); } else { sort2(list); } } 10
11 White Box Testing What is a simple approach that solves our problem here? Statement Coverage How many of the statements did the suite test? Branch Coverage How many of the condition outcomes were tested? void sortefficiently(list list) { if (list.size() < THRESHOLD) { sort1(list); } else { sort2(list); } } 11
12 White Box Testing In this course, we'll mostly look at graph coverage based techniques 12
13 White Box Testing In this course, we'll mostly look at graph coverage based techniques Most commonly used metrics in the real world 13
14 White Box Testing In this course, we'll mostly look at graph coverage based techniques Most commonly used metrics in the real world Most concepts can be modeled through graphs e.g. programs, protocols, use patterns, designs,... So a bit of review... 14
15 Graphs What is a graph G? 15
16 Graphs What is a graph G? A set N of nodes B A C D E 16
17 Graphs What is a graph G? A set N of nodes A set E of edges B A C D E 17
18 Graphs What is a graph G? A set N of nodes A set E of edges When edges are directed from one node to another, the graph is a directed graph B A C D E 18
19 Graphs What is a graph G? A set N of nodes A set E of edges When edges are directed from one node to another, the graph is a directed graph A path is a list of pairwise connected nodes B A C D E 19
20 Graphs What is a graph G? A set N of nodes A set E of edges When edges are directed from one node to another, the graph is a directed graph A path is a list of pairwise connected nodes ABCDE B A C D E 20
21 Graphs What is a graph G? A set N of nodes A set E of edges When edges are directed from one node to another, the graph is a directed graph A path is a list of pairwise connected nodes ABCDE ABDE AE BCDE BD C B D A E 21
22 Control Flow Graphs (CFGs) Programs can be modeled as graphs Used extensively in compilers 22
23 Control Flow Graphs (CFGs) Programs can be modeled as graphs Used extensively in compilers Also used in testing! 23
24 Control Flow Graphs (CFGs) Programs can be modeled as graphs Used extensively in compilers Also used in testing! Control Flow Graphs Nodes comprise the code of a program 24
25 Control Flow Graphs (CFGs) Programs can be modeled as graphs Used extensively in compilers Also used in testing! Control Flow Graphs Nodes comprise the code of a program Edges show the paths that an execution may take through a program 25
26 Control Flow Graphs Example: void sortefficiently(list list) { if (list.size() < THRESHOLD) { sort1(list); } else { sort2(list); } } 26
27 Control Flow Graphs Example: void sortefficiently(list list) { if (list.size() < THRESHOLD) { sort1(list); } else { sort2(list); } } if (list.size() < THRESHOLD) sort1(list); sort2(list); return; 27
28 Control Flow Graphs Example: void sortefficiently(list list) { if (list.size() < THRESHOLD) { sort1(list); } else { sort2(list); } } if (list.size() < THRESHOLD) sort1(list); sort2(list); return; 28
29 Control Flow Graphs Example: void sortefficiently(list list) { if (list.size() < THRESHOLD) { sort1(list); } else { sort2(list); } } if (list.size() < THRESHOLD) size < THRESHOLD sort1(list); size >= THRESHOLD sort2(list); return; 29
30 Control Flow Graphs Example: void sortefficiently(list list) { if (list.size() < THRESHOLD) { sort1(list); } else { sort2(list); } } if (list.size() < THRESHOLD) size < THRESHOLD sort1(list); size >= THRESHOLD sort2(list); return; 30
31 Control Flow Graphs Many types of nodes: list =... if (list.size() < THRESHOLD) size < THRESHOLD size >= THRESHOLD sort1(list); sort2(list); print(list); return; 31
32 Control Flow Graphs Many types of nodes: list =... Entry Node 0 incoming edges if (list.size() < THRESHOLD) size < THRESHOLD size >= THRESHOLD sort1(list); sort2(list); print(list); return; 32
33 Control Flow Graphs Many types of nodes: list =... Entry Node 0 incoming edges if (list.size() < THRESHOLD) size < THRESHOLD size >= THRESHOLD sort1(list); sort2(list); print(list); return; Exit Node 0 outgoing edges 33
34 Control Flow Graphs Entry Node 0 incoming list =... edges Decision/Branch Node >1 outgoing edges if (list.size() < THRESHOLD) size < THRESHOLD size >= THRESHOLD Many types of nodes: sort1(list); sort2(list); print(list); return; Exit Node 0 outgoing edges 34
35 Control Flow Graphs Many types of nodes: Decision/Branch Node >1 outgoing edges list =... Entry Node 0 incoming edges if (list.size() < THRESHOLD) size < THRESHOLD size >= THRESHOLD sort1(list); sort2(list); Join Node >1 incoming edges print(list); return; Exit Node 0 outgoing edges 35
36 Control Flow Graphs Straight-line sequences of code are grouped into basic blocks: list =... if (list.size() < THRESHOLD) size < THRESHOLD size >= THRESHOLD sort1(list); sort2(list); print(list); return; 36
37 Control Flow Graphs Straight-line sequences of code are grouped into basic blocks: list =... if (list.size() < THRESHOLD) size < THRESHOLD size >= THRESHOLD sort1(list); sort2(list); print(list); return; 37
38 Control Flow Graphs Many patterns arise from common constructs 38
39 Control Flow Graphs Many patterns arise from common constructs What is the CFG for this program? q = 0; r = x; while r >= y { r = r - y; q = q + 1; } Tell me how to draw it. 39
40 Control Flow Graphs What is the CFG? for (i = 0; i < n; i++) { foo(i); } 40
41 Control Flow Graphs What is the CFG? Don't forget implicit behavior like default! switch (x) { case a: foo(x); break; case b: bar(x); case c: baz(x); break; } 41
42 Control Flow Graphs What is the CFG? Short circuit operators can lead to subtle behavior! if (x == 0 y/x > 1) { foo(x, y); } 42
43 Control Flow Graphs What is the CFG? Control flow can become complex! From Ammann & Offutt: x = 0; while (x < y) { y = f (x, y); if (y == 0) { break; } else if (y < 0) { y = y*2; continue; } x = x + 1; } print (y); 43
44 Control Flow Graphs What is the CFG? Control flow can become complex! From Ammann & Offutt: x = 0; while (x < y) { y = f (x, y); if (y == 0) { break; } else if (y < 0) { y = y*2; continue; } x = x + 1; } print (y); 44
45 Control Flow Graphs What is the CFG? Control flow can become complex! From Ammann & Offutt: x = 0; while (x < y) { y = f (x, y); if (y == 0) { break; } else if (y < 0) { y = y*2; continue; } x = x + 1; } print (y); 45
46 Control Flow Graphs What is the CFG? Control flow can become complex! From Ammann & Offutt: x = 0; while (x < y) { y = f (x, y); if (y == 0) { break; } else if (y < 0) { y = y*2; continue; } x = x + 1; } print (y); 46
47 Control Flow Graphs What is the CFG? Control flow can become complex! From Ammann & Offutt: x = 0; while (x < y) { y = f (x, y); if (y == 0) { break; } else if (y < 0) { y = y*2; continue; } x = x + 1; } print (y); 47
48 Control Flow Graphs What is the CFG? Control flow can become complex! From Ammann & Offutt: x = 0; while (x < y) { y = f (x, y); if (y == 0) { break; } else if (y < 0) { y = y*2; continue; } x = x + 1; } print (y); 48
49 Control Flow Graphs What is the CFG? Control flow can become complex! From Ammann & Offutt: x = 0; while (x < y) { y = f (x, y); if (y == 0) { break; } else if (y < 0) { y = y*2; continue; } x = x + 1; } print (y); 49
50 Control Flow Graphs What is the CFG? Control flow can become complex! From Ammann & Offutt: x = 0; while (x < y) { y = f (x, y); if (y == 0) { break; } else if (y < 0) { y = y*2; continue; } x = x + 1; } print (y); 50
51 Control Flow Graphs What is the CFG? Control flow can become complex! From Ammann & Offutt: x = 0; while (x < y) { y = f (x, y); if (y == 0) { break; } else if (y < 0) { y = y*2; continue; } x = x + 1; } print (y); 51
52 Control Flow Graphs What is the CFG? Control flow can become complex! From Ammann & Offutt: x = 0; while (x < y) { y = f (x, y); if (y == 0) { break; } else if (y < 0) { y = y*2; continue; } x = x + 1; } print (y); 52
53 CFG Coverage Statement Coverage Node Coverage Try to cover all reachable basic blocks. 53
54 CFG Coverage Statement Coverage Node Coverage Thinking in terms of node coverage can be more efficient. Why? 54
55 CFG Coverage Statement Coverage Node Coverage Branch Coverage Edge Coverage Try to cover all reachable paths of length 1. 55
56 CFG Coverage Statement Coverage Node Coverage Branch Coverage Edge Coverage How do node & edge coverage compare? Why? 56
57 CFG Coverage Statement Coverage Node Coverage Branch Coverage Edge Coverage How do node & edge coverage compare? Why? Are these notions of coverage enough? Why? 57
58 Pragmatic Concerns The goal is full coverage (of whatever criteria) 58
59 Pragmatic Concerns The goal is full coverage (of whatever criteria) Is that reasonable in practice? Why? 59
60 Pragmatic Concerns The goal is full coverage (of whatever criteria) We must consider reachability 60
61 Pragmatic Concerns The goal is full coverage (of whatever criteria) We must consider reachability Try to cover all reachable basic blocks. Try to cover all reachable paths of length 1. 61
62 Pragmatic Concerns The goal is full coverage (of whatever criteria) We must consider reachability Syntactic Reachability Based on the structure of the code ENTER... print( Got Here ) return; return; 62
63 Pragmatic Concerns The goal is full coverage (of whatever criteria) We must consider reachability Syntactic Reachability Based on the structure of the code ENTER How could this happen?... print( Got Here ) return; return; 63
64 Pragmatic Concerns The goal is full coverage (of whatever criteria) We must consider reachability Syntactic Reachability Based on the structure of the code Semantic Reachability Based on the meaning of the code condition = false; if (condition) return; 64
65 Pragmatic Concerns The goal is full coverage (of whatever criteria) We must consider reachability Syntactic Reachability Based on the structure of the code Semantic Reachability Based on the meaning of the code condition = false; if (condition) This can be undecidable! return; 65
66 Pragmatic Concerns The goal is full coverage (of whatever criteria) We must consider reachability Syntactic Reachability Based on the structure of the code Semantic Reachability Based on the meaning of the code So what do you do in practice? No, really. What have you done in practice? 66
67 Pragmatic Concerns The goal is full coverage (of whatever criteria) We must consider reachability Syntactic Reachability Based on the structure of the code Semantic Reachability Based on the meaning of the code So what do you do in practice? No, really. What have you done in practice? Relative degrees of coverage matter (40%? 80%?) 67
68 CFG Coverage The path taken by each test can matter if (condition 1)... x = 0; if (condition2)... z = y/x; return; 68
69 CFG Coverage The path taken by each test can matter if (condition 1)... x = 0; if (condition2)... z = y/x; return; 69
70 CFG Coverage The path taken by each test can matter if (condition 1)... x = 0; if (condition2)... z = y/x; return; 70
71 CFG Coverage The path taken by each test can matter if (condition 1) Full edge coverage & no bugs found... x = 0; if (condition2)... z = y/x; return; 71
72 CFG Coverage The path taken by each test can matter if (condition 1) How can we make sure to find this bug?... x = 0; if (condition2)... z = y/x; return; 72
73 CFG Coverage The path taken by each test can matter if (condition 1)... x = 0; if (condition2)... z = y/x; return; 73
74 CFG Coverage The path taken by each test can matter if (condition 1)... x = 0; if (condition2)... z = y/x; return; 74
75 CFG Coverage The path taken by each test can matter if (condition 1) Testing all paths exposes this bug.... x = 0; if (condition2)... z = y/x; return; 75
76 Path Coverage Complete Path Coverage Test all paths through the graph 76
77 Path Coverage Complete Path Coverage Test all paths through the graph Is this reasonable? Why? 77
78 Path Coverage Complete Path Coverage Test all paths through the graph Is this reasonable? Why? A B D C 78
79 Path Coverage Complete Path Coverage Test all paths through the graph Is this reasonable? Why? A D B C Infeasibility 79
80 Path Coverage Complete Path Coverage Test all paths through the graph Is this reasonable? Why? D A How many paths? B How does this relate to input Infeasibility based C approaches? A B C D E F... G H... I K J 80
81 Path Coverage Complete Path Coverage Test all paths through the graph Is this reasonable? Why? D A How many paths? B How does this relate to input Infeasibility based C approaches? A B C D E F... G H... Intractability I K J 81
82 Compromises? What could we do instead? (How did we handle the input based approaches?) 82
83 Compromises? Edge Pair Coverage Each path of length <= 2 is tested. Specified Path Coverage Given a number k, test k paths 83
84 Compromises? Edge Pair Coverage Each path of length <= 2 is tested. Specified Path Coverage Given a number k, test k paths What do these look like? A B A B C D E F... G H... D C I K J 84
85 Compromises? Edge Pair Coverage Each path of length <= 2 is tested. Specified Path Coverage Given a number k, test k paths What do these look like? Are they good? A B A B C D E F... G H... D C I K J 85
86 Coping With Loops What criteria do you use when testing loops? 86
87 Coping With Loops What criteria do you use when testing loops? Simple Paths A path between nodes is simple if no node appears more than once. (Except maybe the first and last) Captures the acyclic behaviors of a program 87
88 Coping With Loops What criteria do you use when testing loops? Simple Paths A path between nodes is simple if no node appears more than once. (Except maybe the first and last) Captures the acyclic behaviors of a program A How many may there be? B C D 88
89 Coping With Loops What criteria do you use when testing loops? Simple Paths A path between nodes is simple if no node appears more than once. (Except maybe the first and last) Captures the acyclic behaviors of a program A How many may there be? B C D 89
90 Coping With Loops What criteria do you use when testing loops? Simple Paths A path between nodes is simple if no node appears more than once. (Except maybe the first and last) Captures the acyclic behaviors of a program A How many may there be? B C D 90
91 Coping With Loops What criteria do you use when testing loops? Simple Paths A path between nodes is simple if no node appears more than once. (Except maybe the first and last) Captures the acyclic behaviors of a program Prime Paths A simple path that is not a subpath of any other simple path 91
92 Coping With Loops What criteria do you use when testing loops? Simple Paths A path between nodes is simple if no node appears more than once. (Except maybe the first and last) Captures the acyclic behaviors of a program Prime Paths A simple path that is not a subpath of any other simple path B A D C What does this provide? What do they look like? 92
93 Coping With Loops Prime Path Coverage Cover all prime paths 93
94 Coping With Loops Prime Path Coverage Cover all prime paths A B D C E G F Example from Ammann & Offutt 94
95 Coping With Loops Prime Path Coverage Cover all prime paths A What are the prime paths? B D C E G F Example from Ammann & Offutt 95
96 Coping With Loops Prime Path Coverage Cover all prime paths B D A C E What are the prime paths? How many simple paths? G F Example from Ammann & Offutt 96
97 Coping With Loops Prime Path Coverage Cover all prime paths A What are the prime paths? B C How many simple paths? D G E F Can you intuitively explain what prime paths capture? Example from Ammann & Offutt 97
98 Coping With Loops Prime Path Coverage Cover all prime paths Are these tests good or bad? 98
99 Coping With Loops Prime Path Coverage Cover all prime paths Are these tests good or bad? Do they address all of the problems with path coverage? 99
100 Coping With Loops Prime Path Coverage Cover all prime paths Are these tests good or bad? Do they address all of the problems with path coverage? Can you think of things they miss? 100
101 How Do They Relate? All / Complete Path Coverage Prime Path Coverage How do they compare w/ edge coverage? 101
102 How Do They Relate? All / Complete Path Coverage Prime Path Coverage Edge Coverage Node Coverage 102
103 How Do They Relate? All / Complete Path Coverage Prime Path Coverage Edge Coverage How do the compare w/ edge pair coverage? Node Coverage 103
104 How Do They Relate? All / Complete Path Coverage Edge Pair Coverage Prime Path Coverage Edge Coverage Node Coverage 104
105 How Do They Relate? All / Complete Path Coverage Edge Pair Coverage Prime Path Coverage Edge Coverage Node Coverage Consider: A B 105
106 How Do They Relate? All / Complete Path Coverage Edge Pair Coverage Prime Path Coverage Edge Coverage Why did we look at prime paths at all? Node Coverage 106
107 How Do They Relate? All / Complete Path Coverage Edge Pair Coverage Prime Path Coverage Edge Coverage Node Coverage Why did we look at prime paths at all? So let's only consider loops
108 How Do They Relate? Round Trip All / Complete Path Coverage A prime path starting and ending with the same node Edge Pair Coverage Prime Path Coverage Edge Coverage Node Coverage Complete Round Trip Coverage Simple Round Trip Coverage 108
109 How Do They Relate? All / Complete Path Coverage Edge Pair Coverage Prime Path Coverage Edge Coverage Complete Round Trip Coverage Node Coverage All round trips for each node 109
110 How Do They Relate? All / Complete Path Coverage Edge Pair Coverage Prime Path Coverage Edge Coverage Node Coverage Just one round trip for each node Complete Round Trip Coverage Simple Round Trip Coverage 110
111 Turning Them Into Tests Reconsider: A B D C G E F 111
112 Turning Them Into Tests Reconsider: B D A C G E F Is this path prime? 112
113 Turning Them Into Tests Reconsider: B A Is this path prime? D C E Is it still useful? G F 113
114 Turning Them Into Tests Reconsider: B A Is this path prime? D C E Is it still useful? G F One test may cover multiple prime paths! Requirements Tests 114
115 Turning Them Into Tests Relaxing our path requirements can help, too 115
116 Turning Them Into Tests Relaxing our path requirements can help, too Tour A path p tours path q if q is a subpath of p A test covers any prime path it tours 116
117 Turning Them Into Tests Relaxing our path requirements can help, too Tour A path p tours path q if q is a subpath of p A test covers any prime path it tours This is strict! Can we relax it? 117
118 Turning Them Into Tests Relaxing our path requirements can help, too Tour A path p tours path q if q is a subpath of p A test covers any prime path it tours Tour with sidetrips Iff every edge of q appears in the same order in p Return to where you left 118
119 Turning Them Into Tests Relaxing our path requirements can help, too Tour A path p tours path q if q is a subpath of p A test covers any prime path it tours Tour with sidetrips Iff every edge of q appears in the same order in p Return to where you left Tour with detours Iff every node of q appears in the same order in p 119
120 Turning Them Into Tests A B C D E Strict Tour: ABCDE F 120
121 Turning Them Into Tests A B C D E Strict Tour: ABCDE F 121
122 Turning Them Into Tests A B C D E Strict Tour: ABCDE F A B C D E With Sidetrips? F 122
123 Turning Them Into Tests A B C D E Strict Tour: ABCDE F A B C D E With Sidetrips? F 123
124 Turning Them Into Tests A B C D E Strict Tour: ABCDE F A B C D E With Sidetrips? F A B C D E With Detours? F 124
125 Turning Them Into Tests A B C D E Strict Tour: ABCDE F A B C D E With Sidetrips? F A B C D E With Detours? F 125
126 Turning Them Into Tests Do these relaxations help us with problems we have seen? 126
127 Turning Them Into Tests Do these relaxations help us with problems we have seen? Can you see any problems they may introduce? 127
128 Turning Them Into Tests Do these relaxations help us with problems we have seen? Can you see any problems they may introduce? How might this affect how you use them? 128
129 Onward But sometimes the structure of the CFG is not enough
Introduction to Software Testing Chapter 2, Sections: 2.1 & 2.2 Overview Graph Coverage Criteria
Introduction to Software Testing Chapter 2, Sections: 2.1 & 2.2 Overview Graph Coverage Criteria Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/softwa retest/ Ch. 2 : Graph Coverage Four Structures
Introduction to Software Testing Chapter 2.1, 2.2 Overview Graph Coverage Criteria Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 2.1, 2.2 Overview Graph Coverage Criteria Paul Ammann & Jeff Offutt www.introsoftwaretesting.com Ch. 2 : Graph Coverage Four Structures for Modeling Software Graphs
Testing Object Oriented Software 2010
Testing Object Oriented Software 2010 Graph Coverage Barry van Veen Rick van der Zwet 2.1 Graph coverage Introduction into graph coverage Theoretical definition of
Introduction to Software Testing Chapter 2.3 Graph Coverage for Source Code. Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 2.3 Graph Coverage for Source Code Paul Ammann & Jeff Offutt Overview The most common application of graph criteria is to program source Graph : Usually the control
Overview Graph Coverage Criteria
Overview Graph Coverage Criteria Graph Coverage Four Structures for Modeling Software Graphs Logic Input Space Syntax Applied to Applied to Source FSMs Applied to Specs DNF Source Specs Source Models Design
Introduc)on to So,ware Tes)ng (2nd edi(on) Overview Graph Coverage Criteria. Paul Ammann & Jeff OffuA. hap:// so,waretest/
Tes)ng (2nd edi(on) Overview Graph Coverage Criteria Paul Ammann & Jeff OffuA hap://www.cs.gmu.edu/~offua/ so,waretest/ First version, 23 September 2013 Graph Coverage Four Structures for Modeling Software
c. Typically results in an intractably large set of test cases even for small programs
Multiple-Choice Questions: 1. True or false? Generally, in practice, developers exhaustively test software. a. True b. False 2. True or false? All real software contains bugs. a. True b. False 3. Which
Testing: Coverage and Structural Coverage
Testing: Coverage and Structural Coverage Testing, Quality Assurance, and Maintenance Winter 2017 Prof. Arie Gurfinkel based on slides by Prof. Marsha Chechik and Prof. Lin Tan How would you test this
Logic Coverage for Source Code
Logic Coverage for Source Code CS 4501 / 6501 Software Testing [Ammann and Offutt, Introduction to Software Testing, Ch. 8] 1 Structural Logic Coverage for Source Code Aim: to identify test requirements
A PROGRAM IS A SEQUENCE of instructions that a computer can execute to
A PROGRAM IS A SEQUENCE of instructions that a computer can execute to perform some task. A simple enough idea, but for the computer to make any use of the instructions, they must be written in a form
Introduction to Software Testing Chapter 2, Sec#: 2.5 Graph Coverage for Specifications
Introduction to Software Testing Chapter 2, Sec#: 2.5 Graph Coverage for Specifications Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/softwa retest/ Design Specifications A design specification
COMP 3705 Advanced Software Engineering: Software Testing
COMP 3705 Advanced Software Engineering: Software Testing Prof. Matt Rutherford For Next Week Readings For Midterm: Chapter 1 and Sections 2.1-2.6, 3.1 3.5 New material: Chapter 4 Homework #5 http://mjrutherford.org/teaching/2009/winter/comp3705/homeworks
Structural Testing. (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 1
Structural Testing (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 1 Learning objectives Understand rationale for structural testing How structural (code-based or glass-box) testing complements functional
Properties of Criteria. 1. Applicability Property. Criteria
Properties of Criteria Program-based To recognize a good adequacy criteria And to discard poor choices Objective, well-defined properties 1. Applicability Property For every program, there exists an adequate
Introduction to Dynamic Analysis
Introduction to Dynamic Analysis Reading assignment Gary T. Leavens, Yoonsik Cheon, "Design by Contract with JML," draft paper, http://www.eecs.ucf.edu/~leavens/jml//jmldbc.pdf G. Kudrjavets, N. Nagappan,
Introduction to Software Testing Chapter 5.1 Syntax-based Testing
Introduction to Software Testing Chapter 5.1 Syntax-based Testing Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/ softwaretest/ Ch. 5 : Syntax Coverage Four Structures for Modeling Software Graphs
INTRODUCTION TO SOFTWARE ENGINEERING
INTRODUCTION TO SOFTWARE ENGINEERING Structural Testing d_sinnig@cs.concordia.ca Department for Computer Science and Software Engineering Introduction Testing is simple all a tester needs to do is find
Testing: Coverage and Structural Coverage
Testing: Coverage and Structural Coverage Testing, Quality Assurance, and Maintenance Winter 2018 Prof. Arie Gurfinkel based on slides by Prof. Marsha Chechik and Prof. Lin Tan How would you test this
Problem Solving through Programming In C Prof. Anupam Basu Department of Computer Science & Engineering Indian Institute of Technology, Kharagpur
Problem Solving through Programming In C Prof. Anupam Basu Department of Computer Science & Engineering Indian Institute of Technology, Kharagpur Lecture 15 Branching : IF ELSE Statement We are looking
Graph Coverage for Source Code. Data Flow Graph Coverage for Source Code
Graph Coverage for Source Code Data Flow Graph Coverage for Source Code 1 Graph Coverage for Design Elements Use of data abstraction and object oriented software has increased importance on modularity
Reading assignment: Reviews and Inspections
Foundations for SE Analysis Reading assignment: Reviews and Inspections M. E. Fagan, "Design and code inspections to reduce error in program development, IBM Systems Journal, 38 (2&3), 1999, pp. 258-287.
Page 1. Reading assignment: Reviews and Inspections. Foundations for SE Analysis. Ideally want general models. Formal models
Reading assignment: Reviews and Inspections Foundations for SE Analysis M. E. Fagan, "Design and code inspections to reduce error in program development, IBM Systems Journal, 38 (2&3), 999, pp. 258-28.
Software Testing Prof. Rajib Mall Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur. Lecture 13 Path Testing
Software Testing Prof. Rajib Mall Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture 13 Path Testing Welcome to this session and we will discuss about path
Introduction to Software Testing Chapter 3, Sec# 1 & 2 Logic Coverage
Introduction to Software Testing Chapter 3, Sec# 1 & 2 Logic Coverage Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/soft waretest/ Ch. 3 : Logic Coverage Four Structures for Modeling Software
printf( Please enter another number: ); scanf( %d, &num2);
CIT 593 Intro to Computer Systems Lecture #13 (11/1/12) Now that we've looked at how an assembly language program runs on a computer, we're ready to move up a level and start working with more powerful
Input Space Partitioning
CMPT 473 Software Quality Assurance Input Space Partitioning Nick Sumner - Fall 2014 With material from Patrick Lam, Jeff Offutt Recall Testing involves running software and comparing observed behavior
Testing: Dataflow Coverage
Testing: Dataflow Coverage Testing, Quality Assurance, and Maintenance Winter 2018 Arie Gurfinkel based on slides by Thibaud Lutellier, Marsha Chechik and Prof. Lin Tan Non-looping Path Selection Problem
Path Testing + Coverage. Chapter 8
Path Testing + Coverage Chapter 8 Structural Testing n Also known as glass/white/open box testing n A software testing technique whereby explicit knowledge of the internal workings of the item being tested
CS107 Handout 08 Spring 2007 April 9, 2007 The Ins and Outs of C Arrays
CS107 Handout 08 Spring 2007 April 9, 2007 The Ins and Outs of C Arrays C Arrays This handout was written by Nick Parlante and Julie Zelenski. As you recall, a C array is formed by laying out all the elements
Introduction au Génie Logiciel Partie3: Test: Static White-Box Test
Licence Mention Informatique L3/S6 2009/10 Introduction au Génie Logiciel Partie3: Test: Static White-Box Test Burkhart Wolff Département Informatique Difficulties with Static Unit Tests so far The generation
Introduction to Problem Solving and Programming in Python.
Introduction to Problem Solving and Programming in Python http://cis-linux1.temple.edu/~tuf80213/courses/temple/cis1051/ Overview Types of errors Testing methods Debugging in Python 2 Errors An error in
Testing & Symbolic Execution
Testing & Symbolic Execution Software Testing The most common way of measuring & ensuring correctness Input 2 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior
C07: Testing and JUnit
CISC 3120 C07: Testing and JUnit Hui Chen Department of Computer & Information Science CUNY Brooklyn College 9/19/2017 CUNY Brooklyn College 1 Outline Recap and issues Grades and feedback Assignments &
Génie Logiciel Avancé
L3 Mention Informatique Parcours Informatique et MIAGE Génie Logiciel Avancé Part VII : White-Box Test Burkhart Wolff wolff@lri.fr Idea: % Lets exploit the structure of the program!!! (and not, as before
Coding and Unit Testing! The Coding Phase! Coding vs. Code! Coding! Overall Coding Language Trends!
Requirements Spec. Design Coding and Unit Testing Characteristics of System to be built must match required characteristics (high level) Architecture consistent views Software Engineering Computer Science
Testing, Fuzzing, & Symbolic Execution
Testing, Fuzzing, & Symbolic Execution Software Testing The most common way of measuring & ensuring correctness Input 2 Software Testing The most common way of measuring & ensuring correctness Input Observed
(How Not To Do) Global Optimizations
(How Not To Do) Global Optimizations #1 One-Slide Summary A global optimization changes an entire method (consisting of multiple basic blocks). We must be conservative and only apply global optimizations
Warmup : Name that tune!
Warmup : Name that tune! Write, using a loop, Java code to print the lyrics to the song 99 Bottles of Beer on the Wall 99 bottles of beer on the wall. 99 bottles of beer. Take one down, pass it around,
Testing Role in Unified Approach Coverage: Structural/Coverage Model Based Test Generation from Model Checking (project) Interaction of
Testing Role in Unified Approach Coverage: Structural/Coverage Model Based Test Generation from Model Checking (project) Interaction of Coverage/Model Based Testing Will Not Cover Statistical Methods Partition
CS 520 Theory and Practice of Software Engineering Fall 2018
Today CS 52 Theory and Practice of Software Engineering Fall 218 Software testing October 11, 218 Introduction to software testing Blackbox vs. whitebox testing Unit testing (vs. integration vs. system
Program Testing and Analysis: Manual Testing Prof. Dr. Michael Pradel Software Lab, TU Darmstadt
Program Testing and Analysis: Manual Testing Prof. Dr. Michael Pradel Software Lab, TU Darmstadt Partly based on slides from Peter Müller, ETH Zurich 1 Warm-up Quiz What does the following code print?
Input Space Partitioning
CMPT 473 Software Quality Assurance Input Space Partitioning Nick Sumner Recall Testing involves running software and comparing observed behavior against expected behavior Select an input, look at the
Subject Software Testing Structural Testing
Subject Software Testing Structural Testing Objective: 1. Understand Concept of structural testing 2. How structural (code-based or glass-box) testing complements functional (black-box) testing 3. Recognize
[IT6004-SOFTWARE TESTING] UNIT 2
![[IT6004-SOFTWARE TESTING] UNIT 2](/thumbs/90/102012046.jpg "[IT6004-SOFTWARE TESTING] UNIT 2")
1. List the two basic Testing strategies. UNIT 2 Black box testing. White box testing. 2. What are the knowledge sources for Black box testing? Requirements Document specification Domain knowledge Defect
Problem Solving through Programming In C Prof. Anupam Basu Department of Computer Science & Engineering Indian Institute of Technology, Kharagpur
Problem Solving through Programming In C Prof. Anupam Basu Department of Computer Science & Engineering Indian Institute of Technology, Kharagpur Lecture 18 Switch Statement (Contd.) And Introduction to
UNIT I Programming Language Syntax and semantics. Kainjan Sanghavi
UNIT I Programming Language Syntax and semantics B y Kainjan Sanghavi Contents Bird s eye view of programming language concepts Syntax Semantics Pragmatics Programming Language Concepts A programming language
Using the code to measure test adequacy (and derive test cases) Structural Testing
Using the code to measure test adequacy (and derive test cases) Structural Testing Objectives To describe a second approach to testing which is geared to find program defects To explain the use of program
CONTENTS: While loops Class (static) variables and constants Top Down Programming For loops Nested Loops
COMP-202 Unit 4: Programming with Iterations Doing the same thing again and again and again and again and again and again and again and again and again... CONTENTS: While loops Class (static) variables
AXIOMS OF AN IMPERATIVE LANGUAGE PARTIAL CORRECTNESS WEAK AND STRONG CONDITIONS. THE AXIOM FOR nop
AXIOMS OF AN IMPERATIVE LANGUAGE We will use the same language, with the same abstract syntax that we used for operational semantics. However, we will only be concerned with the commands, since the language
3/18/18. Program Analysis. CYSE 411/AIT 681 Secure Software Engineering. Learning Goal. Program Analysis on Security. Why Program Representations
Program Analysis CYSE 411/AIT 681 Secure Software Engineering Topic #14. Program Analysis Instructor: Dr. Kun Sun The process of automatically analyzing the behavior of computer programs regarding a property
CMPSCI 521/621 Homework 2 Solutions
CMPSCI 521/621 Homework 2 Solutions Problem 1 Direct data dependencies: 3 is directly data dependent on 1 and 5 5 is directly data dependent on 1,3, and 5 7 is directly data dependent on 1,3, and 5 Note,
18-642: Unit Testing 1/31/ Philip Koopman
18-642: Unit Testing 1/31/2018 2017-2018 Philip Koopman YOU ARE HERE Product Requirements SPECIFY PRODUCT SPECIFY SOFTWARE Software Requirements TRACEABILITY & VALIDATION Test Plan & Test Results Test
Topics in Software Testing
Dependable Software Systems Topics in Software Testing Material drawn from [Beizer, Sommerville] Software Testing Software testing is a critical element of software quality assurance and represents the
Testing. ECE/CS 5780/6780: Embedded System Design. Why is testing so hard? Why do testing?
Testing ECE/CS 5780/6780: Embedded System Design Scott R. Little Lecture 24: Introduction to Software Testing and Verification What is software testing? Running a program in order to find bugs (faults,
MSc Software Testing and Maintenance MSc Prófun og viðhald hugbúnaðar
MSc Software Testing and Maintenance MSc Prófun og viðhald hugbúnaðar Fyrirlestrar 31 & 32 Structural Testing White-box tests. 27/1/25 Dr Andy Brooks 1 Case Study Dæmisaga Reference Structural Testing
Introduction to Software Testing Chapter 3.1, 3.2 Logic Coverage Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 3.1, 3.2 Logic Coverage Paul Ammann & Jeff Offutt www.introsoftwaretesting.com Ch. 3 : Logic Coverage Four Structures for Modeling Software Graphs Logic Input Space
Introduction to Software Testing Chapter 3.1, 3.2 Logic Coverage Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 3.1, 3.2 Logic Coverage Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/softwaretest/ Ch. 3 : Logic Coverage Four Structures for Modeling Software Graphs
Overview. State-of-the-Art. Relative cost of error correction. CS 619 Introduction to OO Design and Development. Testing.
Overview CS 619 Introduction to OO Design and Development ing! Preliminaries! All sorts of test techniques! Comparison of test techniques! Software reliability Fall 2012! Main issues: There are a great
CYSE 411/AIT 681 Secure Software Engineering Topic #14. Program Analysis
CYSE 411/AIT 681 Secure Software Engineering Topic #14. Program Analysis Instructor: Dr. Kun Sun Program Analysis The process of automatically analyzing the behavior of computer programs regarding a property
Introduction to Programming in C Department of Computer Science and Engineering. Lecture No. #13. Loops: Do - While
Introduction to Programming in C Department of Computer Science and Engineering Lecture No. #13 Loops: Do - While So far we have been using while loops in C, now C programming language also provides you
How to approach a computational problem
How to approach a computational problem A lot of people find computer programming difficult, especially when they first get started with it. Sometimes the problems are problems specifically related to
Compiler Design Prof. Y. N. Srikant Department of Computer Science and Automation Indian Institute of Science, Bangalore
Compiler Design Prof. Y. N. Srikant Department of Computer Science and Automation Indian Institute of Science, Bangalore Module No. # 10 Lecture No. # 16 Machine-Independent Optimizations Welcome to the
CFG (Control flow graph)
CFG (Control flow graph) Class B T12 오지은 200814189 신승우 201011340 이종선 200811448 Introduction to CFG Algorithm to construct Control Flow Graph Statement of Purpose Q & A Introduction to CFG Algorithm to
MTAT : Software Testing
MTAT.03.159: Software Testing Lecture 03: White-Box Testing (Textbook Ch. 5) Dietmar Pfahl Spring 2017 email: dietmar.pfahl@ut.ee Lecture Chapter 5 White-box testing techniques (Lab 3) Structure of Lecture
Software Engineering 2 A practical course in software engineering. Ekkart Kindler
Software Engineering 2 A practical course in software engineering Quality Management Main Message Planning phase Definition phase Design phase Implem. phase Acceptance phase Mainten. phase 3 1. Overview
EXAMINING THE CODE. 1. Examining the Design and Code 2. Formal Review: 3. Coding Standards and Guidelines: 4. Generic Code Review Checklist:
EXAMINING THE CODE CONTENTS I. Static White Box Testing II. 1. Examining the Design and Code 2. Formal Review: 3. Coding Standards and Guidelines: 4. Generic Code Review Checklist: Dynamic White Box Testing
Foundations: Syntax, Semantics, and Graphs
Foundations: Syntax, Semantics, and Graphs Testing, Quality Assurance, and Maintenance Winter 2018 Prof. Arie Gurfinkel based on slides by Ruzica Pizkac, Claire Le Goues, Lin Tan, Marsha Chechik, and others
Comp 151. Control structures.
Comp 151 Control structures. admin For these slides read chapter 7 Yes out of order. Simple Decisions So far, we ve viewed programs as sequences of instructions that are followed one after the other. While
Engin 100 (section 250), Winter 2015, Technical Lecture 3 Page 1 of 5. Use pencil!
Engin 100 (section 250), Winter 2015, Technical Lecture 3 Page 1 of 5 Use pencil! Last time Introduced basic logic and some terms including bus, word, register and combinational logic. Talked about schematic
Program-based Mutation Testing
Program-based Mutation Testing CS 4501 / 6501 Software Testing [Ammann and Offutt, Introduction to Software Testing, Ch. 9.2] 1 Applying Syntax-Based Testing to Programs Test requirements are derived from
Testing & Continuous Integration. Kenneth M. Anderson University of Colorado, Boulder CSCI 5828 Lecture 20 03/19/2010
esting & Continuous Integration Kenneth M. Anderson University of Colorado, Boulder CSCI 5828 Lecture 20 03/1/20 University of Colorado, 20 1 Goals 2 Review material from Chapter of Pilone & Miles esting
CS 510/12. Xiangyu Zhang
CS 510/12 Program Representation Xiangyu Zhang Why Program Representations Original representations Source code (cross languages). Binaries (cross machines and platforms). Source code / binaries + test
Software Testing Lecture 1. Justin Pearson
Software Testing Lecture 1 Justin Pearson 2017 1 / 50 Four Questions Does my software work? 2 / 50 Four Questions Does my software work? Does my software meet its specification? 3 / 50 Four Questions Does
Introduction to Software Testing Chapter 4 Input Space Partition Testing
Introduction to Software Testing Chapter 4 Input Space Partition Testing Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/ softwaretest/ Ch. 4 : Input Space Coverage Four Structures for Modeling
MTAT : Software Testing
MTAT.03.159: Software Testing Lecture 03: White-Box Testing (Textbook Ch. 5) Dietmar Pfahl Spring 2016 email: dietmar.pfahl@ut.ee Lecture Chapter 5 White-box testing techniques (Lab 3) Structure of Lecture
Why testing and analysis. Software Testing. A framework for software testing. Outline. Software Qualities. Dependability Properties
Why testing and analysis Software Testing Adapted from FSE 98 Tutorial by Michal Young and Mauro Pezze Software is never correct no matter what developing testing technique is used All software must be
Structural Testing. Testing Tactics. Why Structural? Structural white box. Functional black box. Structural white box. Functional black box
From Pressman, Software Engineering a practitioner s approach, Chapter 14 and Pezze + Young, Software Testing and Analysis, Chapters 12-13 Structural Testing Software Engineering Andreas Zeller Saarland
Security Testing. John Slankas
Security Testing John Slankas jbslanka@ncsu.edu Course Slides adapted from OWASP Testing Guide v4 CSC 515 Software Security What is Security Testing? Validate security controls operate as expected What
Introduction to Software Testing Chapter 2.4 Graph Coverage for Design Elements Paul Ammann & Jeff Offutt
Introduction to Software Testing Chapter 2.4 Graph Coverage for Design Elements Paul Ammann & Jeff Offutt www.introsoftwaretesting.com OO Software and Designs Emphasis on modularity and reuse puts complexity
Embedded Systems Design Prof. Anupam Basu Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur
Embedded Systems Design Prof. Anupam Basu Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture - 05 Optimization Issues Now I see, that is not been seen there;
Problem Solving through Programming In C Prof. Anupam Basu Department of Computer Science & Engineering Indian Institute of Technology, Kharagpur
Problem Solving through Programming In C Prof. Anupam Basu Department of Computer Science & Engineering Indian Institute of Technology, Kharagpur Lecture 17 Switch Statement (Refer Slide Time: 00:23) In
Testing Tactics. Structural Testing. Why Structural? Why Structural? Functional black box. Structural white box. Functional black box
ing Tactics Structural ing Functional black box Structural white box Software Engineering Andreas Zeller Saarland University s based on spec covers as much specified behavior as possible s based on code
White-Box Testing Techniques
T-76.5613 Software Testing and Quality Assurance Lecture 3, 18.9.2006 White-Box Testing Techniques SoberIT Content What are white-box testing techniques Control flow testing Statement coverage Branch coverage
MITOCW watch?v=0jljzrnhwoi
MITOCW watch?v=0jljzrnhwoi The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high quality educational resources for free. To
CS2 Algorithms and Data Structures Note 10. Depth-First Search and Topological Sorting
CS2 Algorithms and Data Structures Note 10 Depth-First Search and Topological Sorting In this lecture, we will analyse the running time of DFS and discuss a few applications. 10.1 A recursive implementation
Fleury s Algorithm The Adjacency Matrix and Distances Is There a Path From A to B? What is the Path from A to B? Is There a Path From ANY A to ANY B?
Intro Math Problem Solving 3 Graph TheoryNovember Origin Fleury s Algorithm The Adjacency Matrix and Distances Is There a Path From A to B? What is the Path from A to B? Is There a Path From ANY A to ANY
CENTRAL UNIVERSITY OF RAJASTHAN
CLASS : M.TECH SEMESTER: II BRANCH : CSE SESSION: 2011-12 SUBJECT CODE & NAME : CSIS-304 TOPICS IN OPERATING SYSTEM 1. The question paper contains five questions. 4.Before attempting the question paper,
MSc Software Testing MSc Prófun hugbúnaðar
MSc Software Testing MSc Prófun hugbúnaðar Fyrirlestrar 7 & 8 Structural Testing White-box tests. 29/8/27 Dr Andy Brooks 1 Case Study Dæmisaga Reference Structural Testing of Programs, A Survey, A A Omar
6.001 Notes: Section 8.1
6.001 Notes: Section 8.1 Slide 8.1.1 In this lecture we are going to introduce a new data type, specifically to deal with symbols. This may sound a bit odd, but if you step back, you may realize that everything
MTAT : Software Testing
MTAT.03.159: Software Testing Lecture 03: White-Box Testing (Textbook Ch. 5) Spring 2013 Dietmar Pfahl email: dietmar.pfahl@ut.ee Lecture Chapter 5 White-box testing techniques (Lab 3) Structure of Lecture
Structural Testing. Testing Tactics. Why Structural? Structural white box. Functional black box. Structural white box. Functional black box
From Pressman, Software Engineering a practitioner s approach, Chapter 14 and Pezze + Young, Software Testing and Analysis, Chapters 12-13 Structural Testing Software Engineering Andreas Zeller Saarland
Programming language components
Programming language components syntax: grammar rules for defining legal statements what's grammatically legal? how are things built up from smaller things? semantics: what things mean what do they compute?
More Dataflow Analysis
More Dataflow Analysis Steps to building analysis Step 1: Choose lattice Step 2: Choose direction of dataflow (forward or backward) Step 3: Create transfer function Step 4: Choose confluence operator (i.e.,
Control-Flow Analysis
Control-Flow Analysis Dragon book [Ch. 8, Section 8.4; Ch. 9, Section 9.6] Compilers: Principles, Techniques, and Tools, 2 nd ed. by Alfred V. Aho, Monica S. Lam, Ravi Sethi, and Jerey D. Ullman on reserve
The following content is provided under a Creative Commons license. Your support
MITOCW Recitation 1 The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue to offer high-quality educational resources for free. To make
Control Flow Analysis
Control Flow Analysis Last time Undergraduate compilers in a day Today Assignment 0 due Control-flow analysis Building basic blocks Building control-flow graphs Loops January 28, 2015 Control Flow Analysis
Logic Coverage. Moonzoo Kim School of Computing KAIST. The original slides are taken from Chap. 8 of Intro. to SW Testing 2 nd ed by Ammann and Offutt
Logic Coverage Moonzoo Kim School of Computing KAIST The original slides are taken from Chap. 8 of Intro. to SW Testing 2 nd ed by Ammann and Offutt Covering Logic Expressions Logic expressions show up
Combinatorial Clause Coverage CoC
Introduction to Software Testing Chapter 3.2 Logic Coverage Paul Ammann & Jeff Offutt Covering Logic Expressions Logic expressions show up in many situations Covering logic expressions is required by the
Types, Expressions, and States
8/27: solved Types, Expressions, and States CS 536: Science of Programming, Fall 2018 A. Why? Expressions represent values in programming languages, relative to a state. Types describe common properties
perl -MO=Deparse[,-d][,-fFILE][,-p][,-q][,-l] [,-sletters][,-xlevel] prog.pl
![perl -MO=Deparse[,-d][,-fFILE][,-p][,-q][,-l] [,-sletters][,-xlevel] prog.pl](/thumbs/86/93809824.jpg "perl -MO=Deparse[,-d][,-fFILE][,-p][,-q][,-l] [,-sletters][,-xlevel] prog.pl")
NAME SYNOPSIS DESCRIPTION OPTIONS B::Deparse - Perl compiler backend to produce perl code perl -MO=Deparse[,-d][,-fFILE][,-p][,-q][,-l] [,-sletters][,-xlevel] prog.pl B::Deparse is a backend module for