Testing & Symbolic Execution
|
|
- Austen Martin
- 5 years ago
- Views:
Transcription
1 Testing & Symbolic Execution
2 Software Testing The most common way of measuring & ensuring correctness Input 2
3 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior 3
4 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle 4
5 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome 5
6 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome Test Suite Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 6
7 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome Key Issues: Are the tests adequate? Test Suite Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 7
8 Software Testing The most common way of measuring & ensuring correctness Input? Observed Behavior Oracle Outcome Key Issues: Are the tests adequate? Automated input generation Test Suite Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 8
9 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome Key Issues: Are the tests adequate? Automated input generation Automated oracles Test Suite Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 9
10 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome Test 1 Are the tests adequate? Test 2 Test 3 Automated input generation Test 4 Automated oracles Test 5 Test 6 Robustness / flakiness / maintainability Test 7 Key Issues: Test Suite 10
11 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome Test 1 Are the tests adequate? Test 2 Test 3 Automated input generation Test 4 Automated oracles Test 5 Test 6 Robustness / flakiness / maintainability Test 7... Key Issues: Test Suite 11
12 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? 12
13 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage def my_lovely_fun(a,b,c): if (a && b) c: Is each statement executed by at least one test in the test suite? else: print('awesome') score= # covered # statements 13
14 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage score= # covered # branches def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') b T a c F 14
15 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage More common in safety critical systems where full coverage may be required. def my_lovely_fun(a,b,c): if (a & b) c: else: print('awesome') 15
16 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage def my_lovely_fun(a,b,c): if (a & b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') 16
17 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage # covered/killed score= # non-equivalent mutants def my_lovely_fun(a,b,c): if (a & b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') 17
18 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage Path Coverage B E ENTRY A D C F EXIT 18
19 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage Path Coverage Can apply EPP! B E ENTRY A D C F EXIT 19
20 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage Path Coverage... 20
21 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage Path Coverage BUT reducing test suites for St, Br, MC/DC coverage decrease defect detection! 21
22 Generating Inputs Sample all possible inputs for test in allpossibleinputs: check_test(test) 22
23 Generating Inputs Sample all possible inputs for test in allpossibleinputs: check_test(test) Target specific goals or other coverage criteria for s in statements: test = findtestthrough(s) check_test(test) 23
24 Generating Inputs Sample all possible inputs for test in allpossibleinputs: check_test(test) Target specific goals or other coverage criteria for s in statements: test = findtestthrough(s) check_test(test) for i in inputmodel: test = findrepresentative(i) check_test(test) 24
25 Generating Inputs Usually broken into black box & white box approaches 25
26 Generating Inputs Usually broken into black box & white box approaches Black Box Treat the program as opaque / unknown e.g. specification based, naive fuzzing, boundary value analysis,... 26
27 Generating Inputs Usually broken into black box & white box approaches Black Box Treat the program as opaque / unknown White Box Program structure & semantics can be used e.g. symbolic execution, call chain synthesis, white box fuzzing, boundary value analysis,... 27
28 Generating Oracles? 28
29 Generating Oracles Likely invariants? Careful variable selection & monitoring? Differential Testing Metamorphic Testing A very open (hard) problem. 29
30 Interesting Problems Random Testing Test Suite Adequacy Test Suite Minimization Test Generation Oracle Generation Test Maintenance Performance Testing Field Testing Power Testing Test Prioritization... 30
31 Interesting Problems Random Testing Test Suite Adequacy Test Suite Minimization Test Generation Oracle Generation Test Maintenance Performance Testing Field Testing Power Testing Test Prioritization... Test generation techniques have also proven to be critical in security research. 31
32 Symbolic Execution An approach for generating test inputs. Cadar & Sen, 2013 x input() y input() if x == 2*y if x > y+10 32
33 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 33
34 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 34
35 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 x = 2*y y > 10 Path Constraint 35
36 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 A path constraint represents all executions along that path x = 2*y y > 10 Path Constraint 36
37 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Solve for the symbolic symbols to find inputs that yield the path. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 x=30 y=15 37
38 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Solve for the symbolic symbols to find inputs that yield the path. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 x=30 y=15 x=2 y=1 38
39 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Solve for the symbolic symbols to find inputs that yield the path. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 x=30 y=15 x=2 y=1 x=0 y=1 39
40 How Can We Solve Constraints? SMT Solvers Satisfiability Modulo Theories SAT with extra logic Standard interfaces through SMTLIB2 40
41 How Can We Solve Constraints? SMT Solvers Satisfiability Modulo Theories SAT with extra logic Standard interfaces through SMTLIB2 x = 2*y y > 10 (declare-const x Int) (declare-const y Int) (assert (= x (* 2 y))) (assert (> y 10)) (check-sat) (get-model) 41
42 How Can We Solve Constraints? SMT Solvers Satisfiability Modulo Theories SAT with extra logic Standard interfaces through SMTLIB2 x = 2*y y > 10 (declare-const x Int) (declare-const y Int) (assert (= x (* 2 y))) (assert (> y 10)) (check-sat) (get-model) Z3 42
43 How Can We Solve Constraints? SMT Solvers Satisfiability Modulo Theories SAT with extra logic Standard interfaces through SMTLIB2 x = 2*y y > 10 (declare-const x Int) (declare-const y Int) (assert (= x (* 2 y))) (assert (> y 10)) (check-sat) (get-model) x=22 y=11 Z3 sat (model (define-fun y () Int 11) (define-fun x () Int 22) ) 43
44 How Can We Solve Constraints? SMT Solvers Satisfiability Modulo Theories SAT with extra logic Standard interfaces through SMTLIB2 x = 2*y y > 10 (declare-const x Int) (declare-const y Int) (assert (= x (* 2 y))) (assert (> y 10)) (check-sat) (get-model) x=22 y=11 Z3 sat (model (define-fun y () Int 11) (define-fun x () Int 22) ) Try it online: 44
45 Useful Questions If ɸ holds after a statement, what must have been true at the point before? weakest precondition 45
46 Useful Questions If ɸ holds after a statement, what must have been true at the point before? weakest precondition If ɸ holds before a statement, what can we guarantee to be true after? strongest poscondition 46
47 Useful Questions If ɸ holds after a statement, what must have been true at the point before? weakest precondition If ɸ holds before a statement, what can we guarantee to be true after? strongest poscondition e.g. Given two versions of a program v1,v2 and assertions on output ɸi in each from an input I What is wp(ɸ1) wp(ɸ2)? 47
48 Useful Questions If ɸ holds after a statement, what must have been true at the before? weakest precondition If ɸ holds before a statement, what can we guarantee to be true after? strongest poscondition e.g. Given two versions of a program v1,v2 and assertions on output ɸi in each from an input I What is wp(ɸ1) wp(ɸ2)? wp(ɸ1) wp(ɸ2) 48
49 Useful Questions If ɸ holds after a statement, what must have been true at the before? weakest precondition If ɸ holds before a statement, what can we guarantee to be true after? strongest poscondition e.g. Given two versions of a program v1,v2 and assertions on output ɸi in each from an input I What is wp(ɸ1) wp(ɸ2)? wp(ɸ1) wp(ɸ2) What is the intuitive meaning? 49
50 Exploring the Execution Tree The possible paths of a program form an execution tree. x input() y input() Cadar & Sen, 2013 if x == 2*y if x > y+10 50
51 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. x input() y input() if x == 2*y Cadar & Sen, 2013 if x > y+10 51
52 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches x input() y input() if x == 2*y if x > y+10 Cadar & Sen,
53 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) x input() y input() if x == 2*y if x > y+10 Cadar & Sen,
54 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) x input() y input() if x == 2*y if x > y+10 Cadar & Sen,
55 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) x input() y input() if x == 2*y if x > y+10 Cadar & Sen, 2013 (x=2*y) (x>y+10) 55
56 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) x input() y input() if x == 2*y if x > y+10 Cadar & Sen, 2013 (x=2*y) (x>y+10) 56
57 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) x input() y input() if x == 2*y if x > y+10 Cadar & Sen,
58 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) Execution Generated Testing x input() y input() if x == 2*y if x > y+10 Cadar & Sen,
59 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) Execution Generated Testing x input() y input() if x == 2*y if x > y+10 Cadar & Sen, 2013 X=? y=10 59
60 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches X=20 Concolic (dynamic symbolic) y=10 Execution Generated Testing x input() y input() if x == 2*y if x > y+10 Cadar & Sen, 2013 X=? 20 y=10 60
61 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches X=20 Concolic (dynamic symbolic) y=10 Execution Generated Testing Execution on this side is concrete from this point on. x input() y input() if x == 2*y if x > y+10 Cadar & Sen, 2013 X=? 20 y=10 61
62 Symbolic Execution Increasingly scalable every year 62
63 Symbolic Execution Increasingly scalable every year Can automatically generate test inputs from constraints 63
64 Symbolic Execution Increasingly scalable every year Can automatically generate test inputs from constraints The resulting symbolic formulae have many uses beyond just testing. 64
65 Symbolic Execution Increasingly scalable every year Can automatically generate test inputs from constraints The resulting symbolic formulae have many uses beyond just testing. Try it out: 1) 2) Symbolic PathFinder 3) 4) 65
Testing, Fuzzing, & Symbolic Execution
Testing, Fuzzing, & Symbolic Execution Software Testing The most common way of measuring & ensuring correctness Input 2 Software Testing The most common way of measuring & ensuring correctness Input Observed
More informationCS 520 Theory and Practice of Software Engineering Fall 2018
Today CS 52 Theory and Practice of Software Engineering Fall 218 Software testing October 11, 218 Introduction to software testing Blackbox vs. whitebox testing Unit testing (vs. integration vs. system
More informationSymbolic Execution. Wei Le April
Symbolic Execution Wei Le 2016 April Agenda What is symbolic execution? Applications History Interal Design: The three challenges Path explosion Modeling statements and environments Constraint solving
More informationPart I: Preliminaries 24
Contents Preface......................................... 15 Acknowledgements................................... 22 Part I: Preliminaries 24 1. Basics of Software Testing 25 1.1. Humans, errors, and testing.............................
More informationSymbolic and Concolic Execution of Programs
Symbolic and Concolic Execution of Programs Information Security, CS 526 Omar Chowdhury 10/7/2015 Information Security, CS 526 1 Reading for this lecture Symbolic execution and program testing - James
More informationA short manual for the tool Accumulator
A short manual for the tool Accumulator ZHAO Jianhua State Key Laboratory of Novel Software Technology Dept. of Computer Sci. and Tech. Nanjing University Nanjing, Jiangsu, P.R.China 210093 zhaojh@nju.edu.cn
More informationSoftware Testing. Testing: Our Experiences
Software Testing Testing: Our Experiences Test Case Software to be tested Output 1 Test Case Generation When to Stop? Test Case Software to be tested Verification Output No Enough? Test Coverage Yes A
More information6.0 ECTS/4.5h VU Programm- und Systemverifikation ( ) June 22, 2016
6.0 ECTS/4.5h VU Programm- und Systemverifikation (184.741) June 22, 2016 Kennzahl (study id) Matrikelnummer (student id) Familienname (family name) Vorname (first name) Gruppe (version) A 1.) Coverage
More informationTesting! Prof. Leon Osterweil! CS 520/620! Spring 2013!
Testing Prof. Leon Osterweil CS 520/620 Spring 2013 Relations and Analysis A software product consists of A collection of (types of) artifacts Related to each other by myriad Relations The relations are
More information6. Test-Adequacy. Assessment Using Control Flow and Data Flow. Andrea Polini
6. Test-Adequacy Assessment Using Control Flow and Data Flow Andrea Polini Software Engineering II Software Testing MSc in Computer Science University of Camerino (Software Engineering II Software Testing)
More informationFacts About Testing. Cost/benefit. Reveal faults. Bottom-up. Testing takes more than 50% of the total cost of software development
Reveal faults Goals of testing Correctness Reliability Usability Robustness Performance Top-down/Bottom-up Bottom-up Lowest level modules tested first Don t depend on any other modules Driver Auxiliary
More informationn HW7 due in about ten days n HW8 will be optional n No CLASS or office hours on Tuesday n I will catch up on grading next week!
Announcements SMT Solvers, Symbolic Execution n HW7 due in about ten days n HW8 will be optional n No CLASS or office hours on Tuesday n I will catch up on grading next week! n Presentations n Some of
More informationSubject Software Testing Structural Testing
Subject Software Testing Structural Testing Objective: 1. Understand Concept of structural testing 2. How structural (code-based or glass-box) testing complements functional (black-box) testing 3. Recognize
More informationMTAT : Software Testing
MTAT.03.159: Software Testing Lecture 03: White-Box Testing (Textbook Ch. 5) Spring 2013 Dietmar Pfahl email: dietmar.pfahl@ut.ee Lecture Chapter 5 White-box testing techniques (Lab 3) Structure of Lecture
More informationTesting Methods: White Box Testing II
Testing Methods: White Box Testing II Outline Today we continue our look at white box testing with more code coverage methods, and a data coverage method We ll look at : - code coverage testing - decision
More informationMTAT : Software Testing
MTAT.03.159: Software Testing Lecture 03: White-Box Testing (Textbook Ch. 5) Dietmar Pfahl Spring 2016 email: dietmar.pfahl@ut.ee Lecture Chapter 5 White-box testing techniques (Lab 3) Structure of Lecture
More informationStatic Analysis and Bugfinding
Static Analysis and Bugfinding Alex Kantchelian 09/12/2011 Last week we talked about runtime checking methods: tools for detecting vulnerabilities being exploited in deployment. So far, these tools have
More informationWhat is Mutation Testing? Mutation Testing. Test Case Adequacy. Mutation Testing. Mutant Programs. Example Mutation
What is Mutation Testing? Mutation Testing Breaking the application to test it n Mutation Testing is a testing technique that focuses on measuring the adequacy of test cases n Mutation Testing is NOT a
More informationMTAT : Software Testing
MTAT.03.159: Software Testing Lecture 03: White-Box Testing (Textbook Ch. 5) Dietmar Pfahl Spring 2017 email: dietmar.pfahl@ut.ee Lecture Chapter 5 White-box testing techniques (Lab 3) Structure of Lecture
More informationTesting Role in Unified Approach Coverage: Structural/Coverage Model Based Test Generation from Model Checking (project) Interaction of
Testing Role in Unified Approach Coverage: Structural/Coverage Model Based Test Generation from Model Checking (project) Interaction of Coverage/Model Based Testing Will Not Cover Statistical Methods Partition
More informationWhite Box Testing III
White Box Testing III Outline Today we continue our look at white box testing methods, with mutation testing We will look at : definition and role of mutation testing what is a mutation? how is mutation
More informationIntroduction to Dynamic Analysis
Introduction to Dynamic Analysis Reading assignment Gary T. Leavens, Yoonsik Cheon, "Design by Contract with JML," draft paper, http://www.eecs.ucf.edu/~leavens/jml//jmldbc.pdf G. Kudrjavets, N. Nagappan,
More informationThree Important Testing Questions
Testing Part 2 1 Three Important Testing Questions How shall we generate/select test cases? Did this test execution succeed or fail? How do we know when we ve tested enough? 65 1. How do we know when we
More informationSecond assignment came out Monday evening. Find defects in Hnefetafl rules written by your classmates. Topic: Code Inspection and Testing
Announcements Second assignment came out Monday evening Topic: Code Inspection and Testing Find defects in Hnefetafl rules written by your classmates Compare inspection, coverage testing, random testing,
More informationTest Automation. 20 December 2017
Test Automation 20 December 2017 The problem of test automation Testing has repetitive components, so automation is justified The problem is cost-benefit evaluation of automation [Kaner] Time for: test
More informationTesting Tactics. Structural Testing. Why Structural? Why Structural? Functional black box. Structural white box. Functional black box
ing Tactics Structural ing Functional black box Structural white box Software Engineering Andreas Zeller Saarland University s based on spec covers as much specified behavior as possible s based on code
More informationAdvanced Test Coverage Criteria: Specify and Measure, Cover and Unmask
Advanced Test Coverage Criteria: Specify and Measure, Cover and Unmask Sébastien Bardin & Nikolai Kosmatov joint work with Omar Chebaro, Robin David, Mickaël Delahaye, Michaël Marcozzi, Mike Papadakis,
More informationSymbolic Execu.on. Suman Jana
Symbolic Execu.on Suman Jana Acknowledgement: Baishakhi Ray (Uva), Omar Chowdhury (Purdue), Saswat Anand (GA Tech), Rupak Majumdar (UCLA), Koushik Sen (UCB) What is the goal? Tes.ng Tes%ng approaches are
More informationCYSE 411/AIT681 Secure Software Engineering Topic #17: Symbolic Execution
CYSE 411/AIT681 Secure Software Engineering Topic #17: Symbolic Execution Instructor: Dr. Kun Sun Software has bugs To find them, we use testing and code reviews But some bugs are still missed Rare features
More informationSoftware has bugs. Static analysis 4/9/18. CYSE 411/AIT681 Secure Software Engineering. To find them, we use testing and code reviews
CYSE 411/AIT681 Secure Software Engineering Topic #17: Symbolic Execution Instructor: Dr. Kun Sun Software has bugs To find them, we use testing and code reviews But some bugs are still missed Rare features
More informationCMSC 430 Introduction to Compilers. Fall Symbolic Execution
CMSC 430 Introduction to Compilers Fall 2015 Symbolic Execution Introduction Static analysis is great Lots of interesting ideas and tools Commercial companies sell, use static analysis It all looks good
More informationAbstract Interpretation
Abstract Interpretation Ranjit Jhala, UC San Diego April 22, 2013 Fundamental Challenge of Program Analysis How to infer (loop) invariants? Fundamental Challenge of Program Analysis Key issue for any analysis
More informationCounterexample-Driven Genetic Programming
Counterexample-Driven Genetic Programming Iwo Błądek, Krzysztof Krawiec Institute of Computing Science, Poznań University of Technology Poznań, 12.12.2017 I. Błądek, K. Krawiec Counterexample-Driven Genetic
More informationWritten exam TDDD04 Software Testing
LiTH, Linköpings tekniska högskola IDA, Institutionen för datavetenskap Ola Leifler Written exam TDDD04 Software Testing 2016-10-26 Permissible aids Dictionary (printed, NOT electronic) Teacher on duty
More informationTesting: Test design and testing process
Testing: Test design and testing process Zoltán Micskei Based on István Majzik s slides Dept. of Measurement and Information Systems Budapest University of Technology and Economics Department of Measurement
More informationTest Design Techniques ISTQB (International Software Testing Qualifications Board)
Test Design Techniques ISTQB (International Software Testing Qualifications Board) Minsoo Ryu Hanyang University Testing Process Planning and Control Analysis and Design Implementation and Execution Evaluating
More informationGNATprove a Spark2014 verifying compiler Florian Schanda, Altran UK
1 GNATprove a Spark2014 verifying compiler Florian Schanda, Altran UK Tool architecture User view Source gnatprove Verdict 2 Tool architecture More detailed view... Source Encoding CVC4 gnat2why gnatwhy3
More informationSTRUCTURAL TESTING. AKA White Box Testing. Thanks go to Andreas Zeller for allowing incorporation of his materials. F. Tip and M.
F. Tip and M. Weintraub STRUCTURAL TESTING AKA White Box Testing Thanks go to Andreas Zeller for allowing incorporation of his materials STRUCTURAL TESTING Testing based on the structure of the code Test
More informationProperties of Criteria. 1. Applicability Property. Criteria
Properties of Criteria Program-based To recognize a good adequacy criteria And to discard poor choices Objective, well-defined properties 1. Applicability Property For every program, there exists an adequate
More informationStructural Testing. (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 1
Structural Testing (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 1 Learning objectives Understand rationale for structural testing How structural (code-based or glass-box) testing complements functional
More informationUsing the code to measure test adequacy (and derive test cases) Structural Testing
Using the code to measure test adequacy (and derive test cases) Structural Testing Objectives To describe a second approach to testing which is geared to find program defects To explain the use of program
More informationSatisfiability Modulo Theories: ABsolver
Satisfiability Modulo Theories: ABsolver Michael Tautschnig Joint work with: Andreas Bauer Martin Leucker Christian Schallhart Michael Tautschnig 1 Outline 1. Introduction Michael Tautschnig 2 Outline
More informationEECS 481 Software Engineering Exam #1. You have 1 hour and 20 minutes to work on the exam.
EECS 481 Software Engineering Exam #1 Write your name and UM uniqname on the exam. There are ten (10) pages in this exam (including this one) and seven (7) questions, each with multiple parts. Some questions
More informationSTRUCTURAL TESTING. AKA White Box Testing. Thanks go to Andreas Zeller for allowing incorporation of his materials. F. Tip and M.
F. Tip and M. Weintraub STRUCTURAL TESTING AKA White Box Testing Thanks go to Andreas Zeller for allowing incorporation of his materials STRUCTURAL TESTING Testing based on the structure of the code Test
More informationAutomated White-Box Testing Beyond Branch Coverage
Bardin et al. Dagstuhl seminar 14442 1/ 22 Automated White-Box Testing Beyond Branch Coverage Sébastien Bardin, Nikolai Kosmatov, Mickaël Delahaye CEA LIST, Software Safety Lab (Paris-Saclay, France) The
More informationSoftware Testing for Developer Development Testing. Duvan Luong, Ph.D. Operational Excellence Networks
Software Testing for Developer Development Testing Duvan Luong, Ph.D. Operational Excellence Networks Contents R&D Testing Approaches Static Analysis White Box Testing Black Box Testing 4/2/2012 2 Development
More informationClass 17. Discussion. Mutation analysis and testing. Problem Set 7 discuss Readings
Class 17 Questions/comments Graders for Problem Set 6 (4); Graders for Problem set 7 (2-3) (solutions for all); will be posted on T-square Regression testing, Instrumentation Final project presentations:
More informationSoftware Quality Assurance. David Janzen
Software Quality Assurance David Janzen What is quality? Crosby: Conformance to requirements Issues: who establishes requirements? implicit requirements Juran: Fitness for intended use Issues: Who defines
More informationCMPT 473 Software Quality Assurance. Graph Coverage. Nick Sumner
CMPT 473 Software Quality Assurance Graph Coverage Nick Sumner Recall: Coverage/Adequacy Can't look at all possible inputs. Need to determine if a test suite covers / is adequate for our quality objectives.
More informationSymbolic Execution, Dynamic Analysis
Symbolic Execution, Dynamic Analysis http://d3s.mff.cuni.cz Pavel Parízek CHARLES UNIVERSITY IN PRAGUE faculty of mathematics and physics Symbolic execution Pavel Parízek Symbolic Execution, Dynamic Analysis
More informationCUTE: A Concolic Unit Testing Engine for C
CUTE: A Concolic Unit Testing Engine for C Koushik Sen Darko Marinov Gul Agha University of Illinois Urbana-Champaign Goal Automated Scalable Unit Testing of real-world C Programs Generate test inputs
More informationGenerating Small Countermodels. Andrew Reynolds Intel August 30, 2012
Generating Small Countermodels using SMT Andrew Reynolds Intel August 30, 2012 Acknowledgements Intel Corporation AmitGoel, Sava Krstic University of Iowa Cesare Tinelli, Francois Bobot New York University
More informationAn Introduction to Satisfiability Modulo Theories
An Introduction to Satisfiability Modulo Theories Philipp Rümmer Uppsala University Philipp.Ruemmer@it.uu.se February 13, 2019 1/28 Outline From theory... From DPLL to DPLL(T) Slides courtesy of Alberto
More informationTesting. ECE/CS 5780/6780: Embedded System Design. Why is testing so hard? Why do testing?
Testing ECE/CS 5780/6780: Embedded System Design Scott R. Little Lecture 24: Introduction to Software Testing and Verification What is software testing? Running a program in order to find bugs (faults,
More informationStructural Testing. Testing Tactics. Why Structural? Structural white box. Functional black box. Structural white box. Functional black box
From Pressman, Software Engineering a practitioner s approach, Chapter 14 and Pezze + Young, Software Testing and Analysis, Chapters 12-13 Structural Testing Software Engineering Andreas Zeller Saarland
More informationCMSC 631 Program Analysis and Understanding. Spring Symbolic Execution
CMSC 631 Program Analysis and Understanding Spring 2013 Symbolic Execution Introduction Static analysis is great Lots of interesting ideas and tools Commercial companies sell, use static analysis It all
More informationSmall Formulas for Large Programs: On-line Constraint Simplification In Scalable Static Analysis
Small Formulas for Large Programs: On-line Constraint Simplification In Scalable Static Analysis Isil Dillig, Thomas Dillig, Alex Aiken Stanford University Scalability and Formula Size Many program analysis
More informationSoftware Testing TEST CASE SELECTION AND ADEQUECY TEST EXECUTION
Software Testing TEST CASE SELECTION AND ADEQUECY TEST EXECUTION Overview, Test specification and cases, Adequacy criteria, comparing criteria, Overview of test execution, From test case specification
More informationAdvanced Software Testing Understanding Code Coverage
Advanced Software Testing Understanding Code Coverage Advanced Software Testing A series of webinars, this one excerpted from Advanced Software Testing: V3, a book for technical test analysts, programmers,
More informationCSE507. Practical Applications of SAT. Computer-Aided Reasoning for Software. Emina Torlak
Computer-Aided Reasoning for Software CSE507 Practical Applications of SAT courses.cs.washington.edu/courses/cse507/18sp/ Emina Torlak emina@cs.washington.edu Today Past 2 lectures The theory and mechanics
More informationLecture 18: Structure-based Testing
Test Case First Strategy White box testing: Statement Coverage Branch Coverage Condition Coverage Data Path Coverage Lecture 18: Structure-based Testing Testing with good and bad data Testing Object Oriented
More informationSymbolic Execution for Bug Detection and Automated Exploit Generation
Symbolic Execution for Bug Detection and Automated Exploit Generation Daniele Cono D Elia Credits: Emilio Coppa SEASON Lab season-lab.github.io May 27, 2016 1 / 29 Daniele Cono D Elia Symbolic Execution
More informationInformation page for written examinations at Linköping University
Information page for written examinations at Linköping University Examination date 2017-08-23 Room (1) Time 8-12 Course code Exam code Course name Exam name Department Number of questions in the examination
More informationProgramming Embedded Systems
Programming Embedded Systems Lecture 8 Overview of software testing Wednesday Feb 8, 2012 Philipp Rümmer Uppsala University Philipp.Ruemmer@it.uu.se 1/53 Lecture outline Testing in general Unit testing
More informationTesting, Debugging, Program Verification
Testing, Debugging, Program Verification Automated Test Case Generation, Part II Wolfgang Ahrendt & Vladimir Klebanov & Moa Johansson 12 December 2012 TDV: ATCG II /GU 2011-12-12 1 / 17 Recap Specification-/Model-Based
More informationA survey of new trends in symbolic execution for software testing and analysis
Int J Softw Tools Technol Transfer (2009) 11:339 353 DOI 10.1007/s10009-009-0118-1 REGULAR PAPER A survey of new trends in symbolic execution for software testing and analysis Corina S. Păsăreanu Willem
More informationAutomatic Software Verification
Automatic Software Verification Instructor: Mooly Sagiv TA: Oded Padon Slides from Eran Yahav and the Noun Project, Wikipedia Course Requirements Summarize one lecture 10% one lecture notes 45% homework
More informationOpenMath and SMT-LIB
James, Matthew England, Roberto Sebastiani & Patrick Trentin 1 Universities of Bath/Coventry/Trento/Trento J.H.@bath.ac.uk 17 July 2017 1 Thanks to EU H2020-FETOPEN-2016-2017-CSA project SC 2 (712689)
More informationAnalysis/Bug-finding/Verification for Security
Analysis/Bug-finding/Verification for Security VIJAY GANESH University of Waterloo Winter 2013 Analysis/Test/Verify for Security Instrument code for testing Heap memory: Purify Perl tainting (information
More informationVerifying C & C++ with ESBMC
Verifying C & C++ with ESBMC Denis A Nicole dan@ecs.soton.ac.uk CyberSecuritySoton.org [w] @CybSecSoton [fb & tw] ESBMC ESBMC, the Efficient SMT-Based Context-Bounded Model Checker was originally developed
More information18-642: Unit Testing 1/31/ Philip Koopman
18-642: Unit Testing 1/31/2018 2017-2018 Philip Koopman YOU ARE HERE Product Requirements SPECIFY PRODUCT SPECIFY SOFTWARE Software Requirements TRACEABILITY & VALIDATION Test Plan & Test Results Test
More informationAutomated Requirements-Based Testing
Automated Requirements-Based Testing Tuesday, October 7 th 2008 2008 The MathWorks, Inc. Dr. Marc Segelken Senior Application Engineer Overview Purposes of Testing Test Case Generation Structural Testing
More informationFault-based testing. Automated testing and verification. J.P. Galeotti - Alessandra Gorla. slides by Gordon Fraser. Thursday, January 17, 13
Fault-based testing Automated testing and verification J.P. Galeotti - Alessandra Gorla slides by Gordon Fraser How good are my tests? How good are my tests? Path testing Boundary interior testing LCSAJ
More informationMutation Testing. Leaving the Stone Age
Mutation Testing Leaving the Stone Age 2017 whoami ios Developer by day compiler hacker by night https://twitter.com/1101_debian https://lowlevelbits.org https://systemundertest.org Outline Quality of
More informationTest Oracles and Mutation Testing. CSCE Lecture 23-11/18/2015
Test Oracles and Mutation Testing CSCE 740 - Lecture 23-11/18/2015 Software Testing - Back to the Basics Tests are sequences of stimuli and observations. We care about input and output. (I 1 O 1 ) (I 2
More informationWhy testing and analysis. Software Testing. A framework for software testing. Outline. Software Qualities. Dependability Properties
Why testing and analysis Software Testing Adapted from FSE 98 Tutorial by Michal Young and Mauro Pezze Software is never correct no matter what developing testing technique is used All software must be
More informationSpecification Centered Testing
Specification Centered Testing Mats P. E. Heimdahl University of Minnesota 4-192 EE/CS Building Minneapolis, Minnesota 55455 heimdahl@cs.umn.edu Sanjai Rayadurgam University of Minnesota 4-192 EE/CS Building
More informationManuel Oriol, CHCRC-C, Software Testing ABB
Manuel Oriol, CHCRC-C, 08.11.2017 Software Testing Slide 1 About me 1998 2004 2005 2008 2011 Slide 2 Introduction Why do we test? Did you have to deal with testing in the past? Slide 3 Ariane 5 http://www.youtube.com/watch?v=kyurqduyepi
More informationMONIKA HEINER.
LESSON 1 testing, intro 1 / 25 SOFTWARE TESTING - STATE OF THE ART, METHODS, AND LIMITATIONS MONIKA HEINER monika.heiner@b-tu.de http://www.informatik.tu-cottbus.de PRELIMINARIES testing, intro 2 / 25
More informationIn this Lecture you will Learn: Testing in Software Development Process. What is Software Testing. Static Testing vs.
In this Lecture you will Learn: Testing in Software Development Process Examine the verification and validation activities in software development process stage by stage Introduce some basic concepts of
More informationStructural Testing. White Box Testing & Control Flow Analysis
Structural Testing White Box Testing & Control Flow Analysis Functional vs. Structural Functional Have I built the right product? Tests derived from the program specification Internal Structure ignored
More informationReducing Verification Costs through Practical Formal Methods: A Survey
Reducing Verification Costs through Practical Formal Methods: A Survey Mark R. Blackburn, Ph.D. Stevens Institute of Technology Sumit Ray BAE Systems Abstract Verification of software can be as much as
More informationSymbolic Execution. Michael Hicks. for finding bugs. CMSC 631, Fall 2017
Symbolic Execution for finding bugs Michael Hicks CMSC 631, Fall 2017 Software has bugs To find them, we use testing and code reviews But some bugs are still missed Rare features Rare circumstances Nondeterminism
More informationTesting Objectives. Successful testing: discovers previously unknown errors
Testing Objectives Informal view: Testing: a process of executing software with the intent of finding errors Good testing: a high probability of finding as-yetundiscovered errors Successful testing: discovers
More informationMTAT : Software Testing
MTAT.03.159: Software Testing Lecture 02: Basic Black-Box and White-Box Testing Techniques (Textbook Ch. 4 & 5) Spring 2018 Dietmar Pfahl email: dietmar.pfahl@ut.ee Structure of Lecture 2 Black-Box vs.
More informationInformation page for written examinations at Linköping University
Information page for written examinations at Linköping University Examination date 2017-01-14 Room (1) Time 8-12 Course code Exam code Course name Exam name Department Number of questions in the examination
More informationProgram Analysis and Constraint Programming
Program Analysis and Constraint Programming Joxan Jaffar National University of Singapore CPAIOR MasterClass, 18-19 May 2015 1 / 41 Program Testing, Verification, Analysis (TVA)... VS... Satifiability/Optimization
More informationIngegneria del Software Corso di Laurea in Informatica per il Management
Ingegneria del Software Corso di Laurea in Informatica per il Management Software testing Davide Rossi Dipartimento di Informatica Università di Bologna Validation and verification Software testing is
More informationMTAT : Software Testing
MTAT.03.159: Software Testing Lecture 04: White-Box Testing (advanced) Part1 Dietmar Pfahl Spring 2018 email: dietmar.pfahl@ut.ee White-Box Testing Techniques Control-Flow Testing Data-Flow Testing Mutation
More informationCMPSCI 521/621 Homework 2 Solutions
CMPSCI 521/621 Homework 2 Solutions Problem 1 Direct data dependencies: 3 is directly data dependent on 1 and 5 5 is directly data dependent on 1,3, and 5 7 is directly data dependent on 1,3, and 5 Note,
More informationIntroduction to Software Engineering
Introduction to Software Engineering (CS350) Lecture 17 Jongmoon Baik Testing Conventional Applications 2 Testability Operability it operates cleanly Observability the results of each test case are readily
More informationMLSA: a static bugs analysis tool based on LLVM IR
International Journal of Networked and Distributed Computing, Vol. 4, No. 3 (July 2016), 137-144 MLSA: a static bugs analysis tool based on LLVM IR Hongliang Liang 1, Lei Wang 1, Dongyang Wu 1, Jiuyun
More informationProgram Testing and Analysis: Manual Testing Prof. Dr. Michael Pradel Software Lab, TU Darmstadt
Program Testing and Analysis: Manual Testing Prof. Dr. Michael Pradel Software Lab, TU Darmstadt Partly based on slides from Peter Müller, ETH Zurich 1 Warm-up Quiz What does the following code print?
More informationGraphicsFuzz. Secure and Robust Graphics Rendering. Hugues Evrard Paul Thomson
GraphicsFuzz Secure and Robust Graphics Rendering Hugues Evrard h.evrard@imperial.ac.uk, Paul Thomson paul.thomson11@imperial.ac.uk Problem: defective drivers We are here to help! How: automatic generation
More informationTesting: Coverage and Structural Coverage
Testing: Coverage and Structural Coverage Testing, Quality Assurance, and Maintenance Winter 2017 Prof. Arie Gurfinkel based on slides by Prof. Marsha Chechik and Prof. Lin Tan How would you test this
More informationTermComp Proposal: Pushdown Systems as a Model for Programs with Procedures
TermComp Proposal: Pushdown Systems as a Model for Programs with Procedures Marc Brockschmidt Andrey Rybalchenko Microsoft Research June 5, 2014 Abstract A program with procedures can be formally modelled
More informationIntroduction to Problem Solving and Programming in Python.
Introduction to Problem Solving and Programming in Python http://cis-linux1.temple.edu/~tuf80213/courses/temple/cis1051/ Overview Types of errors Testing methods Debugging in Python 2 Errors An error in
More informationLecture Notes on Real-world SMT
15-414: Bug Catching: Automated Program Verification Lecture Notes on Real-world SMT Matt Fredrikson Ruben Martins Carnegie Mellon University Lecture 15 1 Introduction In the previous lecture we studied
More informationTesting Theory. Agenda - What will you learn today? A Software Life-cycle Model Which part will we talk about today? Theory Lecture Plan
heory Lecture Plan 2 esting heory Lecture 8 Software Engineering DDC88/DDC93 autumn 28 Department of Computer and Information Science Linköping University, Sweden L - Course Introduction and Overview L2
More informationRuntime Checking and Test Case Generation for Python
Runtime Checking and Test Case Generation for Python Anna Durrer Master Thesis Chair of Programming Methodology D-INFK ETH Supervisor: Marco Eilers, Prof. Peter Müller 24. Mai 2017 1 Introduction This
More information