Testing, Fuzzing, & Symbolic Execution
|
|
- George Chambers
- 5 years ago
- Views:
Transcription
1 Testing, Fuzzing, & Symbolic Execution
2 Software Testing The most common way of measuring & ensuring correctness Input 2
3 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior 3
4 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle 4
5 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome 5
6 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome Test Suite Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 6
7 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome Key Issues: Are the tests adequate? Test Suite Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 7
8 Software Testing The most common way of measuring & ensuring correctness Input? Observed Behavior Oracle Outcome Key Issues: Are the tests adequate? Automated input generation Test Suite Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 8
9 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome Key Issues: Are the tests adequate? Automated input generation Automated oracles Test Suite Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 9
10 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome Test 1 Are the tests adequate? Test 2 Test 3 Automated input generation Test 4 Automated oracles Test 5 Test 6 Robustness / flakiness / maintainability Test 7 Key Issues: Test Suite 10
11 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior Oracle Outcome Test 1 Are the tests adequate? Test 2 Test 3 Automated input generation Test 4 Automated oracles Test 5 Test 6 Robustness / flakiness / maintainability Test 7... Key Issues: Test Suite 11
12 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? 12
13 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage def my_lovely_fun(a,b,c): if (a && b) c: Is each statement executed by at least one test in the test suite? else: print('awesome') score= # covered # statements 13
14 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage score= # covered # branches def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') b T a c F 14
15 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage More common in safety critical systems where full coverage may be required. def my_lovely_fun(a,b,c): if (a & b) c: else: print('awesome') 15
16 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage def my_lovely_fun(a,b,c): if (a & b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') 16
17 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage # covered/killed score= # non-equivalent mutants def my_lovely_fun(a,b,c): if (a & b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') 17
18 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage # covered/killed score= # non-equivalent mutants def my_lovely_fun(a,b,c): if (a & b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') def my_lovely_fun(a,b,c): if (a && b) c: else: print('awesome') 18
19 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage Path Coverage B E ENTRY A D C F EXIT 19
20 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage Path Coverage Can apply EPP! B E ENTRY A D C F EXIT 20
21 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage Path Coverage... 21
22 Test Suite Adequacy Questions Is a test suite good enough? What parts of software need to be tested better? Metrics Statement Coverage Branch Coverage MC/DC Coverage Mutation Coverage Path Coverage BUT reducing test suites for St, Br, MC/DC coverage decrease defect detection! 22
23 Generating Inputs Sample all possible inputs for test in allpossibleinputs: check_test(test) 23
24 Generating Inputs Sample all possible inputs for test in allpossibleinputs: check_test(test) Target specific goals or other coverage criteria for s in statements: test = findtestthrough(s) check_test(test) 24
25 Generating Inputs Sample all possible inputs for test in allpossibleinputs: check_test(test) Target specific goals or other coverage criteria for s in statements: test = findtestthrough(s) check_test(test) for i in inputmodel: test = findrepresentative(i) check_test(test) 25
26 Generating Inputs Usually broken into black box & white box approaches 26
27 Generating Inputs Usually broken into black box & white box approaches Black Box Treat the program as opaque / unknown e.g. specification based, naive fuzzing, boundary value analysis,... 27
28 Generating Inputs Usually broken into black box & white box approaches Black Box Treat the program as opaque / unknown White Box Program structure & semantics can be used e.g. symbolic execution, call chain synthesis, white box fuzzing, boundary value analysis,... 28
29 Generating Oracles? 29
30 Generating Oracles Likely invariants? Careful variable selection & monitoring? Differential Testing Metamorphic Testing A very open (hard) problem. 30
31 Interesting Problems Random Testing Test Suite Adequacy Test Suite Minimization Test Generation Oracle Generation Test Maintenance Performance Testing Field Testing Power Testing Test Prioritization... 31
32 Interesting Problems Random Testing Test Suite Adequacy Test Suite Minimization Test Generation Oracle Generation Test Maintenance Performance Testing Field Testing Power Testing Test Prioritization... Test generation techniques have also proven to be critical in security research. 32
33 Fuzz Testing An approach for generating test inputs
34 Fuzz Testing An approach for generating test inputs Originally just feeding large random inputs to programs [Miller 1990]./grep 02d6 RandomFile
35 Fuzz Testing An approach for generating test inputs Originally just feeding large random inputs to programs [Miller 1990]./grep 02d6 RandomFile It was distressingly effective at finding buffer overflows (25%-33% of programs).
36 Fuzz Testing An approach for generating test inputs Originally just feeding large random inputs to programs [Miller 1990] Now 2 main types
37 Fuzz Testing An approach for generating test inputs Originally just feeding large random inputs to programs [Miller 1990] Now 2 main types 1) Generational (model based) Creates entirely new inputs
38 Fuzz Testing An approach for generating test inputs Originally just feeding large random inputs to programs [Miller 1990] Now 2 main types 1) Generational (model based) Creates entirely new inputs Needs a model for the input
39 Fuzz Testing An approach for generating test inputs Originally just feeding large random inputs to programs [Miller 1990] Now 2 main types 1) Generational (model based) Creates entirely new inputs Needs a model for the input a*bc(d e)c*
40 Fuzz Testing An approach for generating test inputs Originally just feeding large random inputs to programs [Miller 1990] Now 2 main types 1) Generational (model based) Creates entirely new inputs Needs a model for the input a*bc(d e)c* A aab A ca A ε
41 Fuzz Testing An approach for generating test inputs Originally just feeding large random inputs to programs [Miller 1990] Now 2 main types 1) Generational (model based) Creates entirely new inputs Needs a model for the input a*bc(d e)c* A aab A ca A ε
42 Fuzz Testing An approach for generating test inputs Originally just feeding large random inputs to programs [Miller 1990] Now 2 main types 1) Generational (model based) 2) Mutational (heuristic change based) Modify an existing test suite
43 Fuzz Testing An approach for generating test inputs Originally just feeding large random inputs to programs [Miller 1990] Now 2 main types 1) Generational (model based) 2) Mutational (heuristic change based) Modify an existing test suite Seeing a resurgance via AFL & libfuzzer
44 Symbolic Execution An approach for generating test inputs. Cadar & Sen, 2013 x input() y input() if x == 2*y if x > y+10 44
45 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 45
46 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 46
47 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 x = 2*y y > 10 Path Constraint 47
48 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 A path constraint represents all executions along that path x = 2*y y > 10 Path Constraint 48
49 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Solve for the symbolic symbols to find inputs that yield the path. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 x=30 y=15 49
50 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Solve for the symbolic symbols to find inputs that yield the path. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 x=30 y=15 x=2 y=1 50
51 Symbolic Execution An approach for generating test inputs. Replace the concrete inputs of a program with symbolic values Execute along a path using the symbolic values to build a formula over the input symbols. Solve for the symbolic symbols to find inputs that yield the path. Cadar & Sen, 2013 x symbolic() y symbolic() if x == 2*y if x > y+10 x=30 y=15 x=2 y=1 x=0 y=1 51
52 How Can We Solve Constraints? SMT Solvers Satisfiability Modulo Theories SAT with extra logic Standard interfaces through SMTLIB2 52
53 How Can We Solve Constraints? SMT Solvers Satisfiability Modulo Theories SAT with extra logic Standard interfaces through SMTLIB2 x = 2*y y > 10 (declare-const x Int) (declare-const y Int) (assert (= x (* 2 y))) (assert (> y 10)) (check-sat) (get-model) 53
54 How Can We Solve Constraints? SMT Solvers Satisfiability Modulo Theories SAT with extra logic Standard interfaces through SMTLIB2 x = 2*y y > 10 (declare-const x Int) (declare-const y Int) (assert (= x (* 2 y))) (assert (> y 10)) (check-sat) (get-model) Z3 54
55 How Can We Solve Constraints? SMT Solvers Satisfiability Modulo Theories SAT with extra logic Standard interfaces through SMTLIB2 x = 2*y y > 10 (declare-const x Int) (declare-const y Int) (assert (= x (* 2 y))) (assert (> y 10)) (check-sat) (get-model) x=22 y=11 Z3 sat (model (define-fun y () Int 11) (define-fun x () Int 22) ) 55
56 How Can We Solve Constraints? SMT Solvers Satisfiability Modulo Theories SAT with extra logic Standard interfaces through SMTLIB2 x = 2*y y > 10 (declare-const x Int) (declare-const y Int) (assert (= x (* 2 y))) (assert (> y 10)) (check-sat) (get-model) x=22 y=11 Z3 sat (model (define-fun y () Int 11) (define-fun x () Int 22) ) Try it online: 56
57 Useful Questions If ɸ holds after a statement, what must have been true at the point before? weakest precondition 57
58 Useful Questions If ɸ holds after a statement, what must have been true at the point before? weakest precondition If ɸ holds before a statement, what can we guarantee to be true after? strongest poscondition 58
59 Useful Questions If ɸ holds after a statement, what must have been true at the point before? weakest precondition If ɸ holds before a statement, what can we guarantee to be true after? strongest poscondition e.g. Given two versions of a program v1,v2 and assertions on output ɸi in each from an input I What is wp(ɸ1) wp(ɸ2)? 59
60 Useful Questions If ɸ holds after a statement, what must have been true at the before? weakest precondition If ɸ holds before a statement, what can we guarantee to be true after? strongest poscondition e.g. Given two versions of a program v1,v2 and assertions on output ɸi in each from an input I What is wp(ɸ1) wp(ɸ2)? wp(ɸ1) wp(ɸ2) 60
61 Useful Questions If ɸ holds after a statement, what must have been true at the before? weakest precondition If ɸ holds before a statement, what can we guarantee to be true after? strongest poscondition e.g. Given two versions of a program v1,v2 and assertions on output ɸi in each from an input I What is wp(ɸ1) wp(ɸ2)? wp(ɸ1) wp(ɸ2) What is the intuitive meaning? 61
62 Exploring the Execution Tree The possible paths of a program form an execution tree. x input() y input() Cadar & Sen, 2013 if x == 2*y if x > y+10 62
63 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. x input() y input() if x == 2*y Cadar & Sen, 2013 if x > y+10 63
64 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches x input() y input() if x == 2*y if x > y+10 Cadar & Sen,
65 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) x input() y input() if x == 2*y if x > y+10 Cadar & Sen,
66 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) x input() y input() if x == 2*y if x > y+10 Cadar & Sen,
67 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) x input() y input() if x == 2*y if x > y+10 Cadar & Sen, 2013 (x=2*y) (x>y+10) 67
68 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) x input() y input() if x == 2*y if x > y+10 Cadar & Sen, 2013 (x=2*y) (x>y+10) 68
69 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) x input() y input() if x == 2*y if x > y+10 Cadar & Sen,
70 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) Execution Generated Testing x input() y input() if x == 2*y if x > y+10 Cadar & Sen,
71 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches Concolic (dynamic symbolic) Execution Generated Testing x input() y input() if x == 2*y if x > y+10 Cadar & Sen, 2013 X=? y=10 71
72 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches X=20 Concolic (dynamic symbolic) y=10 Execution Generated Testing x input() y input() if x == 2*y if x > y+10 Cadar & Sen, 2013 X=? 20 y=10 72
73 Exploring the Execution Tree The possible paths of a program form an execution tree. Traversing the tree will yield tests for all paths. Mechanizing the traversal yields two main approaches X=20 Concolic (dynamic symbolic) y=10 Execution Generated Testing Execution on this side is concrete from this point on. x input() y input() if x == 2*y if x > y+10 Cadar & Sen, 2013 X=? 20 y=10 73
74 Symbolic Execution Increasingly scalable every year 74
75 Symbolic Execution Increasingly scalable every year Can automatically generate test inputs from constraints 75
76 Symbolic Execution Increasingly scalable every year Can automatically generate test inputs from constraints The resulting symbolic formulae have many uses beyond just testing. 76
77 Symbolic Execution Increasingly scalable every year Can automatically generate test inputs from constraints The resulting symbolic formulae have many uses beyond just testing. Try it out: 1) 2) Symbolic PathFinder 3) 4) 77
Testing & Symbolic Execution
Testing & Symbolic Execution Software Testing The most common way of measuring & ensuring correctness Input 2 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior
More informationCS 520 Theory and Practice of Software Engineering Fall 2018
Today CS 52 Theory and Practice of Software Engineering Fall 218 Software testing October 11, 218 Introduction to software testing Blackbox vs. whitebox testing Unit testing (vs. integration vs. system
More informationSymbolic Execution. Wei Le April
Symbolic Execution Wei Le 2016 April Agenda What is symbolic execution? Applications History Interal Design: The three challenges Path explosion Modeling statements and environments Constraint solving
More informationPart I: Preliminaries 24
Contents Preface......................................... 15 Acknowledgements................................... 22 Part I: Preliminaries 24 1. Basics of Software Testing 25 1.1. Humans, errors, and testing.............................
More informationSymbolic and Concolic Execution of Programs
Symbolic and Concolic Execution of Programs Information Security, CS 526 Omar Chowdhury 10/7/2015 Information Security, CS 526 1 Reading for this lecture Symbolic execution and program testing - James
More informationTesting! Prof. Leon Osterweil! CS 520/620! Spring 2013!
Testing Prof. Leon Osterweil CS 520/620 Spring 2013 Relations and Analysis A software product consists of A collection of (types of) artifacts Related to each other by myriad Relations The relations are
More informationn HW7 due in about ten days n HW8 will be optional n No CLASS or office hours on Tuesday n I will catch up on grading next week!
Announcements SMT Solvers, Symbolic Execution n HW7 due in about ten days n HW8 will be optional n No CLASS or office hours on Tuesday n I will catch up on grading next week! n Presentations n Some of
More informationA short manual for the tool Accumulator
A short manual for the tool Accumulator ZHAO Jianhua State Key Laboratory of Novel Software Technology Dept. of Computer Sci. and Tech. Nanjing University Nanjing, Jiangsu, P.R.China 210093 zhaojh@nju.edu.cn
More informationThree Important Testing Questions
Testing Part 2 1 Three Important Testing Questions How shall we generate/select test cases? Did this test execution succeed or fail? How do we know when we ve tested enough? 65 1. How do we know when we
More informationSoftware Testing. Testing: Our Experiences
Software Testing Testing: Our Experiences Test Case Software to be tested Output 1 Test Case Generation When to Stop? Test Case Software to be tested Verification Output No Enough? Test Coverage Yes A
More informationFacts About Testing. Cost/benefit. Reveal faults. Bottom-up. Testing takes more than 50% of the total cost of software development
Reveal faults Goals of testing Correctness Reliability Usability Robustness Performance Top-down/Bottom-up Bottom-up Lowest level modules tested first Don t depend on any other modules Driver Auxiliary
More information6.0 ECTS/4.5h VU Programm- und Systemverifikation ( ) June 22, 2016
6.0 ECTS/4.5h VU Programm- und Systemverifikation (184.741) June 22, 2016 Kennzahl (study id) Matrikelnummer (student id) Familienname (family name) Vorname (first name) Gruppe (version) A 1.) Coverage
More informationStatic Analysis and Bugfinding
Static Analysis and Bugfinding Alex Kantchelian 09/12/2011 Last week we talked about runtime checking methods: tools for detecting vulnerabilities being exploited in deployment. So far, these tools have
More information6. Test-Adequacy. Assessment Using Control Flow and Data Flow. Andrea Polini
6. Test-Adequacy Assessment Using Control Flow and Data Flow Andrea Polini Software Engineering II Software Testing MSc in Computer Science University of Camerino (Software Engineering II Software Testing)
More informationMTAT : Software Testing
MTAT.03.159: Software Testing Lecture 03: White-Box Testing (Textbook Ch. 5) Spring 2013 Dietmar Pfahl email: dietmar.pfahl@ut.ee Lecture Chapter 5 White-box testing techniques (Lab 3) Structure of Lecture
More informationSubject Software Testing Structural Testing
Subject Software Testing Structural Testing Objective: 1. Understand Concept of structural testing 2. How structural (code-based or glass-box) testing complements functional (black-box) testing 3. Recognize
More informationTesting Methods: White Box Testing II
Testing Methods: White Box Testing II Outline Today we continue our look at white box testing with more code coverage methods, and a data coverage method We ll look at : - code coverage testing - decision
More informationWritten exam TDDD04 Software Testing
LiTH, Linköpings tekniska högskola IDA, Institutionen för datavetenskap Ola Leifler Written exam TDDD04 Software Testing 2016-10-26 Permissible aids Dictionary (printed, NOT electronic) Teacher on duty
More informationMTAT : Software Testing
MTAT.03.159: Software Testing Lecture 03: White-Box Testing (Textbook Ch. 5) Dietmar Pfahl Spring 2016 email: dietmar.pfahl@ut.ee Lecture Chapter 5 White-box testing techniques (Lab 3) Structure of Lecture
More informationCYSE 411/AIT681 Secure Software Engineering Topic #17: Symbolic Execution
CYSE 411/AIT681 Secure Software Engineering Topic #17: Symbolic Execution Instructor: Dr. Kun Sun Software has bugs To find them, we use testing and code reviews But some bugs are still missed Rare features
More informationSoftware has bugs. Static analysis 4/9/18. CYSE 411/AIT681 Secure Software Engineering. To find them, we use testing and code reviews
CYSE 411/AIT681 Secure Software Engineering Topic #17: Symbolic Execution Instructor: Dr. Kun Sun Software has bugs To find them, we use testing and code reviews But some bugs are still missed Rare features
More informationCMSC 430 Introduction to Compilers. Fall Symbolic Execution
CMSC 430 Introduction to Compilers Fall 2015 Symbolic Execution Introduction Static analysis is great Lots of interesting ideas and tools Commercial companies sell, use static analysis It all looks good
More informationTesting: Coverage and Structural Coverage
Testing: Coverage and Structural Coverage Testing, Quality Assurance, and Maintenance Winter 2017 Prof. Arie Gurfinkel based on slides by Prof. Marsha Chechik and Prof. Lin Tan How would you test this
More informationInformation page for written examinations at Linköping University
Information page for written examinations at Linköping University Examination date 2017-08-23 Room (1) Time 8-12 Course code Exam code Course name Exam name Department Number of questions in the examination
More informationWhat is Mutation Testing? Mutation Testing. Test Case Adequacy. Mutation Testing. Mutant Programs. Example Mutation
What is Mutation Testing? Mutation Testing Breaking the application to test it n Mutation Testing is a testing technique that focuses on measuring the adequacy of test cases n Mutation Testing is NOT a
More informationWhite Box Testing III
White Box Testing III Outline Today we continue our look at white box testing methods, with mutation testing We will look at : definition and role of mutation testing what is a mutation? how is mutation
More informationMTAT : Software Testing
MTAT.03.159: Software Testing Lecture 03: White-Box Testing (Textbook Ch. 5) Dietmar Pfahl Spring 2017 email: dietmar.pfahl@ut.ee Lecture Chapter 5 White-box testing techniques (Lab 3) Structure of Lecture
More informationTesting Role in Unified Approach Coverage: Structural/Coverage Model Based Test Generation from Model Checking (project) Interaction of
Testing Role in Unified Approach Coverage: Structural/Coverage Model Based Test Generation from Model Checking (project) Interaction of Coverage/Model Based Testing Will Not Cover Statistical Methods Partition
More informationTesting: Test design and testing process
Testing: Test design and testing process Zoltán Micskei Based on István Majzik s slides Dept. of Measurement and Information Systems Budapest University of Technology and Economics Department of Measurement
More informationGNATprove a Spark2014 verifying compiler Florian Schanda, Altran UK
1 GNATprove a Spark2014 verifying compiler Florian Schanda, Altran UK Tool architecture User view Source gnatprove Verdict 2 Tool architecture More detailed view... Source Encoding CVC4 gnat2why gnatwhy3
More informationIntroduction to Dynamic Analysis
Introduction to Dynamic Analysis Reading assignment Gary T. Leavens, Yoonsik Cheon, "Design by Contract with JML," draft paper, http://www.eecs.ucf.edu/~leavens/jml//jmldbc.pdf G. Kudrjavets, N. Nagappan,
More informationClass 17. Discussion. Mutation analysis and testing. Problem Set 7 discuss Readings
Class 17 Questions/comments Graders for Problem Set 6 (4); Graders for Problem set 7 (2-3) (solutions for all); will be posted on T-square Regression testing, Instrumentation Final project presentations:
More informationSoftware Quality Assurance. David Janzen
Software Quality Assurance David Janzen What is quality? Crosby: Conformance to requirements Issues: who establishes requirements? implicit requirements Juran: Fitness for intended use Issues: Who defines
More informationCMPT 473 Software Quality Assurance. Graph Coverage. Nick Sumner
CMPT 473 Software Quality Assurance Graph Coverage Nick Sumner Recall: Coverage/Adequacy Can't look at all possible inputs. Need to determine if a test suite covers / is adequate for our quality objectives.
More informationTest Automation. 20 December 2017
Test Automation 20 December 2017 The problem of test automation Testing has repetitive components, so automation is justified The problem is cost-benefit evaluation of automation [Kaner] Time for: test
More informationAutomated Software Analysis Techniques For High Reliability: A Concolic Testing Approach. Moonzoo Kim
Automated Software Analysis Techniques For High Reliability: A Concolic Testing Approach Moonzoo Kim Contents Automated Software Analysis Techniques Background Concolic testing process Example of concolic
More informationAutomatic Software Verification
Automatic Software Verification Instructor: Mooly Sagiv TA: Oded Padon Slides from Eran Yahav and the Noun Project, Wikipedia Course Requirements Summarize one lecture 10% one lecture notes 45% homework
More informationSecond assignment came out Monday evening. Find defects in Hnefetafl rules written by your classmates. Topic: Code Inspection and Testing
Announcements Second assignment came out Monday evening Topic: Code Inspection and Testing Find defects in Hnefetafl rules written by your classmates Compare inspection, coverage testing, random testing,
More informationCMSC 631 Program Analysis and Understanding. Spring Symbolic Execution
CMSC 631 Program Analysis and Understanding Spring 2013 Symbolic Execution Introduction Static analysis is great Lots of interesting ideas and tools Commercial companies sell, use static analysis It all
More informationTesting. ECE/CS 5780/6780: Embedded System Design. Why is testing so hard? Why do testing?
Testing ECE/CS 5780/6780: Embedded System Design Scott R. Little Lecture 24: Introduction to Software Testing and Verification What is software testing? Running a program in order to find bugs (faults,
More informationTesting Tactics. Structural Testing. Why Structural? Why Structural? Functional black box. Structural white box. Functional black box
ing Tactics Structural ing Functional black box Structural white box Software Engineering Andreas Zeller Saarland University s based on spec covers as much specified behavior as possible s based on code
More informationSymbolic Execu.on. Suman Jana
Symbolic Execu.on Suman Jana Acknowledgement: Baishakhi Ray (Uva), Omar Chowdhury (Purdue), Saswat Anand (GA Tech), Rupak Majumdar (UCLA), Koushik Sen (UCB) What is the goal? Tes.ng Tes%ng approaches are
More informationAdvanced Test Coverage Criteria: Specify and Measure, Cover and Unmask
Advanced Test Coverage Criteria: Specify and Measure, Cover and Unmask Sébastien Bardin & Nikolai Kosmatov joint work with Omar Chebaro, Robin David, Mickaël Delahaye, Michaël Marcozzi, Mike Papadakis,
More informationVerifying C & C++ with ESBMC
Verifying C & C++ with ESBMC Denis A Nicole dan@ecs.soton.ac.uk CyberSecuritySoton.org [w] @CybSecSoton [fb & tw] ESBMC ESBMC, the Efficient SMT-Based Context-Bounded Model Checker was originally developed
More informationAbstract Interpretation
Abstract Interpretation Ranjit Jhala, UC San Diego April 22, 2013 Fundamental Challenge of Program Analysis How to infer (loop) invariants? Fundamental Challenge of Program Analysis Key issue for any analysis
More informationCounterexample-Driven Genetic Programming
Counterexample-Driven Genetic Programming Iwo Błądek, Krzysztof Krawiec Institute of Computing Science, Poznań University of Technology Poznań, 12.12.2017 I. Błądek, K. Krawiec Counterexample-Driven Genetic
More informationAnalysis/Bug-finding/Verification for Security
Analysis/Bug-finding/Verification for Security VIJAY GANESH University of Waterloo Winter 2013 Analysis/Test/Verify for Security Instrument code for testing Heap memory: Purify Perl tainting (information
More informationTest Design Techniques ISTQB (International Software Testing Qualifications Board)
Test Design Techniques ISTQB (International Software Testing Qualifications Board) Minsoo Ryu Hanyang University Testing Process Planning and Control Analysis and Design Implementation and Execution Evaluating
More informationCMPSCI 521/621 Homework 2 Solutions
CMPSCI 521/621 Homework 2 Solutions Problem 1 Direct data dependencies: 3 is directly data dependent on 1 and 5 5 is directly data dependent on 1,3, and 5 7 is directly data dependent on 1,3, and 5 Note,
More informationGenerating Small Countermodels. Andrew Reynolds Intel August 30, 2012
Generating Small Countermodels using SMT Andrew Reynolds Intel August 30, 2012 Acknowledgements Intel Corporation AmitGoel, Sava Krstic University of Iowa Cesare Tinelli, Francois Bobot New York University
More informationSTRUCTURAL TESTING. AKA White Box Testing. Thanks go to Andreas Zeller for allowing incorporation of his materials. F. Tip and M.
F. Tip and M. Weintraub STRUCTURAL TESTING AKA White Box Testing Thanks go to Andreas Zeller for allowing incorporation of his materials STRUCTURAL TESTING Testing based on the structure of the code Test
More informationProperties of Criteria. 1. Applicability Property. Criteria
Properties of Criteria Program-based To recognize a good adequacy criteria And to discard poor choices Objective, well-defined properties 1. Applicability Property For every program, there exists an adequate
More informationOpenMath and SMT-LIB
James, Matthew England, Roberto Sebastiani & Patrick Trentin 1 Universities of Bath/Coventry/Trento/Trento J.H.@bath.ac.uk 17 July 2017 1 Thanks to EU H2020-FETOPEN-2016-2017-CSA project SC 2 (712689)
More informationSmall Formulas for Large Programs: On-line Constraint Simplification In Scalable Static Analysis
Small Formulas for Large Programs: On-line Constraint Simplification In Scalable Static Analysis Isil Dillig, Thomas Dillig, Alex Aiken Stanford University Scalability and Formula Size Many program analysis
More informationTesting: Coverage and Structural Coverage
Testing: Coverage and Structural Coverage Testing, Quality Assurance, and Maintenance Winter 2018 Prof. Arie Gurfinkel based on slides by Prof. Marsha Chechik and Prof. Lin Tan How would you test this
More informationSpecification Centered Testing
Specification Centered Testing Mats P. E. Heimdahl University of Minnesota 4-192 EE/CS Building Minneapolis, Minnesota 55455 heimdahl@cs.umn.edu Sanjai Rayadurgam University of Minnesota 4-192 EE/CS Building
More informationEECS 481 Software Engineering Exam #1. You have 1 hour and 20 minutes to work on the exam.
EECS 481 Software Engineering Exam #1 Write your name and UM uniqname on the exam. There are ten (10) pages in this exam (including this one) and seven (7) questions, each with multiple parts. Some questions
More informationIntroduction to Symbolic Execution
Introduction to Symbolic Execution Classic Symbolic Execution 1 Problem 1: Infinite execution path Problem 2: Unsolvable formulas 2 Problem 3: symbolic modeling External function calls and system calls
More informationStructural Testing. (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 1
Structural Testing (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 1 Learning objectives Understand rationale for structural testing How structural (code-based or glass-box) testing complements functional
More informationSymbolic Execution for Bug Detection and Automated Exploit Generation
Symbolic Execution for Bug Detection and Automated Exploit Generation Daniele Cono D Elia Credits: Emilio Coppa SEASON Lab season-lab.github.io May 27, 2016 1 / 29 Daniele Cono D Elia Symbolic Execution
More informationUsing the code to measure test adequacy (and derive test cases) Structural Testing
Using the code to measure test adequacy (and derive test cases) Structural Testing Objectives To describe a second approach to testing which is geared to find program defects To explain the use of program
More informationSatisfiability Modulo Theories: ABsolver
Satisfiability Modulo Theories: ABsolver Michael Tautschnig Joint work with: Andreas Bauer Martin Leucker Christian Schallhart Michael Tautschnig 1 Outline 1. Introduction Michael Tautschnig 2 Outline
More informationInformation page for written examinations at Linköping University
Information page for written examinations at Linköping University Examination date 2017-01-14 Room (1) Time 8-12 Course code Exam code Course name Exam name Department Number of questions in the examination
More informationSTRUCTURAL TESTING. AKA White Box Testing. Thanks go to Andreas Zeller for allowing incorporation of his materials. F. Tip and M.
F. Tip and M. Weintraub STRUCTURAL TESTING AKA White Box Testing Thanks go to Andreas Zeller for allowing incorporation of his materials STRUCTURAL TESTING Testing based on the structure of the code Test
More informationFormal Methods: Model Checking and Other Applications. Orna Grumberg Technion, Israel. Marktoberdorf 2017
Formal Methods: Model Checking and Other Applications Orna Grumberg Technion, Israel Marktoberdorf 2017 1 Outline Model checking of finite-state systems Assisting in program development Program repair
More informationAutomated White-Box Testing Beyond Branch Coverage
Bardin et al. Dagstuhl seminar 14442 1/ 22 Automated White-Box Testing Beyond Branch Coverage Sébastien Bardin, Nikolai Kosmatov, Mickaël Delahaye CEA LIST, Software Safety Lab (Paris-Saclay, France) The
More informationSoftware Testing for Developer Development Testing. Duvan Luong, Ph.D. Operational Excellence Networks
Software Testing for Developer Development Testing Duvan Luong, Ph.D. Operational Excellence Networks Contents R&D Testing Approaches Static Analysis White Box Testing Black Box Testing 4/2/2012 2 Development
More informationSymbolic Execution, Dynamic Analysis
Symbolic Execution, Dynamic Analysis http://d3s.mff.cuni.cz Pavel Parízek CHARLES UNIVERSITY IN PRAGUE faculty of mathematics and physics Symbolic execution Pavel Parízek Symbolic Execution, Dynamic Analysis
More informationSymbolic Execution. Michael Hicks. for finding bugs. CMSC 631, Fall 2017
Symbolic Execution for finding bugs Michael Hicks CMSC 631, Fall 2017 Software has bugs To find them, we use testing and code reviews But some bugs are still missed Rare features Rare circumstances Nondeterminism
More informationTest Oracles and Mutation Testing. CSCE Lecture 23-11/18/2015
Test Oracles and Mutation Testing CSCE 740 - Lecture 23-11/18/2015 Software Testing - Back to the Basics Tests are sequences of stimuli and observations. We care about input and output. (I 1 O 1 ) (I 2
More informationlogistics: ROP assignment
bug-finding 1 logistics: ROP assignment 2 2013 memory safety landscape 3 2013 memory safety landscape 4 different design points memory safety most extreme disallow out of bounds usually even making out-of-bounds
More informationReducing Verification Costs through Practical Formal Methods: A Survey
Reducing Verification Costs through Practical Formal Methods: A Survey Mark R. Blackburn, Ph.D. Stevens Institute of Technology Sumit Ray BAE Systems Abstract Verification of software can be as much as
More informationCUTE: A Concolic Unit Testing Engine for C
CUTE: A Concolic Unit Testing Engine for C Koushik Sen Darko Marinov Gul Agha University of Illinois Urbana-Champaign Goal Automated Scalable Unit Testing of real-world C Programs Generate test inputs
More informationCSE 565 Computer Security Fall 2018
CSE 565 Computer Security Fall 2018 Lecture 16: Building Secure Software Department of Computer Science and Engineering University at Buffalo 1 Review A large number of software vulnerabilities various
More informationTesting Theory. Agenda - What will you learn today? A Software Life-cycle Model Which part will we talk about today? Theory Lecture Plan
heory Lecture Plan 2 esting heory Lecture 8 Software Engineering DDC88/DDC93 autumn 28 Department of Computer and Information Science Linköping University, Sweden L - Course Introduction and Overview L2
More informationSoftware Testing CS 408. Lecture 6: Dynamic Symbolic Execution and Concolic Testing 1/30/18
Software Testing CS 408 Lecture 6: Dynamic Symbolic Execution and Concolic Testing 1/30/18 Relevant Papers CUTE: A Concolic Unit Testing Engine for C Koushik Sen, Darko Marinov, Gul Agha Department of
More informationStructural Testing. White Box Testing & Control Flow Analysis
Structural Testing White Box Testing & Control Flow Analysis Functional vs. Structural Functional Have I built the right product? Tests derived from the program specification Internal Structure ignored
More informationAn Introduction to Satisfiability Modulo Theories
An Introduction to Satisfiability Modulo Theories Philipp Rümmer Uppsala University Philipp.Ruemmer@it.uu.se February 13, 2019 1/28 Outline From theory... From DPLL to DPLL(T) Slides courtesy of Alberto
More informationTesting. Prof. Clarkson Fall Today s music: Wrecking Ball by Miley Cyrus
Testing Prof. Clarkson Fall 2017 Today s music: Wrecking Ball by Miley Cyrus Review Previously in 3110: Modules Specification (functions, modules) Today: Validation Testing Black box Glass box Randomized
More informationStructural Testing. Testing Tactics. Why Structural? Structural white box. Functional black box. Structural white box. Functional black box
From Pressman, Software Engineering a practitioner s approach, Chapter 14 and Pezze + Young, Software Testing and Analysis, Chapters 12-13 Structural Testing Software Engineering Andreas Zeller Saarland
More informationA survey of new trends in symbolic execution for software testing and analysis
Int J Softw Tools Technol Transfer (2009) 11:339 353 DOI 10.1007/s10009-009-0118-1 REGULAR PAPER A survey of new trends in symbolic execution for software testing and analysis Corina S. Păsăreanu Willem
More informationSoftware Testing TEST CASE SELECTION AND ADEQUECY TEST EXECUTION
Software Testing TEST CASE SELECTION AND ADEQUECY TEST EXECUTION Overview, Test specification and cases, Adequacy criteria, comparing criteria, Overview of test execution, From test case specification
More informationAdvanced Software Testing Understanding Code Coverage
Advanced Software Testing Understanding Code Coverage Advanced Software Testing A series of webinars, this one excerpted from Advanced Software Testing: V3, a book for technical test analysts, programmers,
More informationCSE507. Practical Applications of SAT. Computer-Aided Reasoning for Software. Emina Torlak
Computer-Aided Reasoning for Software CSE507 Practical Applications of SAT courses.cs.washington.edu/courses/cse507/18sp/ Emina Torlak emina@cs.washington.edu Today Past 2 lectures The theory and mechanics
More informationQemu code fault automatic discovery with symbolic search. Paul Marinescu, Cristian Cadar, Chunjie Zhu, Philippe Gabriel
Qemu code fault automatic discovery with symbolic search Paul Marinescu, Cristian Cadar, Chunjie Zhu, Philippe Gabriel Goals of this presentation Introduction of KLEE (symbolic execution tool) Qemu fault/patch
More informationSoftware security, secure programming
Software security, secure programming Fuzzing and Dynamic Analysis Master on Cybersecurity Master MoSiG Academic Year 2017-2018 Outline Fuzzing (or how to cheaply produce useful program inputs) A concrete
More informationLecture 18: Structure-based Testing
Test Case First Strategy White box testing: Statement Coverage Branch Coverage Condition Coverage Data Path Coverage Lecture 18: Structure-based Testing Testing with good and bad data Testing Object Oriented
More informationMutation Testing. Leaving the Stone Age
Mutation Testing Leaving the Stone Age 2017 whoami ios Developer by day compiler hacker by night https://twitter.com/1101_debian https://lowlevelbits.org https://systemundertest.org Outline Quality of
More informationMTAT : Software Testing
MTAT.03.159: Software Testing Lecture 02: Basic Black-Box and White-Box Testing Techniques (Textbook Ch. 4 & 5) Spring 2018 Dietmar Pfahl email: dietmar.pfahl@ut.ee Structure of Lecture 2 Black-Box vs.
More informationProgramming Embedded Systems
Programming Embedded Systems Lecture 8 Overview of software testing Wednesday Feb 8, 2012 Philipp Rümmer Uppsala University Philipp.Ruemmer@it.uu.se 1/53 Lecture outline Testing in general Unit testing
More informationSecurity Testing. Ulf Kargén Department of Computer and Information Science (IDA) Division for Database and Information Techniques (ADIT)
Security Testing TDDC90 Software Security Ulf Kargén Department of Computer and Information Science (IDA) Division for Database and Information Techniques (ADIT) Security testing vs regular testing Regular
More informationKLEE Workshop Feeding the Fuzzers. with KLEE. Marek Zmysłowski MOBILE SECURITY TEAM R&D INSTITUTE POLAND
Feeding the Fuzzers with KLEE Marek Zmysłowski MOBILE SECURITY TEAM R&D INSTITUTE POLAND This presentation was created with help and commitment of the Samsung R&D Poland Mobile Security team. KLEE and
More informationTesting, Debugging, Program Verification
Testing, Debugging, Program Verification Automated Test Case Generation, Part II Wolfgang Ahrendt & Vladimir Klebanov & Moa Johansson 12 December 2012 TDV: ATCG II /GU 2011-12-12 1 / 17 Recap Specification-/Model-Based
More informationSoftware Architecture and Engineering: Part II
Software Architecture and Engineering: Part II ETH Zurich, Spring 2016 Prof. http://www.srl.inf.ethz.ch/ Framework SMT solver Alias Analysis Relational Analysis Assertions Second Project Static Analysis
More informationMLSA: a static bugs analysis tool based on LLVM IR
International Journal of Networked and Distributed Computing, Vol. 4, No. 3 (July 2016), 137-144 MLSA: a static bugs analysis tool based on LLVM IR Hongliang Liang 1, Lei Wang 1, Dongyang Wu 1, Jiuyun
More informationFault-based testing. Automated testing and verification. J.P. Galeotti - Alessandra Gorla. slides by Gordon Fraser. Thursday, January 17, 13
Fault-based testing Automated testing and verification J.P. Galeotti - Alessandra Gorla slides by Gordon Fraser How good are my tests? How good are my tests? Path testing Boundary interior testing LCSAJ
More informationAppendix to The Health of Software Engineering Research
Appendix to The Health of Software Engineering Research David Lo School of Information Systems Singapore Management University Singapore davidlo@smu.edu.sg Nachiappan Nagappan and Thomas Zimmermann Research
More informationLecture Notes on Real-world SMT
15-414: Bug Catching: Automated Program Verification Lecture Notes on Real-world SMT Matt Fredrikson Ruben Martins Carnegie Mellon University Lecture 15 1 Introduction In the previous lecture we studied
More information18-642: Unit Testing 1/31/ Philip Koopman
18-642: Unit Testing 1/31/2018 2017-2018 Philip Koopman YOU ARE HERE Product Requirements SPECIFY PRODUCT SPECIFY SOFTWARE Software Requirements TRACEABILITY & VALIDATION Test Plan & Test Results Test
More informationCS 4387/5387 SOFTWARE V&V LECTURE 4 BLACK-BOX TESTING
1 CS 4387/5387 SOFTWARE V&V LECTURE 4 BLACK-BOX TESTING Outline 2 Quiz Black-Box Testing Equivalence Class Testing (Equivalence Partitioning) Boundary value analysis Decision Table Testing 1 3 Quiz - 1
More information