ICSE MET 17 Keynote. Metamorphic Testing: Beyond Testing Numerical Computations
|
|
|
- Theresa Simmons
- 6 years ago
- Views:
Transcription
1 ICSE MET 17 Keynote Metamorphic Testing: Beyond Testing Numerical Computations Dr. Zhi Quan (George) Zhou Associate Professor University of Wollongong Australia 1
2 Why this topic? Observation: MT is often illustrated using mathematical functions, e.g. sin(x) = sin (x+360) sin(-x) = -sin(x) This is because mathematical functions have well known properties that everyone understands. Z.Q. (George) Zhou, University of Wollongong 2
3 Why this topic? Misunderstanding: Is MT just for numerical programs? Z.Q. (George) Zhou, University of Wollongong 3
4 Why this topic? The fact: Source: Segura et al., A survey on metamorphic testing, TSE 2016 Z.Q. (George) Zhou, University of Wollongong 4
5 Presentation outline MT in security testing A motivating example: the Heartbleed bug Detecting obfuscator bugs MT for online search MT beyond verification Z.Q. (George) Zhou, University of Wollongong 5
6 Presentation outline MT in security testing A motivating example: the Heartbleed bug Detecting obfuscator bugs MT for online search MT beyond verification Z.Q. (George) Zhou, University of Wollongong 6
7 MT in Security Testing A motivating example: the Heartbleed bug Under the Heartbeat protocol, the client sends a message and the message length to the server, and the server sends back the same message. Z.Q. (George) Zhou, University of Wollongong 7
8 MT in Security Testing A motivating example: the Heartbleed bug Server, if you are still there, send me banana (6 letters) Z.Q. (George) Zhou, University of Wollongong 8
9 MT in Security Testing A motivating example: the Heartbleed bug banana Z.Q. (George) Zhou, University of Wollongong 9
10 MT in Security Testing A motivating example: the Heartbleed bug The specification (RFC 6520) (TLS and DTLS Heartbeat Extension) Z.Q. (George) Zhou, University of Wollongong 10
11 MT in Security Testing A motivating example: the Heartbleed bug The specification (RFC 6520) (TLS and DTLS Heartbeat Extension) Z.Q. (George) Zhou, University of Wollongong 11
12 MT in Security Testing A motivating example: the Heartbleed bug The specification (RFC 6520) (TLS and DTLS Heartbeat Extension) Z.Q. (George) Zhou, University of Wollongong 12
13 MT in Security Testing A motivating example: the Heartbleed bug The specification (RFC 6520) (TLS and DTLS Heartbeat Extension) Z.Q. (George) Zhou, University of Wollongong 13
14 MT in Security Testing A motivating example: the Heartbleed bug The specification (RFC 6520) (TLS and DTLS Heartbeat Extension) Z.Q. (George) Zhou, University of Wollongong 14
15 MT in Security Testing A motivating example: the Heartbleed bug The specification (RFC 6520) (TLS and DTLS Heartbeat Extension) Z.Q. (George) Zhou, University of Wollongong 15
16 The bug: Z.Q. (George) Zhou, University of Wollongong 16
17 MT in Security Testing A motivating example: the Heartbleed bug Server, if you are still there, send me banana (600 letters) Z.Q. (George) Zhou, University of Wollongong 17
18 MT in Security Testing 600 characters in server buffer starting from b. A They motivating can be anything. example: the Heartbleed bug banana. User George set password to abcdefg. Server master key is set to Z.Q. (George) Zhou, University of Wollongong 18
19 MT in Security Testing A motivating example: the Heartbleed bug Difficulty in detecting the bug No crash No hang Undetectable with fuzz testing alone. Z.Q. (George) Zhou, University of Wollongong 19
20 MT in Security Testing A motivating example: the Heartbleed bug How MT can help? To identify MRs, the tester will ask: What if I change some of the parameter values? Z.Q. (George) Zhou, University of Wollongong 20
21 What if I change type to a different value? Z.Q. (George) Zhou, University of Wollongong 21
22 What if I change payload_length to a different value? This MR will generate failure test cases! Z.Q. (George) Zhou, University of Wollongong 22
23 What if I change two ore more parameters? MT can guide the tester to think beyond the normal range of input values (negative testing). Z.Q. (George) Zhou, University of Wollongong 23
24 MT in Security Testing Related work: MT in penetration test COMPSAC'17 Z.Q. (George) Zhou, University of Wollongong 24
25 Presentation outline MT in security testing A motivating example: the Heartbleed bug Detecting obfuscator bugs MT for online search MT beyond verification Z.Q. (George) Zhou, University of Wollongong 25
26 Detecting obfuscator bugs A pilot study Critical security infrastructure Transform an original program into an equivalent program less readable to prevent attackers from analyzing and understanding the original code. Z.Q. (George) Zhou, University of Wollongong 26
27 Detecting obfuscator bugs A pilot study As important as compilers But difficult to test A surprising observation: Compiler testing: a lot of research Obfuscator testing: almost nothing Our work: first to report real bugs. Z.Q. (George) Zhou, University of Wollongong 27
28 Detecting obfuscator bugs Obfuscators under test Cobfusc open source Linux utility that makes a C source file unreadable, but compilable. Z.Q. (George) Zhou, University of Wollongong 28
29 Detecting obfuscator bugs Obfuscators under test Z.Q. (George) Zhou, University of Wollongong 29
30 Detecting obfuscator bugs Obfuscators under test Stunnix CXX-Obfus commercial C/C++ obfuscator users include many Fortune 500 companies Z.Q. (George) Zhou, University of Wollongong 30
31 Detecting obfuscator bugs Obfuscators under test Z.Q. (George) Zhou, University of Wollongong 31
32 Detecting obfuscator bugs Obfuscators under test Tigress C obfuscator developed at the University of Arizona that supports novel defenses against both static and dynamic reverse engineering. Z.Q. (George) Zhou, University of Wollongong 32
33 Detecting obfuscator bugs Obfuscators under test Obfuscator-LLVM open source tool in LLVM compilation suite. Given a C source program, Obfuscator-LLVM outputs obfuscated and compiled binary code LLVM users include Adobe Systems, Apple, Intel, and Sony. Z.Q. (George) Zhou, University of Wollongong 33
34 Detecting obfuscator bugs Obfuscators under test Z.Q. (George) Zhou, University of Wollongong 34
35 Detecting obfuscator bugs The metamorphic relations (MRs) MR 1 Same outputs for the same inputs Q O(Q) C(O(Q)) P O(P) C(O(P)) obfuscator compiler Z.Q. (George) Zhou, University of Wollongong 35
36 Detecting obfuscator bugs The metamorphic relations (MRs) Q MR 1 How do we generate equivalent (P, Q)? MR 1.1 : by a separate tool MR 1.2 : by the obfuscator. C(O(Q)) P O(P) C(O(P)) obfuscator compiler Z.Q. (George) Zhou, University of Wollongong 36
37 Detecting obfuscator bugs The metamorphic relations (MRs) MR 2 For the same input program, the obfuscator should generate (behaviourally) equivalent programs regardless of when the obfuscator is run. P O(P) C(O(P)) obfuscator compiler P O(P) C(O(P)) obfuscator compiler Z.Q. (George) Zhou, University of Wollongong 37
38 Detecting obfuscator bugs The metamorphic relations (MRs) MR 3 Looks at obfuscated source code without compiling it. P O(P) C(O(P)) obfuscator compiler Checks whether obfuscation rules have been applied consistently each time the obfuscator runs. Z.Q. (George) Zhou, University of Wollongong 38
39 Detecting obfuscator bugs Tigress Z.Q. (George) Zhou, University of Wollongong 39
40 Detecting obfuscator bugs Tigress either true or false, i.e. 1 or 0. Z.Q. (George) Zhou, University of Wollongong 40
41 Detecting obfuscator bugs Tigress either or i.e., true. Z.Q. (George) Zhou, University of Wollongong 41
42 Detecting obfuscator bugs Tigress true else branch is unreachable true else branch is unreachable Z.Q. (George) Zhou, University of Wollongong 42
43 Any MT test case (i, j) will Detecting obfuscator bugs Tigress detect a failure because i-10 i+10. true i = i - 10 else branch is unreachable true i = i + 10 else branch is unreachable Z.Q. (George) Zhou, University of Wollongong 43
44 true i = i - 10 else branch is unreachable true Conventional testing without MT can detect the i = i + 10 else branch is unreachable failure only when i<=j (hence not guaranteed). Z.Q. (George) Zhou, University of Wollongong 44
45 In comparison, MT guarantees detection of the failure in this example, because O(P 1 ) and O(P 2 ) always output different values for i. true i = i - 10 else branch is unreachable true i = i + 10 else branch is unreachable Z.Q. (George) Zhou, University of Wollongong 45
46 Detecting obfuscator bugs Cobfusc MR 1.2 P O(P) C(O(P)) obfuscator compiler P: O(O(O(P))): Can t be compiled. Z.Q. (George) Zhou, University of Wollongong 46
47 Detecting obfuscator bugs Obfuscator-LLVM MR 2 : When an obfuscator runs at different times for the same program, the behaviour of the output programs should be equivalent.
48 Detecting obfuscator bugs Obfuscator-LLVM
49 Detecting obfuscator bugs Obfuscator-LLVM
50 Detecting obfuscator bugs Stunnix CXX-Obfus MR 3 : checking that obfuscated source files based on the same input source file are consistent. Z.Q. (George) Zhou, University of Wollongong 50
51 Detecting obfuscator bugs Stunnix CXX-Obfus Z.Q. (George) Zhou, University of Wollongong 51
52 Detecting obfuscator bugs Stunnix CXX-Obfus Z.Q. (George) Zhou, University of Wollongong 52
53 Relevant publication Metamorphic testing for cybersecurity, Computer, Z.Q. (George) Zhou, University of Wollongong 53
54 Presentation outline MT in security testing A motivating example: the Heartbleed bug Detecting obfuscator bugs MT for online search MT beyond verification Z.Q. (George) Zhou, University of Wollongong 54
55 MT for Online Search Related publications Automated functional testing of web search engines in the absence of an oracle, The University of Hong Kong, Tech. Rep. TR , 2007 Automated functional testing of online search services, Software Testing, Verification and Reliability, Metamorphic testing for software quality assessment: A study of search engines, IEEE Transactions on Software Engineering, 2016 Ongoing projects to report more interesting findings Z.Q. (George) Zhou, University of Wollongong 55
56 MT for Online Search Significance Post-industrial society / knowledge society / information society Key: to find the information. Z.Q. (George) Zhou, University of Wollongong 56
57 MT for Online Search Significance Online search / information retrieval The main activity to discover information cf. browsing Very difficult to test. Z.Q. (George) Zhou, University of Wollongong 57
58 MT for Online Search Applications Z.Q. (George) Zhou, University of Wollongong
59 MT for Online Search Applications Z.Q. (George) Zhou, University of Wollongong 59
60 MT for Online Search Applications Z.Q. (George) Zhou, University of Wollongong 60
61 MT for Online Search Applications Z.Q. (George) Zhou, University of Wollongong 61
62 MT for Online Search Applications Z.Q. (George) Zhou, University of Wollongong 62
63 MT for Online Search Applications
64 MT for Online Search Applications Location based service / local search Source of image:
65 MT for Online Search Traditional evaluation methods Precision Recall C / B expensive C / A impossible Difficulties: (1) the sheer volume of online data (2) how to decide relevance? (3) how to assess ranking quality? C B A 65
66 MT for Online Search How to identify MRs? A top-down approach Z.Q. (George) Zhou, University of Wollongong 66
67 MT for Online Search MR examples First identify some General MRs, e.g.: If A implies B then * results for A results for B, and * results for A results for B. A general MR can generate many concrete MRs. Z.Q. (George) Zhou, University of Wollongong 67
68 a general MR: If A implies B then results for A results for B. a concrete MR: filtering by location Example: Z.Q. (George) Zhou, University of Wollongong 68
69 a general MR: If A implies B then results for A results for B. a concrete MR: filtering by location failure Z.Q. (George) Zhou, University of Wollongong 69
70 a general MR: If A implies B then results for A results for B. concrete MRs: using logical operations such as AND, OR, EXCLUDE: results for (A 1 AND A 2 ) results for A 1 results for (A 1 - A 2 ) results for A 1 results for (A 1 ) results for (A 1 OR A 2 ) Z.Q. (George) Zhou, University of Wollongong 70
71 a general MR: If A implies B then results for A results for B. concrete MRs: using logical operators such as AND, OR, EXCLUDE: results for (A 1 AND A 2 ) results for A 1 results for (A 1 - A 2 ) results for A 1 results for (A 1 ) results for (A 1 OR A 2 ) A 1, A 2 can be any search criteria, e.g. query terms and filters Z.Q. (George) Zhou, University of Wollongong 71
72 a general MR: If A implies B then results for A results for B. concrete MRs: using logical operators such as AND, OR, EXCLUDE: results for (A 1 AND A 2 ) results for A 1 results for (A 1 - A 2 ) results for A 1 results for (A 1 ) results for (A 1 OR A 2 ) A 1, A 2 can be any search criteria, e.g. query terms and filters Z.Q. (George) Zhou, University of Wollongong 72
73 a general MR: If A implies B then results for A results for B. concrete MRs: using logical operators such as AND, OR, EXCLUDE: results for (A 1 AND A 2 ) results for A 1 results for (A 1 - A 2 ) results for A 1 results for (A 1 ) results for (A 1 OR A 2 ) A 1, A 2 can be any search criteria, e.g. query terms and filters Z.Q. (George) Zhou, University of Wollongong 73
74 concrete MRs: using logical operators such as AND, OR, EXCLUDE: results for (A 1 AND A 2 ) results for A 1 results for (A 1 - A 2 ) results for A 1 results for (A 1 ) results for (A 1 OR A 2 ) A 1, A 2 can be any search criteria, e.g. query terms and filters. Z.Q. (George) Zhou, University of Wollongong 74
75 concrete MRs: using logical operators such as AND, OR, EXCLUDE: results for (A 1 AND A 2 ) results for A 1 results for (A 1 - A 2 ) results for A 1 results for (A 1 ) results for (A 1 OR A 2 ) A 1, A 2 can be any search criteria, e.g. query terms and filters. Z.Q. (George) Zhou, University of Wollongong 75
76 MT for Online Search Z.Q. (George) Zhou, University of Wollongong 76
77 MT for Online Search Validity must be carefully considered Because online search may only return approximate results. We cannot use simple mathematics such as: If A=1, B=2 then A+B=3. Z.Q. (George) Zhou, University of Wollongong 77
78 MT for Online Search Validity must be carefully considered Dynamic nature of online data? All tests must be repeatable. Z.Q. (George) Zhou, University of Wollongong 78
79 MT for Online Search Validity must be carefully considered Relationship between users information needs and logical consistency (MRs)? Users perceived quality of ranking decreases if they find strong inconsistencies in search results. Z.Q. (George) Zhou, University of Wollongong 79
80 MT for Online Search Ranking quality Most users are more concerned about the first few items. Z.Q. (George) Zhou, University of Wollongong 80
81 MT for Online MR: An Search example of Web search engines Ranking quality filter Z.Q. (George) Zhou, University of Wollongong 81
82 MT for Online Search Ranking quality Studied different file types HTML, PDF, TXT Sensitive / vulnerable to hyperlink structure / topology of Web components Needs improvement for less structured files How about multimedia search? Further research topic Any bias in search results? Z.Q. (George) Zhou, University of Wollongong 82
83 Useful for detecting security vulnerabilities Undetectable with traditional approaches.
84 Presentation outline MT in security testing A motivating example: the Heartbleed bug Detecting obfuscator bugs MT for online search MT beyond verification Z.Q. (George) Zhou, University of Wollongong 84
85 MT beyond verification Software testing can be performed for different purposes, with verification and validation being the core tasks. Testing is also used for other types of quality assessment beyond V&V, such as usability and scalability. Z.Q. (George) Zhou, University of Wollongong 85
86 MT beyond verification Meeting the challenges lack of specifications lack of oracles when verifying, validating, and assessing large and complex software systems. User-oriented approach Z.Q. (George) Zhou, University of Wollongong 86
87 Figure adapted from: Mauro Pezzè and Michal Young, Software Testing and Analysis: Process, Principles and Techniques
88 x x x x x x x Note: User manuals are not specifications --- an adequate guide to building a product that will fulfill its goals.
89 Difficulties in user validation A large body of software systems come without specifications, or with incomplete specifications, e.g.: online systems open source development poor software evolution web services Z.Q. (George) Zhou, University of Wollongong 89
90 Difficulties in user validation The lack of knowledge about the software design and specifications, coupled with the oracle problem, makes user validation extremely difficult. It is hard for users to decide if, and to what degree, the systems are appropriate for their needs. Z.Q. (George) Zhou, University of Wollongong 90
91 Difficulties in user validation The lack of knowledge about the software design and specifications, coupled with the oracle problem, makes user validation extremely difficult. It is hard Solution: for users define to decide MRs if, and from to the what degree, the user s systems perspective. are appropriate for their needs. Beyond traditional verification and fault detection. Z.Q. (George) Zhou, University of Wollongong 91
92 MT beyond verification From verification to validation
93 MT beyond verification From verification to validation Z.Q. (George) Zhou, University of Wollongong 93
94 MT beyond verification From V&V to quality assessment A crucial deficiency in Bing s functional completeness, which refers to the degree to which the set of functions covers all the specified tasks and user objectives. Z.Q. (George) Zhou, University of Wollongong 94
95 MT beyond verification From V&V to quality assessment Z.Q. (George) Zhou, University of Wollongong 95
96 MT beyond verification From V&V to quality assessment Functional completeness Z.Q. (George) Zhou, University of Wollongong 96
97 MT beyond verification From V&V to quality assessment Functional correctness Z.Q. (George) Zhou, University of Wollongong 97
98 Functional correctness: more examples Search for a paper in the ACM digital library: In black and white: an integrated approach to class-level testing of object-oriented programs Z.Q. (George) Zhou, University of Wollongong 98
99 Functional correctness: more examples Z.Q. (George) Zhou, University of Wollongong 99
100 Functional correctness: more examples Oracle on the fly: Shoot first and then point to the target. filter Z.Q. (George) Zhou, University of Wollongong 100
101 Functional correctness: more examples Z.Q. (George) Zhou, University of Wollongong 101
102 Z.Q. (George) Zhou, University of Wollongong 102
103 MT beyond verification Measuring software performance Performance efficiency: MT revealed scalability problems of the SUTs. Z.Q. (George) Zhou, University of Wollongong 103
104 MT beyond verification Measuring software performance: scalability steeper performance degradation when searching large domains. Z.Q. (George) Zhou, University of Wollongong 104
105 MT beyond verification Measuring software performance: scalability performance degradation when searching large domains. Z.Q. (George) Zhou, University of Wollongong 105
106 MT beyond verification Reliability: degree to which a system performs specified functions under specified conditions for a specified period of time. Measuring reliability We performed MT under different operational profiles and quantified the test results on an hourly basis. Z.Q. (George) Zhou, University of Wollongong 106
107 MT beyond verification Measuring reliability Z.Q. (George) Zhou, University of Wollongong 107
108 MT beyond verification Measuring reliability Z.Q. (George) Zhou, University of Wollongong 108
109 MT beyond verification Measuring reliability Z.Q. (George) Zhou, University of Wollongong 109
110 MT beyond verification Measuring reliability Z.Q. (George) Zhou, University of Wollongong 110
111 MT beyond verification Measuring reliability Z.Q. (George) Zhou, University of Wollongong 111
112 MT beyond verification Measuring usability Operability: easy to operate and control. A lot of search operators were tested in MT. Z.Q. (George) Zhou, University of Wollongong 112
113 MT beyond verification Measuring usability User error protection: e.g. even if users didn't issue their queries by following the optimal order. Z.Q. (George) Zhou, University of Wollongong 113
114 MT beyond verification Quality in Use Model Quality in Use Model: characterizes the impact that the product has on its stakeholders. Effectiveness: accuracy and completeness with which users achieve specified goals. Z.Q. (George) Zhou, University of Wollongong 114
115 MT beyond verification Quality in Use Model Context completeness: A system "can be used... in all the specified contexts of use." We tested with different operational profiles e.g. different languages, query types, and domains to search. Z.Q. (George) Zhou, University of Wollongong 115
116 Ongoing Research Location-based search GPS navigation software ecommerce websites Z.Q. (George) Zhou, University of Wollongong 116
117 Questions are welcome Z.Q. (George) Zhou, University of Wollongong 117
Bridge Course On Software Testing
G. PULLAIAH COLLEGE OF ENGINEERING AND TECHNOLOGY Accredited by NAAC with A Grade of UGC, Approved by AICTE, New Delhi Permanently Affiliated to JNTUA, Ananthapuramu (Recognized by UGC under 2(f) and 12(B)
No Source Code. EEC 521: Software Engineering. Specification-Based Testing. Advantages
No Source Code : Software Testing Black-Box Testing Test-Driven Development No access to source code So test cases don t worry about structure Emphasis is only on ensuring that the contract is met Specification-Based
Case studies of metamorphic tesing
University of Wollongong Research Online University of Wollongong Thesis Collection 2017+ University of Wollongong Thesis Collections 2017 Case studies of metamorphic tesing Wenjuan Ma University of Wollongong
Black Box Testing. EEC 521: Software Engineering. Specification-Based Testing. No Source Code. Software Testing
Black Box Testing EEC 521: Software Engineering Software Testing Black-Box Testing Test-Driven Development Also known as specification-based testing Tester has access only to running code and the specification
Verification and Validation. Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 22 Slide 1
Verification and Validation Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 22 Slide 1 Verification vs validation Verification: "Are we building the product right?. The software should
Part 5. Verification and Validation
Software Engineering Part 5. Verification and Validation - Verification and Validation - Software Testing Ver. 1.7 This lecture note is based on materials from Ian Sommerville 2006. Anyone can use this
Heartbleed Bug. Anthony Postiglione. Department of Electrical & Computer Engineering Missouri University of Science and Technology
Heartbleed Bug Anthony Postiglione Department of Electrical & Computer Engineering Missouri University of Science and Technology avp275@mst.edu rev. 14.0 Introduction What is Heartbleed? Discovery Presentation
Digitized Engineering Notebook
Governors State University OPUS Open Portal to University Scholarship All Capstone Projects Student Capstone Projects Spring 2017 Digitized Engineering Notebook Sarath Garimella Governors State University
International Journal of Computer Engineering and Applications, Volume XII, Special Issue, April- ICITDA 18,
International Journal of Computer Engineering and Applications, Volume XII, Special Issue, April- ICITDA 18, www.ijcea.com ISSN 2321-3469 SOFTWARE TESTING Rajat Galav, Shivank Lavania Student, Department
International Journal of Computer Engineering and Applications, Volume XII, Special Issue, September 18, ISSN SOFTWARE TESTING
International Journal of Computer Engineering and Applications, Volume XII, Special Issue, September 18, www.ijcea.com ISSN 2321-3469 SOFTWARE TESTING Rajat Galav 1, Shivank Lavania 2, Brijesh Kumar Singh
Testing. Prof. Clarkson Fall Today s music: Wrecking Ball by Miley Cyrus
Testing Prof. Clarkson Fall 2017 Today s music: Wrecking Ball by Miley Cyrus Review Previously in 3110: Modules Specification (functions, modules) Today: Validation Testing Black box Glass box Randomized
Three General Principles of QA. COMP 4004 Fall Notes Adapted from Dr. A. Williams
Three General Principles of QA COMP 4004 Fall 2008 Notes Adapted from Dr. A. Williams Software Quality Assurance Lec2 1 Three General Principles of QA Know what you are doing. Know what you should be doing.
Objectives. Chapter 19. Verification vs. validation. Topics covered. Static and dynamic verification. The V&V process
Objectives Chapter 19 Verification and Validation Assuring that a software system meets a user s need are to introduce software verification and validation (V&V) and to discuss the distinction between
Case Studies on the Selection of Useful Relations in Metamorphic Testing
Case Studies on the Selection of Useful Relations in Metamorphic Testing T. Y. Chen 1, D. H. Huang 1, T. H. Tse 2, and Zhi Quan Zhou 1 1 School of Information Technology, Swinburne University of Technology,
Overview AEG Conclusion CS 6V Automatic Exploit Generation (AEG) Matthew Stephen. Department of Computer Science University of Texas at Dallas
CS 6V81.005 Automatic Exploit Generation (AEG) Matthew Stephen Department of Computer Science University of Texas at Dallas February 20 th, 2012 Outline 1 Overview Introduction Considerations 2 AEG Challenges
XVIII. Software Testing. Laurea Triennale in Informatica Corso di Ingegneria del Software I A.A. 2006/2007 Andrea Polini
XVIII. Software Testing Laurea Triennale in Informatica Corso di Objective General discussion on Testing Testing Phases Approaches to testing Structural testing Functional testing Testing non functional
Verification and Validation. Ian Sommerville 2004 Software Engineering, 7th edition. Chapter 22 Slide 1
Verification and Validation 1 Objectives To introduce software verification and validation and to discuss the distinction between them To describe the program inspection process and its role in V & V To
Verification and Validation
Verification and Validation Assuring that a software system meets a user's needs Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 19 Slide 1 Objectives To introduce software verification
Test Oracles and Mutation Testing. CSCE Lecture 23-11/18/2015
Test Oracles and Mutation Testing CSCE 740 - Lecture 23-11/18/2015 Software Testing - Back to the Basics Tests are sequences of stimuli and observations. We care about input and output. (I 1 O 1 ) (I 2
Software Engineering
Software Engineering Lecture 13: Testing and Debugging Testing Peter Thiemann University of Freiburg, Germany SS 2014 Recap Recap Testing detect the presence of bugs by observing failures Recap Testing
The Bizarre Truth! Automating the Automation. Complicated & Confusing taxonomy of Model Based Testing approach A CONFORMIQ WHITEPAPER
The Bizarre Truth! Complicated & Confusing taxonomy of Model Based Testing approach A CONFORMIQ WHITEPAPER By Kimmo Nupponen 1 TABLE OF CONTENTS 1. The context Introduction 2. The approach Know the difference
Structural Testing. (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 1
Structural Testing (c) 2007 Mauro Pezzè & Michal Young Ch 12, slide 1 Learning objectives Understand rationale for structural testing How structural (code-based or glass-box) testing complements functional
Software Quality. Chapter What is Quality?
Chapter 1 Software Quality 1.1 What is Quality? The purpose of software quality analysis, or software quality engineering, is to produce acceptable products at acceptable cost, where cost includes calendar
Taking White Hats to the Laundry: How to Strengthen Testing in Common Criteria
Taking White Hats to the Laundry: How to Strengthen Testing in Common Criteria Apostol Vassilev, Principal Consultant September 23,2009. Product Testing in Common Criteria Product Testing in Common Criteria
10. Software Testing Fundamental Concepts
10. Software Testing Fundamental Concepts Department of Computer Science and Engineering Hanyang University ERICA Campus 1 st Semester 2016 Testing in Object-Oriented Point of View Error Correction Cost
Security Testing. John Slankas
Security Testing John Slankas jbslanka@ncsu.edu Course Slides adapted from OWASP Testing Guide v4 CSC 515 Software Security What is Security Testing? Validate security controls operate as expected What
Software Testing Strategies. Slides copyright 1996, 2001, 2005, 2009, 2014 by Roger S. Pressman. For non-profit educational use only
Chapter 22 Software Testing Strategies Slide Set to accompany Software Engineering: A Practitioner s Approach, 8/e by Roger S. Pressman and Bruce R. Maxim Slides copyright 1996, 2001, 2005, 2009, 2014
Ian Sommerville 2006 Software Engineering, 8th edition. Chapter 22 Slide 1
Verification and Validation Slide 1 Objectives To introduce software verification and validation and to discuss the distinction between them To describe the program inspection process and its role in V
Lecture 15 Software Testing
Lecture 15 Software Testing Includes slides from the companion website for Sommerville, Software Engineering, 10/e. Pearson Higher Education, 2016. All rights reserved. Used with permission. Topics covered
Greats Bugs in History
Semidoctus, 23 November 2016 Semidoctus, 23 November 2016 1 / 1/ Plan 1 Introduction: what s a bug? 2 The Y2K Bug 3 The case of Ariane 5 4 Heartbleed 5 The Intel Division Bug 6 500-mile emails 7 Conclusion
Topics in Software Testing
Dependable Software Systems Topics in Software Testing Material drawn from [Beizer, Sommerville] Software Testing Software testing is a critical element of software quality assurance and represents the
It is primarily checking of the code and/or manually reviewing the code or document to find errors This type of testing can be used by the developer
Static testing Static testing is a software testing method that involves examination of the program's code and its associated documentation but does not require the program be executed. Dynamic testing,
How to Break Software by James Whittaker
How to Break Software by James Whittaker CS 470 Practical Guide to Testing Consider the system as a whole and their interactions File System, Operating System API Application Under Test UI Human invokes
Web Security Vulnerabilities: Challenges and Solutions
Web Security Vulnerabilities: Challenges and Solutions A Tutorial Proposal for ACM SAC 2018 by Dr. Hossain Shahriar Department of Information Technology Kennesaw State University Kennesaw, GA 30144, USA
MSc Software Testing MSc Prófun hugbúnaðar
MSc Software Testing MSc Prófun hugbúnaðar Fyrirlestrar 7 & 8 Structural Testing White-box tests. 29/8/27 Dr Andy Brooks 1 Case Study Dæmisaga Reference Structural Testing of Programs, A Survey, A A Omar
Verification and Validation. Assuring that a software system meets a user s needs. Verification vs Validation. The V & V Process
Verification and Validation Assuring that a software system meets a user s needs Ian Sommerville 1995/2000 (Modified by Spiros Mancoridis 1999) Software Engineering, 6th edition. Chapters 19,20 Slide 1
DART: Directed Automated Random Testing
DART: Directed Automated Random Testing Patrice Godefroid Nils Klarlund Koushik Sen Bell Labs Bell Labs UIUC Presented by Wei Fang January 22, 2015 PLDI 2005 Page 1 June 2005 Motivation Software testing:
Administrivia. ECE/CS 5780/6780: Embedded System Design. Acknowledgements. What is verification?
Administrivia ECE/CS 5780/6780: Embedded System Design Scott R. Little Lab 8 status report. Set SCIBD = 52; (The Mclk rate is 16 MHz.) Lecture 18: Introduction to Hardware Verification Scott R. Little
Software Testing Interview Question and Answer
Software Testing Interview Question and Answer What is Software Testing? A process of analyzing a software item to detect the differences between existing and required conditions (i.e., defects) and to
Ontology- and Bayesian- based Information Security Risk Management
Ontology- and Bayesian- based Information Security Risk Management Stefan Fenz sfenz@securit 4 th ETSI Security Workshop 13 14 January 2009 ETSI, Sophia Antipolis, France Motivation Almost every business
CYSE 411/AIT 681 Secure Software Engineering. Topic #6. Seven Software Security Touchpoints (III) Instructor: Dr. Kun Sun
CYSE 411/AIT 681 Secure Software Engineering Topic #6. Seven Software Security Touchpoints (III) Instructor: Dr. Kun Sun Reading This lecture [McGraw]: Ch. 7-9 2 Seven Touchpoints 1. Code review 2. Architectural
Software Engineering Testing and Debugging Testing
Software Engineering Testing and Debugging Testing Prof. Dr. Peter Thiemann Universitt Freiburg 08.06.2011 Recap Testing detect the presence of bugs by observing failures Debugging find the bug causing
Agent-based Modeling using L S D
Agent-based Modeling using L S D Marco VALENTE 1,2,3 1 LEM, S. Anna School of Advanced Studies, Pisa 2 University of L Aquila 3 University of Sussex, SPRU ABM: a definition Agent-based models are based
4. Risk-Based Security Testing. Reading. CYSE 411/AIT 681 Secure Software Engineering. Seven Touchpoints. Application of Touchpoints
Reading This lecture [McGraw]: Ch. 7-9 CYSE 411/AIT 681 Secure Software Engineering Topic #6. Seven Software Security Touchpoints (III) Instructor: Dr. Kun Sun 2 Seven Touchpoints Application of Touchpoints
Metamorphic Testing and Beyond
Postprint of article in Proceedings of the International Workshop on Software Technology and Engineering Practice (STEP 03), IEEE Computer Society, Los Alamitos, CA, pp. pp. 94 100 (2004) Metamorphic Testing
Types of Software Testing: Different Testing Types with Details
Types of Software Testing: Different Testing Types with Details What are the different Types of Software Testing? We, as testers are aware of the various types of Software Testing such as Functional Testing,
INTRODUCTION TO CLOAKWARE/TRS TECHNOLOGY
INTRODUCTION TO CLOAKWARE/TRS TECHNOLOGY VERSION 2.2 OCTOBER 2001 SUMMARY Software is easy to tamper with and reverse engineer so unprotected software deployed on malicious hosts can t be trusted by corporations
Manuel Oriol, CHCRC-C, Software Testing ABB
Manuel Oriol, CHCRC-C, 08.11.2017 Software Testing Slide 1 About me 1998 2004 2005 2008 2011 Slide 2 Introduction Why do we test? Did you have to deal with testing in the past? Slide 3 Ariane 5 http://www.youtube.com/watch?v=kyurqduyepi
Outline. software testing: search bugs black-box and white-box testing static and dynamic testing
Outline 1 Verification Techniques software testing: search bugs black-box and white-box testing static and dynamic testing 2 Programming by Contract assert statements in Python using preconditions and
An Introduction to Systematic Software Testing. Robert France CSU
An Introduction to Systematic Software Testing Robert France CSU Why do we need to systematically test software? Poor quality products can Inconvenience direct and indirect users Result in severe financial
A Template-Based Approach to Describing Metamorphic Relations
A -Based Approach to Describing Metamorphic Relations Sergio Segura, Amador Durán, Javier Troya, Antonio Ruiz-Cortés Applied Software Engineering Group University of Seville Buenos Aires, MET 2017 Conclusions
Static Analysis and Bugfinding
Static Analysis and Bugfinding Alex Kantchelian 09/12/2011 Last week we talked about runtime checking methods: tools for detecting vulnerabilities being exploited in deployment. So far, these tools have
Honours/Master/PhD Thesis Projects Supervised by Dr. Yulei Sui
Honours/Master/PhD Thesis Projects Supervised by Dr. Yulei Sui Projects 1 Information flow analysis for mobile applications 2 2 Machine-learning-guide typestate analysis for UAF vulnerabilities 3 3 Preventing
Chapter 8 Software Testing. Chapter 8 Software testing
Chapter 8 Software Testing 1 Topics covered Introduction to testing Stages for testing software system are: Development testing Release testing User testing Test-driven development as interleave approach.
Chap 2. Introduction to Software Testing
Chap 2. Introduction to Software Testing 2.1 Software Testing Concepts and Processes 2.2 Test Management 1 2.1 Software Testing Concepts and Processes 1. Introduction 2. Testing Dimensions 3. Test Concepts
MSc Software Testing and Maintenance MSc Prófun og viðhald hugbúnaðar
MSc Software Testing and Maintenance MSc Prófun og viðhald hugbúnaðar Fyrirlestrar 31 & 32 Structural Testing White-box tests. 27/1/25 Dr Andy Brooks 1 Case Study Dæmisaga Reference Structural Testing
Verification Overview Testing Theory and Principles Testing in Practice. Verification. Miaoqing Huang University of Arkansas 1 / 80
1 / 80 Verification Miaoqing Huang University of Arkansas Outline 1 Verification Overview 2 Testing Theory and Principles Theoretical Foundations of Testing Empirical Testing Principles 3 Testing in Practice
Automatizing vulnerability research
Innova&on & Research Symposium Cisco and Ecole Polytechnique 8-9 April 2018 CEDRIC TESSIER INSTRUMENTATION TEAM LEADER / ctessier@quarkslab.com Automatizing vulnerability research to better face new software
n Explain penetration testing concepts n Explain vulnerability scanning concepts n Reconnaissance is the first step of performing a pen test
Chapter Objectives n Explain penetration testing concepts n Explain vulnerability scanning concepts Chapter #4: Threats, Attacks, and Vulnerabilities Vulnerability Scanning and Penetration Testing 2 Penetration
MONIKA HEINER.
LESSON 1 testing, intro 1 / 25 SOFTWARE TESTING - STATE OF THE ART, METHODS, AND LIMITATIONS MONIKA HEINER monika.heiner@b-tu.de http://www.informatik.tu-cottbus.de PRELIMINARIES testing, intro 2 / 25
Low level security. Andrew Ruef
Low level security Andrew Ruef What s going on Stuff is getting hacked all the time We re writing tons of software Often with little regard to reliability let alone security The regulatory environment
CS/ECE 5780/6780: Embedded System Design
CS/ECE 5780/6780: Embedded System Design John Regehr Lecture 18: Introduction to Verification What is verification? Verification: A process that determines if the design conforms to the specification.
Requirements Validation and Negotiation
REQUIREMENTS ENGINEERING LECTURE 2017/2018 Joerg Doerr Requirements Validation and Negotiation AGENDA Fundamentals of Requirements Validation Fundamentals of Requirements Negotiation Quality Aspects of
L04. Model-based Testing: Principles Henry Muccini. Dipartimento di Informatica, Universityof L Aquila
Università degli Studi dell Aquila L04. Model-based Testing: Principles Henry Muccini Dipartimento di Informatica, Universityof L Aquila henry.muccini@univaq.it Copyright Notice The material in these slides
USER-CENTERED DESIGN KRANACK / DESIGN 4
USER-CENTERED DESIGN WHAT IS USER-CENTERED DESIGN? User-centered design (UCD) is an approach to design that grounds the process in information about the people who will use the product. UCD processes focus
Database Security MET CS 674 On-Campus/Blended
Database Security MET CS 674 On-Campus/Blended George Ultrino gultrino@bu.edu Office hours: by appointment Course Description The course provides a strong foundation in database security and auditing.
Software testing. Ian Sommerville 2006 Software Engineering, 8th edition. Chapter 23 Slide 1
Software testing Ian Sommerville 2006 Software Engineering, 8th edition. Chapter 23 Slide 1 Objectives To discuss the distinctions between validation testing and defect testing To describe the principles
Overview. State-of-the-Art. Relative cost of error correction. CS 619 Introduction to OO Design and Development. Testing.
Overview CS 619 Introduction to OO Design and Development ing! Preliminaries! All sorts of test techniques! Comparison of test techniques! Software reliability Fall 2012! Main issues: There are a great
Lecture 17: Testing Strategies. Developer Testing
Lecture 17: Testing Strategies Structural Coverage Strategies (White box testing): Statement Coverage Branch Coverage Condition Coverage Data Path Coverage Function Coverage Strategies (Black box testing):
Why testing and analysis. Software Testing. A framework for software testing. Outline. Software Qualities. Dependability Properties
Why testing and analysis Software Testing Adapted from FSE 98 Tutorial by Michal Young and Mauro Pezze Software is never correct no matter what developing testing technique is used All software must be
Introduction to Dynamic Analysis
Introduction to Dynamic Analysis Reading assignment Gary T. Leavens, Yoonsik Cheon, "Design by Contract with JML," draft paper, http://www.eecs.ucf.edu/~leavens/jml//jmldbc.pdf G. Kudrjavets, N. Nagappan,
Predictive malware response testing methodology. Contents. 1.0 Introduction. Methodology version 1.0; Created 17/01/2018
Predictive malware response testing methodology Methodology version 1.0; Created 17/01/2018 Contents Contents... 1 1.0 Introduction... 1 2.0 Test framework... 2 3.0 Threat selection and management... 3
Analysis/Bug-finding/Verification for Security
Analysis/Bug-finding/Verification for Security VIJAY GANESH University of Waterloo Winter 2013 Analysis/Test/Verify for Security Instrument code for testing Heap memory: Purify Perl tainting (information
Testing & Symbolic Execution
Testing & Symbolic Execution Software Testing The most common way of measuring & ensuring correctness Input 2 Software Testing The most common way of measuring & ensuring correctness Input Observed Behavior
Tonight s Agenda. CSC340: Requirements Engineering. Course Objectives. Requirements Engineering. Software Engineering. What is Software Engineering?
Tonight s Agenda CSC340: Engineering Jennifer Campbell Lecturer Part 1 Introduction to course content Course information Changes to the SE courses/program Part 2 What are requirements? CSC340 University
Static Analysis of C++ Projects with CodeSonar
Static Analysis of C++ Projects with CodeSonar John Plaice, Senior Scientist, GrammaTech jplaice@grammatech.com 25 July 2017, Meetup C++ de Montréal Abstract Static program analysis consists of the analysis
COMP 3500 Introduction to Operating Systems Project 5 Virtual Memory Manager
COMP 3500 Introduction to Operating Systems Project 5 Virtual Memory Manager Points Possible: 100 Submission via Canvas No collaboration among groups. Students in one group should NOT share any project
Effizientere IT-Sicherheitstests mit Hilfe von Usage-based Testing
Effizientere IT-Sicherheitstests mit Hilfe von Usage-based Testing GI TAV 37 5. Februar 2015 Martin Schneider Fraunhofer FOKUS Steffen Herbold Universität Göttingen Outline Challenge: Efficiency of Security
Software Quality Assurance. David Janzen
Software Quality Assurance David Janzen What is quality? Crosby: Conformance to requirements Issues: who establishes requirements? implicit requirements Juran: Fitness for intended use Issues: Who defines
ThousandEyes for. Application Delivery White Paper
ThousandEyes for Application Delivery White Paper White Paper Summary The rise of mobile applications, the shift from on-premises to Software-as-a-Service (SaaS), and the reliance on third-party services
CCNA Cybersecurity Operations 1.1 Scope and Sequence
CCNA Cybersecurity Operations 1.1 Scope and Sequence Last updated June 18, 2018 Introduction Today's organizations are challenged with rapidly detecting cybersecurity breaches and effectively responding
[IT6004-SOFTWARE TESTING] UNIT 2
![[IT6004-SOFTWARE TESTING] UNIT 2](/thumbs/90/102012046.jpg "[IT6004-SOFTWARE TESTING] UNIT 2")
1. List the two basic Testing strategies. UNIT 2 Black box testing. White box testing. 2. What are the knowledge sources for Black box testing? Requirements Document specification Domain knowledge Defect
Improving software testing cost-effectiveness through dynamic partitioning
University of Wollongong Research Online Faculty of Informatics - Papers (Archive) Faculty of Engineering and Information Sciences 2009 Improving software testing cost-effectiveness through dynamic partitioning
1 Visible deviation from the specification or expected behavior for end-user is called: a) an error b) a fault c) a failure d) a defect e) a mistake
Sample ISTQB examination 1 Visible deviation from the specification or expected behavior for end-user is called: a) an error b) a fault c) a failure d) a defect e) a mistake 2 Regression testing should
Advanced Security Tester Course Outline
Advanced Security Tester Course Outline General Description This course provides test engineers with advanced skills in security test analysis, design, and execution. In a hands-on, interactive fashion,
Bug Finding with Under-approximating Static Analyses. Daniel Kroening, Matt Lewis, Georg Weissenbacher
Bug Finding with Under-approximating Static Analyses Daniel Kroening, Matt Lewis, Georg Weissenbacher Overview Over- vs. underapproximating static analysis Path-based symbolic simulation Path merging Acceleration
CCNA Cybersecurity Operations. Program Overview
Table of Contents 1. Introduction 2. Target Audience 3. Prerequisites 4. Target Certification 5. Curriculum Description 6. Curriculum Objectives 7. Virtual Machine Requirements 8. Course Outline 9. System
In this Lecture you will Learn: Testing in Software Development Process. What is Software Testing. Static Testing vs.
In this Lecture you will Learn: Testing in Software Development Process Examine the verification and validation activities in software development process stage by stage Introduce some basic concepts of
Publications Database
Getting Started Guide Publications Database To w a r d s a S u s t a i n a b l e A s i a - P a c i f i c!1 Table of Contents Introduction 3 Conventions 3 Getting Started 4 Suggesting a Topic 11 Appendix
Metamorphic Testing and Its Applications
To appear in Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004), Software Engineers Association, Japan (2004) Metamorphic Testing and Its Applications Zhi Quan Zhou,
Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement
Tian: Software Quality Engineering Slide (Ch.12) 1 Software Quality Engineering: Testing, Quality Assurance, and Quantifiable Improvement Jeff Tian, tian@engr.smu.edu www.engr.smu.edu/ tian/sqebook Chapter
Test Plan. KSU Student Portal. Version 2.0. Submitted in partial fulfillment of the requirements of the degree of MSE
Test Plan KSU Student Portal Version 2.0 Submitted in partial fulfillment of the requirements of the degree of MSE Javier Ramos Rodríguez CIS 895 MSE Project Kansas State University Table of Contents 1.
CIS 5373 Systems Security
CIS 5373 Systems Security Topic 1: Introduction to Systems Security Endadul Hoque 1 Why should you care? Security impacts our day-to-day life Become a security-aware user Make safe decisions Become a security-aware
Subject Software Testing Structural Testing
Subject Software Testing Structural Testing Objective: 1. Understand Concept of structural testing 2. How structural (code-based or glass-box) testing complements functional (black-box) testing 3. Recognize
Software change. Software maintenance
Software change 1 Software change is inevitable New requirements emerge when the software is used The business environment changes Errors must be repaired New equipment must be accommodated The performance
Sample Exam Syllabus
ISTQB Foundation Level 2011 Syllabus Version 2.9 Release Date: December 16th, 2017. Version.2.9 Page 1 of 46 Dec 16th, 2017 Copyright 2017 (hereinafter called ISTQB ). All rights reserved. The authors
Web Application Testing in Fifteen Years of WSE
Web Application Testing in Fifteen Years of WSE Anna Rita Fasolino Domenico Amalfitano Porfirio Tramontana Dipartimento di Ingegneria Elettrica e Tecnologie dell Informazione University of Naples Federico
Semi-Proving: An Integrated Method for Program Proving, Testing, and Debugging
Postprint of acticle in IEEE Transactions on Software Engineering 37 (1): 109 125 (2011) Semi-Proving: An Integrated Method for Program Proving, Testing, and Debugging Tsong Yueh Chen, Member, IEEE, T.H.
Testing Tools. Software Testing and Verification. Lecture 14. Stephen M. Thebaut, Ph.D. Prepared by. University of Florida
Testing Tools Software Testing and Verification Lecture 14 Prepared by Stephen M. Thebaut, Ph.D. University of Florida Overview of Topics Why is test automation desirable? Types of test automation test