Software Testing CS 408. Lecture 11: Review 2/20/18

Transcription

1 Software Testing CS 408 Lecture 11: Review 2/20/18

2 Lecture 1: Basics 2

3 Two Views Verification: Prove the absence, and conjecture the presence, of bugs Ex: types: Not all ill-typed programs are wrong But, no erroneous program should be well-typed Testing: Prove the presence, and the conjecture the absence, of bugs Ex: random testing Every failed test is a true failure But, not finding a failure doesn t mean the program is correct Verification is desirable, but sometimes infeasible Testing is always feasible, but sometimes insufficient 3

4 Basic definitions A failure is an unacceptable behavior exhibited by a system The frequency of failures measures the reliability of the system An important design objective is to achieve low failure rates and hence high reliability. A defect is a flaw in any aspect of the system that contributes, or may potentially contribute, to the occurrence of one or more failures could be in the requirements, the design, and/or the code It might take several defects to cause a particular failure An error is an unintentional decision by a software developer that leads to the introduction of a defect 4

5 Black-box testing Testers provide the system with inputs and observe the outputs They can see none of: The source code The internal data Any of the design documentation describing the system s internals Lecture 1 5

6 Equivalence classes It is inappropriate to test by brute force, using every possible input value Takes a huge amount of time Is impractical Is pointless! You should divide the possible inputs into groups which you believe will be treated similarly by all algorithms. Such groups are called equivalence classes. A tester needs only to run one test per equivalence class The tester has to - understand the required input, - appreciate how the software may have been designed Lecture 1 6

7 White-box testing Also called glass-box or structural testing Testers have access to the system design They can Examine the design documents View the code Observe at run time the steps taken by algorithms and their internal data Lecture 1 7

8 Strategies for Testing Large Systems Big bang testing versus integration testing In big bang testing, you take the entire system and test it as a unit A better strategy in most cases is integration testing: You test each individual subsystem in isolation Continue testing as you add more and more subsystems to the final product Lecture 1 8

9 Lecture 2: Scrum Methodology 9

10 Scrum Framework Roles Product Owner Scrum Master Development Team Events Sprint Sprint Planning Daily Scrum Sprint Review Sprint Retrospective Artifacts Project Charter Product Backlog Sprint Backlog Burn down chart 10

11 Sprint Considered the heart of Scrum A sprint is a time-box of one month or less (usually between 2-4 weeks) during which a Done, useable, and potentially releasable product increment is created. Sprints have consistent durations throughout a development effort. A new Sprint starts immediately after the conclusion of the previous Sprint. During the Sprint: No changes are made that would affect the Sprint Goal Development Team composition remains constant Quality goals do not decrease Scope may be clarified and re-negotiated between the Product Owner and Development Team as more is learned 11

12 Lecture 3: Testing Strategies 12

13 Pre and Post-conditions A pre-condition defines assumptions expected to hold prior to the execution of a piece of code (e.g., basic block, function) A post-condition defines guarantees on a program state after the execution of a piece of code. class Stack<T> { T[] array; int size; Pre: s.size() > 0 T pop() { return array[--size]; } Post: s.size() == s.size() - 1 } int size() { return size; } 4 13

14 Code Coverage Function coverage: which functions were called? Statement coverage: which statements were executed? Branch coverage: which brancheswere taken? Many others: line coverage, condition coverage, basic block coverage, path coverage, 8 14

15 Incremental Design Should you test a program by testing each module independently and then by combining the modules to form the program (non-incremental or big-bang testing)? Or should you combine the next module to be tested with the set of previouslytested modules (incremental testing or integration testing)? Lines connecting the modules represent the control hierarchy of the program 12 15

16 Non-incremental Testing First, a module test is performed on each of the six modules, testing each module as a stand-alone entity Modules might be tested at the same time or in succession The testing of each module can require a special driver module and one or more stub modules B requires a driver for A and a stub for E When the module testing of all six modules has been completed, the modules are combined to form the program 13 16

17 Lecture 3: Testing Strategies Incremental Testing Top Down: Test module A Test modules B, C, D Test modules E, F Need stubs Nonincremental testing requires more work (more drivers and stubs) Less work is required for incremental testing because previously-tested modules are used instead of the driver modules (if you start from the top) or stub modules (if you start from the bottom) needed in the nonincremental approach 15 17

18 Lecture 4: Automated Testing 18

19 Revised algorithm Korat Enumerate shapes through their vector representation Initial candidate vector: all fields are null Generate next shape by: - expanding the last field accessed in the pre-condition - backtracking if all possibilities for the field are exhausted Key idea: never expand parts of the input not examined by the precondition. Discard shapes isomorphic to previously-generated shapes 14 19

20 Randoop A test is a sequence of method calls over a library Basic idea: - build new testcases incrementally, extending past sequences - Once a sequence is generated, execute it - Use the output of an executed test to guide generation of new tests that produce new object states 20 20

21 Summary 27 21

22 Lecture 5: Quickcheck 22

23 Quickcheck Rather than exhaustively thinking of new unit tests, think of properties (a predicate that must hold on all executions) and check validity of executions against that property A critical property of sorting is that it is idempotent: trying to sort a sorted list returns the same list. Quickcheck is a tool that lets you express properties and uses random testing to check when those properties hold. 23

24 Basic Idea Programmers define properties they would like to be true about their programs Programmers use default or user defined random test data generators The QuickCheck framework tests the properties against a large number of random inputs It reports the success and failure (along with a counterexample) QuickCheck is in some sense orthogonal to the kind of testing done in other unit testing frameworks. 24

25 Lecture 6: Dynamic Symbolic Execution 25

26 Dynamic Symbolic Execution A specific (and particularly useful) kind of whitebox testing is dynamic symbolic execution Basic idea: - Maintain both concrete and symbolic representations of program state at every program point - Use a theorem prover or constraint solver to guide execution along unexplored paths - Goal: explore all paths in the method under test 26

27 Symbolic Execution Rather than simply relying on random execution to reach an error state, we might symbolically evaluate the program. A symbolic value represents a set of concrete values (e.g., a type). Collect constraints on these values along different paths. Use a theorem prover to reason about these constraints. - To reach the error state, we need to discover a value for x such that (x * 3 == 15) /\!(x % 5) == 0 - A theorem prover would respond that no such x exists, allowing us to conclude that the error condition cannot arise. 27

28 Dynamic Symbolic Execution (Concolic Testing) This approach allows the theorem prover to be incomplete - concrete values are used to simplify constraints No unsatisfiable constraint will be deemed satisfiable 28

29 Dynamic vs. Symbolic Execution A dynamic analysis never examines an execution that could not occur. - Never returns false positives - every error is a true error - But, may have false negatives - not all (latent) errors will be discovered Symbolic execution attempts to examine all paths - But, not all such paths are actually realizable (feasible) - It can thus have false positives - not every error it reports will be a true error - But, will never have false negatives - it will never incorrectly declare a program to be error-free (thus, it is always sound) 29

30 Lecture 7: Delta Debugging 30

31 Inputs and Failures Let R be the set of possible inputs r P R corresponds to an input that passes r F R corresponds to an input that fails Changes 31

32 Decomposing Changes 32

33 Minimization 33

34 1-minimality 34

35 Lecture 8: Higher-Order Testing 35

36 Basics A software error occurs when the program does not do what its end user reasonably expects it to do Even if you could perform an absolutely-perfect module test, you still could not guarantee that you have found all software errors To complete testing, higher-order testing is necessary - Chapter 6, Myers Software development is largely a process of communicating information about the eventual product and translating this information from one form to another 36

37 Basic Idea Goal: detect configuration errors early Observation: code already has checks built-in (but, triggered too late) Idea: Copy-paste configuration checks to initialization time use(cfg1) use(cfg1) use(cfg2) use(cfg2) initialization rollout workload error observation period production 37 Xu 16

38 Problem /* sys_24-7.c */ static int sys_init() { load_config(); } int rv = execvp(dial_prog_path, args); Paste int rv = execvp(diag_prog_path, args); Problem 1 Executing code needs context args is undefined Problem 2 Execution can have side effects prank calls are crimes. exec() removes the current process image 38 Xu 16

39 Lecture 9: Concurrency Testing 39

40 The Heisenbug problem Concurrent executions are highly nondeterminisitic Rare thread interleavings result in Heisenbugs Difficult to find, reproduce, and debug Observing the bug can fix it Likelihood of interleavings changes, say, when you add printfs A huge productivity problem Developers and testers can spend weeks chasing a single Heisenbug

41 CHESS in a nutshell CHESS is a user-mode scheduler Controls all scheduling nondeterminism Replace the OS scheduler Guarantees: Every program run takes a different thread interleaving Reproduce the interleaving for every run

42 CHESS architecture While(not done) { TestScenario() } Win32 API CHESS Program TestScenario() { } Kernel: Threads, Scheduler, Synchronization Objects CHESS runs the scenario in a loop Every run takes a different interleaving Every run is repeatable Intercept synch. & threading calls To control and introduce nondeterminism Detect Assertion violations Deadlocks Dataraces Livelocks

43 Lecture 10: Compiler Testing 43

44 Requirements Unambiguous: avoid undefined or unspecified behaviors that create ambiguous meanings of a program Integer undefined behavior Use without initialization Unspecified evaluation order Use of dangling pointer Null pointer dereference OOB array access Expressiveness: support most commonly used C features Integer operations Loops (with break/continue) Conditionals Function calls Const and volatile Structs and Bitfields Pointers and arrays Goto 44

45 Avoiding Undefined/unspecified Behaviors Problem Generation Time Solution Run Time Solution Integer undefined behaviors Use without initialization Constant folding/ propagation Algebraic simplification explicit initializers Safe math wrappers OOB array access Force index within range Take modulus Null pointer dereference Use of dangling pointers Unspecified evaluation order Inter-procedural points-to analysis Inter-procedural points-to analysis Inter-procedural effect analysis 45

46 Equivalence Modulo Inputs! Relax equiv. wrt a given input " Variants must satisfy P(i) = P k (i) on input i " But may differ on other input j: P(j) P k (j)! Exploit close interplay between " Dynamic program execution on some input " Static compilation for all input 46