Software Testing Minsoo Ryu Hanyang University
Topics covered 1. Testing Goals and Principles 2. Testing Process 3. Testing Strategies Component testing Integration testing Validation/system testing 4. Testing Tactics 5. Debugging 2 2
1. Testing Goals and Principles Testing goals (Glen Meyer) Testing is a process of executing a program with the intent of finding an error A good test case is one that has a high probability of finding an as-yet- undiscovered error A successful test is one that uncovers an as-yetundiscovered error 3 3
Testing Goals and Principles Five testing principles by Davis All tests should be traceable to customer requirements Tests should be planned long before testing begins Test planning can begin as soon as the analysis model complete The Pareto principle applies to software testing 80 percent of all errors uncovered during testing will likely be traceable to 20 percent of all program components The Pareto principle (also known as the 80-20 rule, the law of the vital few and the principle of factor sparsity) states that for many phenomena, 80% of the consequences stem from 20% of the causes Testing should begin in the small and progress toward testing in the large Exhaustive testing is not possible 4 4
Topics covered 1. Testing Goals and Principles 2. Testing Process 3. Testing Strategies Component testing Integration testing Validation/system testing 4. Testing Tactics 5. Debugging 5 5
2. Testing Process Testing begins in the small and progresses to the large Testing process Unit (component or module) test Integration test Validation and system test 6 6
The Testing Process 7 7
Topics covered 1. Testing Goals and Principles 2. Testing Process 3. Testing Strategies Component testing Integration testing Validation/system testing 4. Testing Tactics 5. Debugging 8 8
3. Testing Strategies A testing strategy integrates software test case design methods into a well-planned series of steps The strategy provides a road map that describes the steps to be constructed as part of testing, when these steps are planned and then undertaken, and how much effort, time, and resources will be required Therefore any testing strategy must incorporate test planning, test case design, test execution, and resultant data collection and evaluation 9 9
Organizing for Software Testing Who tests the software? From a psychological point of view, software analysis, design, and construction (coding) are constructive tasks From the point of view of the builder, testing can be considered destructive So the builder treads lightly, designing and executing tests that will demonstrate that the program works, rather than uncovering errors The role of independent test group (ITG) is to remove the above inherent problems The developer and the ITG work closely throughout a software project to ensure that thorough tests will be conducted 10 10
Organizing for Software Testing developer Understands the system but, will test "gently" and, is driven by "delivery" independent tester Must learn about the system, but, will attempt to break it and, is driven by quality 11 11
Component (Unit or Module) Testing Each module is tested to uncover errors within the boundary of the module Module interface Local data structures All independent paths (basis paths) Boundary conditions Error handling paths 12 12
Component (Unit or Module) Testing Module interface Tests of data flow across a module interface are required before any other test is initiated Local data structures Local data structures should be exercised and the local impact on global data should be ascertained Execution paths Test cases should be designed to uncover errors due to erroneous computations, incorrect comparisons, or improper control flow Boundary conditions Tests cases that exercise data structures, control flow, and data values just below, at, and just above maxima and minima are very likely to uncover errors Error handling paths Good design dictates that error conditions be anticipated and errorhandling paths set up to reroute or cleanly terminate processing when an error does occur Unfortunately, there is a tendency to incorporate error handling into software and then never test it 13 13
Component (Unit or Module) Testing Unit test environment A driver is a main program that accepts tests case data, passes such data to the component, and prints relevant results Stubs serve to replace modules that are subordinate to the component to be tested A stub or dummy subprogram uses the subordinate module interface, may do minimal data manipulation, provides verification of entry, and returns control to the module undergoing testing Some comments on unit testing Drivers and stubs are overhead (not delivered) Unit testing is simplified when a component with high cohesion is designed 14 14
Unit Test Environment driver stub Module stub interface local data structures boundary conditions independent paths error handling paths test cases RESULTS 15 15
Integration Testing Integration testing is a systematic technique for constructing the software architecture while at the same time conducting tests to uncover errors associated with interfacing Two approaches Big bang approach All components are combined in advance and the entire program is tested as a whole Chaos usually results, and correction is difficult because isolation of cause is complicated by the vast expense of the entire program Incremental approach The program is constructed and tested in small increments Top-down vs. bottom-up 16 16
Incremental Integration Testing A T1 A T1 A T1 T2 B T2 T2 B T3 B T3 C T3 T4 C T4 D T5 Test sequence 1 Test sequence 2 Test sequence 3 17 17
Top-Down Integration Modules are integrated by moving downward through the control hierarchy, beginning with the main control module Modules subordinate to the main control module are incorporated into the structure in either a depth-first or breadth-first manner The top-down integration strategy verifies major control or decision points early in the test process 18 18
Top-Down Integration A top module is tested with stubs B F G D C E stubs are replaced one at a time, "depth first" as new modules are integrated, some subset of tests is re-run 19 19
Top-Down Integration Top-down strategy sounds relatively uncomplicated, but, in practice, logistical problems can arise The most common of these problems occurs when processing at low levels in the hierarchy is required adequately test upper levels Stubs replace low-level modules at the beginning of topdown testing; therefore no significant data can flow upward in the program structure Two choices: (1) delay many tests until stubs are replaced with actual modules, or develop stubs that perform limited functions that simulate actual module 20 20
Bottom-Up Integration Bottom-up integration testing begins construction and testing with atomic modules Low-level components are combined into clusters (builds) A driver is written to coordinate test case input and output The cluster is tested Derivers are removed and clusters are combined moving upward in the program structure 21 21
Bottom-Up Integration A B F G C drivers are replaced one at a time, "depth first" D E worker modules are grouped into builds and integrated cluster 22 22
Regression Testing Each time a new module is added as part of integration testing, the software changes New data flow paths are established, new I/O may occur, and new control logic is invoked Regression testing is the re-execution of some subset of tests that have already been conducted to ensure that changes have not propagated unintended side effects 23 23
Validation and System Testing Ultimately, software is incorporated with other system elements (e.g., hardware, people, information), and a series of system integration and validation tests are conducted Software validation is achieved through a series of tests that demonstrate conformity with requirements A classic system testing problem is finger-pointing When an error uncovered, each system element developer blames the other for the problem 24 24
Types of Validation and System Testing Recovery testing Forces the software to fail in a variety of ways and verifies that recovery is properly performed If recovery is automatic, reinitialization, checkpointing mechanisms, data recovery, and restart are evaluated If recovery requires human intervention, the mean-time-tofailure (MTTR) is evaluated whether it is within acceptable limits Security testing Verifies that protection mechanisms built into a system will protect it from improper penetration 25 25
Types of Validation and System Testing Stress testing Executes a system in a manner that demands resources in abnormal quantity, frequency, or volume For example, input data rates may be increased by an order of magnitude to determine how input functions will respond Performance testing Test the run-time performance of software within the context of an integrated system 26 26
Types of Validation and System Testing Alpha testing Alpha test is conducted in a controlled manner at the developer s site by end-users The software is used in a natural setting with the developer looking over the shoulder of typical users and recording errors and usage problems Beta test Beta test is conducted at end-user sites The developer is generally not present The beta test is a live application of the software in an environment that cannot be controlled by the developer The end-user records and reports all problems 27 27
Topics covered 1. Testing Goals and Principles 2. Testing Process 3. Testing Strategies Component testing Integration testing Validation/system testing 4. Testing Tactics 5. Debugging 28 28
4. Testing Tactics (Methods) white-box methods black-box methods Testing Methods Testing Strategies 29 29
White Box vs. Black Box Testing White box testing Knowing the internal workings of a product Can be conducted only when component-level design (or source code) exists e.g.) Basis path testing and control structure testing Black box testing Knowing the specified function that a product has been to perform Alludes to tests that are conducted at the software interface e.g.) Equivalence partitioning, boundary value analysis, and orthogonal array testing 30 30
Basis Path Testing Basis path testing is a white box testing proposed by Tom McCabe The objective of path testing is to ensure that the set of test cases is such that each path through the program is executed at least once The starting point for path testing is a program flow graph that shows nodes representing program decisions and arcs representing the flow of control Statements with conditions are therefore nodes in the flow graph Sequence If While Until Case 31 31
Exhaustive testing Basis Path Testing loop < 20 X There are 10 14 paths for the above small program! If the processor can exercise each test case in one millisecond, working 24 hours a day, 365 days a year, the processor would work for 3170 years 32 32
Selective testing Basis Path Testing Selected path loop < 20 X 33 33
Independent Program Path An independent path is any path through the program that introduces at least one new set of processing statements or a new condition An independent path must move along at least one edge that has not been traversed before the path is defined Path 1: 1,2,3,6,7,8 Path 2: 1,2,3,5,7,8 Path 3: 1,2,4,7,8 Path 4: 1,2,4,7,2,4,...7,8 4 1 2 3 5 6 Path <1, 2, 3, 5, 7, 2, 4, 7, 8> is not an independent path (simply a combination) 7 8 34 34
Independent Program Path Paths 1, 2, 3, 4 constitute a basis set The basis set is not unique A number of different basis sets can be derived for a given design How do we know how many paths to look for? The computation of cyclomatic complexity provides the answer Cyclomatic complexity V(G) The number of regions The number of edges the number of nodes + 2 The number of predicate nodes + 1 Each node that contains a condition is called a predicate node 35 35
Condition Testing Condition testing exercises the logical conditions contained in a program module A simple condition is a Boolean variable or a relational expression A relational expression E 1 <relational-operator> E 2 <relational-operator> Less (greater) than, less (greater) than or equal to, equal to, not equal to A compound condition is composed of two or more simple conditions, Boolean operators, and parentheses Boolean operators: OR, AND, NOT 36 36
Data Flow Testing Selects test paths of a program according to the locations of definitions and uses of variables in the program For a statement with S as its statement number DEF(S) = {X statement S contains a definition of X} USE(S) = {X statement S contains a use of X} The definition of variable X at statement S is said to be live at statement S if there exists a path from statement S to statement S that contains no other definition of X A definition-use (DU) chain of variable X is of the form [X, S, S ] One simple data flow testing is to require that every DU chain be covered at least once 37 37
Loop Testing Loop testing focuses exclusively on the validity of loop constructs Simple loop Nested Loops Concatenated Loops Unstructured Loops 38 38
Loop Testing Minimum conditions Simple Loops 1. skip the loop entirely 2. only one pass through the loop 3. two passes through the loop 4. m passes through the loop m < n 5. (n-1), n, and (n+1) passes through the loop where n is the maximum number of allowable passes 39 39
Loop Testing Nested Loops Start at the innermost loop. Set all outer loops to their minimum iteration parameter values. Test the min+1, typical, max-1 and max for the innermost loop, while holding the outer loops at their minimum values. Move out one loop and set it up as in step 2, holding all other loops at typical values. Continue this step until the outermost loop has been tested. Concatenated Loops If the loops are independent of one another then treat each as a simple loop else* treat as nested loops endif* for example, the final loop counter value of loop 1 is used to initialize loop 2. 40 40
Black Box Testing requirements output input events 41 41
Equivalence Partitioning Equivalence partitioning divides the input domain of a program into classes of data from which test cases can be derived Invalid inputs Valid inputs System Outputs 42 42
Boundary Value Analysis Equivalence class boundaries Elements < Mid Elements > Mid Mid-point 43 43
Orthogonal Array Testing Orthogonal array testing can be applied to problems in which the input domain is relatively small but too large to accommodate exhaustive testing In any two of columns, each pair of symbols occurs exactly the same number of times Z Z Y X Y X One input item at a time L9 orthogonal array 44 44
Orthogonal Array Testing P1 = {0, 1, 2}, P2 = {0, 1, 2}, P3 = {0, 1, 2}, P4 = {0, 1, 2} P1 P2 P3 P4 0 0 0 0 1 0 0 0 2 0 0 0 0 1 0 0 0 2 0 0 0 0 1 0 0 0 2 0 0 0 0 1 0 0 0 2 <One at a time> P1 P2 P3 P4 0 0 2 1 0 1 0 0 0 2 1 2 2 0 0 2 2 1 1 1 2 2 2 0 1 0 1 0 1 1 2 2 1 2 0 1 <Orthogonal array> 45 45
46 46
47 47
48 48
Topics covered 1. Testing Goals and Principles 2. Testing Process 3. Testing Strategies Component testing Integration testing Validation/system testing 4. Testing Tactics 5. Debugging 49 49
5. Debugging Debugging is an action that results in the removal of the error uncovered by the testing process test cases regression tests corrections new test cases suspected causes identified causes Debugging results 50 50
Debugging Effort Brian Kingdom, Everyone knows that debugging is twice as hard as writing a program in the first place. So if you are as clever as you can be when you write it, how will you ever debug it? time required to correct the error and conduct regression tests time required to diagnose the symptom and determine the cause 51 51
Symptoms & Causes symptom and cause may be geographically separated symptom may disappear when another problem is fixed cause may be due to a combination of non-errors cause may be due to a system or compiler error symptom cause cause may be due to assumptions that everyone believes symptom may be intermittent 52 52
Brute Force Debugging Brute force debugging is probably the most common and least efficient method for isolating the cause of a software error Using a let the computer find the error philosophy, memory dumps are taken, run-time traces are invoked, and the program is loaded with output statements We hope that somewhere in the morass of information we will find a clue that can lead us to the cause of an error Although the mass of information produced may ultimately lead to success, it more frequently leads to wasted effort and time 53 53
Backtracking Backtracking is a fairly a common debugging approach that can be used successfully in small programs Beginning at the site where a symptom has been uncovered, the source code is traced backward (manually) until the site of the cause is found Unfortunately, as the number of source lines increases, the number of potential backward paths may become unmanageably large 54 54
Cause Elimination Cause elimination is manifested by induction or deduction and introduces the concept of binary partitioning Data related to the error occurrence are organized to isolate potential causes A cause hypothesis is devised, and the aforementioned data are used to prove or disprove the hypothesis Alternatively, a list of all possible causes is developed, and tests are conducted to eliminate each If initial tests indicate that a particular cause hypothesis shows promise, data are refined in an attempt to isolate the bug 55 55
The Final Maxim When all else fails, get help! Set a time limit, say, one hour, on the time you spend trying to debug a problem on your own After that, get help 56 56