Chapter 11, Testing. Using UML, Patterns, and Java. Object-Oriented Software Engineering

Transcription

1 Chapter 11, Testing Using UML, Patterns, and Java Object-Oriented Software Engineering

2 Outline Terminology Types of errors Dealing with errors Quality assurance vs Testing Component Testing! Unit testing! Integration testing Testing Strategy Design Patterns & Testing System testing! Function testing! Structure Testing! Performance testing! Acceptance testing! Installation testing Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 2

3 What is this? A failure? An error? A fault? Need to specify the desired behavior first! Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 3

4 Erroneous State ( Error ) Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 4

5 Algorithmic Fault Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 5

6 Mechanical Fault Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 6

7 Terminology Reliability: The measure of success with which the observed behavior of a system confirms to some specification of its behavior. Failure: Any deviation of the observed behavior from the specified behavior. Error: The system is in a state such that further processing by the system will lead to a failure. Fault (Bug): The mechanical or algorithmic cause of an error. There are many different types of errors and different ways how we can deal with them. Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 7

8 How do we deal with Errors and Faults? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 8

9 Verification? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 9

10 Modular Redundancy? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 10

11 Declaring the Bug as a Feature? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 11

12 Patching? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 12

13 Testing? Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 13

14 Examples of Faults and Errors Faults in in the the Interface specification! Mismatch between what the the client needs and and what the the server offers! Mismatch between requirements and and implementation Algorithmic Faults! Missing initialization! Branching errors (too (too soon, too too late) late)! Missing test test for for null null Mechanical Faults (very hard to to find)! Documentation does does not not match actual conditions or or operating procedures Errors! Stress or or overload errors! Capacity or or boundary errors! Timing errors! Throughput or or performance errors Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 14

15 Dealing with Errors Verification:! Assumes hypothetical environment that does not match real environment! Proof might be buggy (omits important constraints; simply wrong) Modular redundancy:! Expensive Declaring a bug to be a feature! Bad practice Patching! Slows down performance Testing (this lecture)! Testing is never good enough Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 15

16 Another View on How to Deal with Errors Error prevention (before the system is released):! Use good programming methodology to reduce complexity! Use version control to prevent inconsistent system! Apply verification to prevent algorithmic bugs Error detection (while system is running):! Testing: Create failures in a planned way! Debugging: Start with an unplanned failures! Monitoring: Deliver information about state. Find performance bugs Error recovery (recover from failure once the system is released):! Data base systems (atomic transactions)! Modular redundancy! Recovery blocks Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 16

17 Some Observations It is impossible to completely test any nontrivial module or any system! Theoretical limitations: Halting problem! Practial limitations: Prohibitive in time and cost Testing can only show the presence of bugs, not their absence (Dijkstra) Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 17

18 Testing takes creativity Testing often viewed as dirty work. To develop an effective test, one must have: " Detailed understanding of the system " Knowledge of the testing techniques " Skill to apply these techniques in an effective and efficient manner Testing is done best by independent testers! We often develop a certain mental attitude that the program should in a certain way when in fact it does not. Programmer often stick to the data set that makes the program work! "Don t mess up my code!" A program often does not work when tried by somebody else.! Don't let this be the end-user. Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 18

19 Testing Activities Subsystem Code Subsystem Code Unit Test Unit Test Tested Subsystem Tested Subsystem System Design Document Integration Test Integrated Subsystems Requirements Analysis Document Functional Test User Manual Functioning System Tested Subsystem Subsystem Code Unit Test All All tests tests by by developer Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 19

20 Testing Activities continued Global Requirements Client s Understanding of Requirements User Environment Functioning System Performance Test Validated System Acceptance Test Accepted System Installation Test Tests by by developer Tests by by client User s understanding Tests (?) (?) by by user user System in Use Usable System Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 20

21 Fault Handling Techniques Fault Handling Fault Avoidance Fault Detection Fault Tolerance Design Methodology Reviews Atomic Transactions Modular Redundancy Verification Configuration Management Testing Debugging Unit Testing Integration Testing System Testing Correctness Debugging Performance Debugging Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 21

22 Quality Assurance encompasses Testing Quality Assurance Usability Testing Scenario Testing Prototype Testing Product Testing Fault Avoidance Fault Tolerance Verification Configuration Management Fault Detection Atomic Transactions Modular Redundancy Reviews Walkthrough Inspection Testing Debugging Unit Testing Integration Testing System Testing Correctness Debugging Performance Debugging Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 22

23 Types of Testing Unit Testing:! Individual subsystem! Carried out by developers! Goal: Confirm that subsystems is correctly coded and carries out the intended functionality Integration Testing:! Groups of subsystems (collection of classes) and eventually the entire system! Carried out by developers! Goal: Test the interface among the subsystem Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 23

24 System Testing System Testing:! The entire system! Carried out by developers! Goal: Determine if the system meets the requirements (functional and global) Acceptance Testing:! Evaluates the system delivered by developers! Carried out by the client. May involve executing typical transactions on site on a trial basis! Goal: Demonstrate that the system meets customer requirements and is ready to use Implementation (Coding) and testing go hand in hand Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 24

25 Unit Testing Informal:! Incremental coding Static Analysis:! Hand execution: Reading the source code! Walk-Through (informal presentation to others)! Code Inspection (formal presentation to others)! Automated Tools checking for " syntactic and semantic errors " departure from coding standards Dynamic Analysis:! Black-box testing (Test the input/output behavior)! White-box testing (Test the internal logic of the subsystem or object)! Data-structure based testing (Data types determine test cases) Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 25

26 Black-box Testing Focus: I/O behavior. If for any given input, we can predict the output, then the module passes the test.! Almost always impossible to generate all possible inputs ("test cases") Goal: Reduce number of test cases by equivalence partitioning:! Divide input conditions into equivalence classes! Choose test cases for each equivalence class. (Example: If an object is supposed to accept a negative number, testing one negative number is enough) Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 26

27 Black-box Testing (Continued) Selection of equivalence classes (No rules, only guidelines):! Input is valid across range of values. Select test cases from 3 equivalence classes: " Below the range " Within the range " Above the range! Input is valid if it is from a discrete set. Select test cases from 2 equivalence classes: " Valid discrete value " Invalid discrete value Another solution to select only a limited amount of test cases:! Get knowledge about the inner workings of the unit being tested => white-box testing Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 27

28 White-box Testing Focus: Thoroughness (Coverage). Every statement in the component is executed at least once. Four types of white-box testing! Statement Testing! Loop Testing! Path Testing! Branch Testing Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 28

29 White-box Testing (Continued) Statement Testing (Algebraic Testing): Test single statements (Choice of operators in polynomials, etc) Loop Testing:! Cause execution of the loop to be skipped completely. (Exception: Repeat loops)! Loop to be executed exactly once! Loop to be executed more than once Path testing:! Make sure all paths in the program are executed Branch Testing (Conditional Testing): Make sure that each possible outcome from a condition is tested at least once Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 29

30 White-box Testing Example FindMean(float Mean, FILE ScoreFile) { SumOfScores = 0.0; NumberOfScores = 0; Mean = 0; Read(ScoreFile, Score); /*Read in and sum the scores*/ while (! EOF(ScoreFile) { if ( Score > 0.0 ) { SumOfScores = SumOfScores + Score; NumberOfScores++; } Read(ScoreFile, Score); } /* Compute the mean and print the result */ if (NumberOfScores > 0 ) { Mean = SumOfScores/NumberOfScores; printf("the mean score is %f \n", Mean); } else printf("no scores found in file\n"); } Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 30

31 White-box Testing Example: Determining the Paths FindMean (FILE ScoreFile) { float SumOfScores = 0.0; int NumberOfScores = 0; 1 float Mean=0.0; float Score; Read(ScoreFile, Score); 2 while (! EOF(ScoreFile) { 3 if (Score > 0.0 ) { SumOfScores = SumOfScores + Score; NumberOfScores++; } 5 } 7 Read(ScoreFile, Score); 6 } /* Compute the mean and print the result */ if (NumberOfScores > 0) { Mean = SumOfScores / NumberOfScores; printf( The mean score is %f\n, Mean); } else printf ( No scores found in file\n ); Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 31

32 Constructing the Logic Flow Diagram Start 1 F 2 T 3 T F T F 8 9 Exit Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 32

33 Finding the Test Cases Start 1 a (Covered by any data) 2 b (Data set must contain at least one value) (Positive score) 3 d e (Negative score) c 4 5 (Data set must f g h (Reached if either f or be empty) 6 e is reached) (Total score < 0.0) i 7 j (Total score > 0.0) 8 9 k l Exit Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 33

34 Comparison of White & Black-box Testing White-box Testing:! Potentially infinite number of paths have to be tested! White-box testing often tests what is done, instead of what should be done! Cannot detect missing use cases Black-box Testing:! Potential combinatorical explosion of test cases (valid & invalid data)! Often not clear whether the selected test cases uncover a particular error! Does not discover extraneous use cases ("features") Both types of testing are needed White-box testing and black box testing are the extreme ends of a testing continuum. Any choice of test case lies in between and depends on the following:! Number of possible logical paths! Nature of input data! Amount of computation! Complexity of algorithms and data structures Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 34

35 The 4 Testing Steps Select what has to to be be measured! Analysis: Completeness of of requirements! Design: tested for for cohesion! Implementation: Code tests tests Decide how the the testing is is done! Code inspection! Proofs (Design by by Contract)! Black-box, white box, box,! Select integration testing strategy (big (big bang, bottom up, up, top top down, sandwich) Develop test cases! A test test case case is is a set set of of test test data data or or situations that that will will be be used used to to exercise the the unit unit (code, module, system) being tested or or about the the attribute being measured Create the the test oracle! An An oracle contains of of the the predicted results for for a set set of of test test cases! The The test test oracle has has to to be be written down before the the actual testing takes place Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 35

36 Guidance for Test Case Selection Use analysis knowledge about functional requirements (black-box testing):! Use Use cases! Expected input data data! Invalid input data data Use design knowledge about system structure, algorithms, data structures (white-box testing):! Control structures " Test " Test branches, loops, loops,......! Data Data structures " Test " Test records records fields, fields, arrays, arrays, Use implementation knowledge about algorithms:! Examples:! Force division by by zero zero! Use Use sequence of of test test cases for for interrupt handler Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 36

37 Unit-testing Heuristics Create unit unit tests tests as as soon soon as as object design is is completed:! Black-box test: test: Test Test the the use use cases & functional model! White-box test: test: Test Test the the dynamic model! Data-structure test: test: Test Test the the object model Develop the the test test cases! Goal: Find Find the the minimal number of of test test cases to to cover as as many paths as as possible Cross-check the the test test cases to to eliminate duplicates! Don't waste your your time! Desk Desk check your your source code code! Reduces testing time time Create a test test harness! Test Test drivers and and test test stubs are are needed for for integration testing Describe the the test test oracle! Often the the result of of the the first first successfully executed test test Execute the the test test cases! Don t forget regression testing! Re-execute test test cases every time time a change is is made Compare the the results of of the the test test with with the the test test oracle! Automate as as much as as possible Bernd Bruegge & Allen H. Dutoit Object-Oriented Software Engineering: Using UML, Patterns, and Java 37