Software Engineering II Testing

Transcription

1 Object-Oriented Software Engineering! Using UML, Patterns, and Java! Software Engineering II Testing 5 May 2009 Bernd Bruegge Applied Software Engineering Technische Universität München

2 Outline of Testing Today:! Terminology! Testing Activities! Unit testing! Integration testing! Testing strategy! Design patterns & testing! System testing! Function testing! Acceptance testing. Next Lecture:! Model-based Testing! U2TP Testing Profile 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 2!

3 Famous Problems! A F-16 fighter plane crossed the equator using autopilot! Result: plane flipped over! Reason?! Reuse of autopilot software from a rocket! The Therac-25 was a machine used for radiation therapy! Result: Patients were given massive overdoses of radiation, approximately 100 times the intended dose (6 deaths)"! Reason: Failure only occurred when a particular sequence of keys was entered on the VT-100 controlling a PDP-11: X,!, E, Enter! The "X erroneously selected 25MV photon mode. The Cursor up was an attempt to correct the mistake. The "E followed by "Enter then attempted to correctly select 25 MeV Electron mode.! The machine responded with an error message MALFUNCTION 54, which was not explained in the user manual. Machine shut down briefly and then showed under dosage of radiation.! Because of this the operator hit P to continue treatment.

4 Terminology! Failure: Any deviation of the observed behavior from the specified behavior! Erroneous state (error): The system is in a state such that further processing by the system can lead to a failure! Fault: The mechanical or algorithmic cause of an error ( bug )! Validation: Activity of checking for deviations between the observed behavior of a system and its specification Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 4!

5 What is this? A failure? An error? A fault? We eed to describe specified and desired behavior first! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 5!

6 Erroneous State ( Error ) 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 6!

7 Algorithmic Fault 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 7!

8 Mechanical Fault Superuser! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 8!

9 The F-16 Problem! Where is the failure?! Where is the error?! Where is the fault?! Bad use of implementation inheritance! A Plane is not a rocket. Rocket! Plane! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 9!

10 Examples of Faults and Errors! Faults in the Interface specification! Mismatch between what the client needs and what the server offers! Mismatch between requirements and implementation! Algorithmic Faults! Missing initialization! Incorrect branching condition! Missing test for null! Mechanical Faults (very hard to find)! Operating temperature outside of equipment specification! Errors! Null reference errors! Concurrency errors! Exceptions Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 10!

11 How do we deal with Errors, Failures and Faults? 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 11!

12 Modular Redundancy 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 12!

13 Declaring the Bug as a Feature 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 13!

14 Patching 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 14!

15 Testing 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 15!

16 Another View on How to Deal with Faults! Fault avoidance! Use a good methodology to reduce complexity! Use configuration management to prevent inconsistency! Apply verification to prevent algorithmic faults! Use reviews! Fault detection! Testing: Activity to provoke failures in a planned way! Debugging: Find and remove the cause (Faults) of an observed failure! Monitoring: Deliver information about state => Used during debugging! Fault tolerance! Enable exception handling! Provide modular redundancy Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 16!

17 Taxonomy for Fault Handling Techniques Fault Handling! Fault " Avoidance! Fault " Detection! Fault " Tolerance! Methodoloy! Configuration" Management! Atomic" Transactions! Modular" Redundancy! Verification! Testing! Debugging! Unit! Testing! Integration" Testing! System " Testing! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 17!

18 Observations! It is impossible to completely test any nontrivial module or system! Practical limitations: Complete testing is prohibitive in time and cost! Theoretical limitations: e.g. Halting problem! Testing can only show the presence of bugs, not their absence (Dijkstra).! Testing is not for free => Define your goals and priorities Edsger W. Dijkstra ( )"" " " "! - First Algol 60 Compiler! "- 1968:! - T.H.E.! - Go To considered Harmful, CACM! "- Since 1970 Focus on Verification! and Foundations of Computer Science! "-"1972 A. M. Turing Award! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 18!

19 Testing takes creativity! To develop an effective test, one must have:! Detailed understanding of the system! Application and solution domain knowledge! Knowledge of the testing techniques! Skill to apply these techniques! Testing is done best by independent testers! We often develop a certain mental attitude that the program should in a certain way when in fact it does not behave!! Programmers often stick to the data set that makes the program work! A program often does not work when tried by somebody else Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 19!

20 Testing Activities Unit" Testing! Integration" Testing! System" Testing! Acceptance" Testing! Object" Design! Document! System" Design! Document! Requirements" Analysis! Document! Client! Expectation! Unit" Testing! Integration" Testing! System" Testing! Acceptance" Testing! Developer! Client! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 20!

21 Types of Testing Unit" Testing! Integration" Testing! System" Testing! Acceptance" Testing!! Unit Testing! Individual component (class or subsystem)! Carried out by developers! Goal: Confirm that the component or subsystem is correctly coded and carries out the intended functionality! Integration Testing! Groups of subsystems (collection of subsystems) and eventually the entire system! Carried out by developers! Goal: Test the interfaces among the subsystems Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 21!

22 Types of Testing continued... Unit" Testing! Integration" Testing! System" Testing! Acceptance" Testing!! System Testing! The entire system! Carried out by developers! Goal: Determine if the system meets the requirements (functional and nonfunctional)! Acceptance Testing! Evaluates the system delivered by developers! Carried out by the client. May involve executing typical transactions on site on a trial basis! Goal: Demonstrate that the system meets the requirements and is ready to use Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 22!

23 When should you write a test?! Traditionally after the source code to be tested! In XP before the source code to be tested! Test-Driven Development Cycle! Add a test! Run the automated tests => see the new one fail! Write some code! Run the automated tests => see them succeed! Refactor code Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 23!

24 Unit Testing Unit" Testing! Integration" Testing! System" Testing! Acceptance" Testing!! Static Testing (at compile time)! Static Analysis! Review! Walk-through (informal)! Code inspection (formal)! Dynamic Testing (at run time)! Black-box testing! White-box testing Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 24!

25 Static Analysis with Eclipse Unit" Testing! Integration" Testing! System" Testing! Acceptance" Testing!! Compiler Warnings and Errors! Possibly uninitialized Variable! Undocumented empty block! Assignment has no effect! Checkstyle! Check for code guideline violations! FindBugs! Check for code anomalies! Metrics! Check for structural anomalies! Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 25!

26 Black-box testing Unit" Testing! Integration" Testing! System" Testing! Acceptance" Testing!! Focus: I/O behavior! If for any given input, we can predict the output, then the component passes the test! Requires test oracle! Goal: Reduce number of test cases by equivalence partitioning:! Divide input conditions into equivalence classes! Choose test cases for each equivalence class Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 26!

27 Black-box testing: Test case selection a) Input is valid across range of values! Developer selects test cases from 3 equivalence classes:! Below the range! Within the range! Above the range b) Input is only valid, if it is a member of a discrete set! Developer selects test cases from 2 equivalence classes:! Valid discrete values! Invalid discrete values! No rules, only guidelines Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 27!

28 Lecture Check: Where are we? Unit" Testing! Integration" Testing! System" Testing! Acceptance" Testing!! Terminology! Testing Activities! Unit testing! Static Testing! Dynamic Testing! Blackbox Testing!!Example 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 28!

29 Black box testing: An example public class MyCalendar { public int getnumdaysinmonth(int month, int year) throws InvalidMonthException { } } Representation for month:! 1: January, 2: February,., 12: December! Representation for year:! 1904, 1999, 2000,, 2006,! How many test cases do we need for the black box testing of getnumdaysinmonth()?! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 29!

30 White-box testing overview Unit" Testing! Integration" Testing! System" Testing! Acceptance" Testing!! Code coverage! Branch coverage! Condition coverage! Path coverage => Details in the exercise session about testing 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 30!

31 Unit Testing Heuristics 1. Create unit tests when object design is completed! Black-box test: Test the functional model! White-box test: Test the dynamic model 2. Develop the test cases! Goal: Find effective number of test cases 3. Cross-check the test cases to eliminate duplicates! Don't waste your time! 4. Desk check your source code! Sometimes reduces testing time 5. Create a test harness! Test drivers and test stubs are needed for integration testing 6. Describe the test oracle! Often the result of the first successfully executed test 7. Execute the test cases! Re-execute test whenever a change is made ( regression testing ) 8. Compare the results of the test with the test oracle! Automate this if possible Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 31!

32 Unit" Testing! Excursion J_Unit: A Java Framework for unit testing Integration" Testing! System" Testing! Acceptance" Testing!! A Java framework for writing and running unit tests! Test cases and fixtures! Test suites! Test runner! Written by Kent Beck and Erich Gamma! Written with test first and pattern-based development in mind! Tests written before code! Allows for regression testing! Facilitates refactoring! JUnit is Open Source! JUnit Version 4.6, released April Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 32!

33 JUnit Classes TestResult Test * run(testresult) Skip! J_Unit! Slides! TestCase testname:string run(testresult) setup() teardown() runtest() ConcreteTestCase setup() teardown() runtest() TestSuite run(testresult) addtest() UnitToBeTested Methods under Test! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 33!

34 An example: Testing MyList! Unit to be tested! MyList! Methods under test! add()! remove()! contains()! size()! Concrete Test case! MyListTestCase 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 34!

35 TestResult Test * run(testresult) TestCase testname:string run(testresult) setup() teardown() runtest() TestSuite run(testresult) addtest() MyListTestCase MyList setup() teardown() runtest() testadd() testremove()! add() remove() contains() size()

36 Writing TestCases in JUnit public class MyListTestCase extends TestCase { public MyListTestCase(String name) { super(name); } public void testadd() { // Set up the test List alist = new MyList(); String anelement = a string ; // Perform the test alist.add(anelement); TestResult // Check if test succeeded asserttrue(alist.size() == 1); asserttrue(alist.contains(anelement)); } protected void runtest() { testadd(); } } TestCase testname:string run(testresult) setup() teardown() runtest() MyListTestCase setup() teardown() runtest() testadd() testremove()! Test run(testresult) * TestSuite run(testresult) addtest() MyList add() remove() contains() size() 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 36!

37 Writing Fixtures and Test Cases public class MyListTestCase extends TestCase { // private MyList alist; private String anelement; public void setup() { alist = new MyList(); anelement = a string ; } public void testadd() { alist.add(anelement); asserttrue(alist.size() == 1); asserttrue(alist.contains(anelement)); } public void testremove() { alist.add(anelement); alist.remove(anelement); asserttrue(alist.size() == 0); assertfalse(alist.contains(anelement)); } Test Fixture Test Case Test Case 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 37!

38 Collecting TestCases into TestSuites public static Test suite() { TestSuite suite = new TestSuite(); suite.addtest(new MyListTest( testadd )); suite.addtest(new MyListTest( testremove )); return suite; } Test * Composite Pattern! run(testresult) TestCase TestSuite testname:string run(testresult) setup() teardown() runtest() run(testresult) addtest() 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 38!

39 Design patterns in JUnit TestResult Test * Command Pattern run(testresult) Composite Pattern Template Method Pattern TestCase testname:string run(testresult) setup() teardown() runtest() TestSuite run(testresult) addtest() Adapter Pattern ConcreteTestCase setup() teardown() runtest() TestedUnit 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 39!

40 Design patterns in JUnit TestResult Test * Command Pattern run(testresult) Composite Pattern Template Method Pattern TestCase testname:string run(testresult) setup() teardown() runtest() TestSuite run(testresult) addtest() Adapter Pattern ConcreteTestCase setup() teardown() runtest() TestedUnit 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 40!

41 Other JUnit features! Textual and GUI interface! Displays status of tests! Displays stack trace when tests fail! Integrated with Maven and Continuous Integration! Build and Release Management Tool! Continous integration server for Java programs! All tests are run before release (regression tests)! Test results are advertised as a project report! Many specialized variants! Unit testing of web applications! J2EE applications 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 41!

42 Additional Readings! [Herman Bruyninckx] Embedded Control Systems Design, WikiBook, Learning from Failure: Embedded_Control_Systems_Design/ Learning_from_failure! [Dijkstra 68]! E. W. Dijkstra, "Letters to the editor: go to statement considered harmful, Communications of the ACM 11 (3): , March 1968! [Meszaros 2007]! G. Meszaros, xunit Test Patterns, Refactoring Test Code, Addison Wesley, 2007! Resources for Test Driven Development 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 42!

43 Object-Oriented Software Engineering! Using UML, Patterns, and Java! Testing Part 2: Integration & System Testing

44 Overview! Integration testing! Big bang! Bottom up! Top down! Sandwich! Continuous! System testing! Functional! Performance! Acceptance testing! Summary 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 44!

45 Integration Testing! The entire system is viewed as a collection of subsystems (sets of classes) determined during the system and object design! Goal: Test all interfaces between subsystems and the interaction of subsystems! The Integration testing strategy determines the order in which the subsystems are selected for testing and integration Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 45!

46 Why do we do integration testing?! Unit tests only test the unit in isolation! Many failures result from faults in the interaction of subsystems! Often many Off-the-shelf components are used that cannot be unit tested! Without integration testing the system test will be very time consuming! Failures that are not discovered in integration testing will be discovered after the system is deployed and can be very expensive Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 46!

47 Stubs and drivers! Driver:! A component, that calls the TestedUnit! Controls the test cases! Stub:! A component, the TestedUnit depends on! Partial implementation! Returns fake values. Driver! Tested! Unit! Stub! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 47!

48 Example: A 3-Layer-Design A! Spread" A! SheetView! (Spreadsheet) Layer I! B! Entity! Data" B! Model! C! Calculator! C! D! Currency! D! Converter! Layer II! E! F! G! BinaryFile" XMLFile" Currency" E! F! G! Storage! Storage! DataBase! Layer III! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 48!

49 Big-Bang Approach A! Test A! B! C! D! Test B! Test C! Test D! Test E! Test F! Test G! Test! A, B, C, D,! E, F, G! E! F! G! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 49!

50 Bottom-up Testing Strategy! The subsystems in the lowest layer of the call hierarchy are tested individually! Then the next subsystems are tested that call the previously tested subsystems! This is repeated until all subsystems are included! Drivers are needed Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 50!

51 Bottom-up Integration A! B! C! D! Test E! E! F! G! Test B, E, F! Test F! Test C! Test! A, B, C, D,! E, F, G! Test G! Test D,G! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 51!

52 Pros and Cons of Bottom-Up Integration Testing! Con:! Pro! Tests the most important subsystem (user interface) last! Drivers needed! No stubs needed! Useful for integration testing of the following systems! Object-oriented systems! Real-time systems! Systems with strict performance requirements Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 52!

53 Top-down Testing Strategy! Test the top layer or the controlling subsystem first! Then combine all the subsystems that are called by the tested subsystems and test the resulting collection of subsystems! Do this until all subsystems are incorporated into the test! Stubs are needed to do the testing Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 53!

54 Top-down Integration A! B! C! D! E! F! G! Test A! Test A, B, C, D! Test! A, B, C, D,! E, F, G! Layer I! Layer I + II! All Layers! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 54!

55 Pros and Cons of Top-down Integration Testing Pro! Test cases can be defined in terms of the functionality of the system (functional requirements)! No drivers needed Cons! Writing stubs is difficult: Stubs must allow all possible conditions to be tested.! Large number of stubs may be required, especially if the lowest level of the system contains many methods.! Some interfaces are not tested separately Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 55!

56 Sandwich Testing Strategy! Combines top-down strategy with bottom-up strategy! The system is viewed as having three layers! A target layer in the middle! A layer above the target! A layer below the target! Testing converges at the target layer Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 56!

57 Sandwich Testing Strategy A! B! C! D! Test A! Test E! Test A,B,C, D! E! F! G! Test F! Test B, E, F! Test! A, B, C, D,! E, F, G! Test D,G! Test G! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 57!

58 Pros and Cons of Sandwich Testing! Top and Bottom Layer Tests can be done in parallel! Problem: Does not test the individual subsystems and their interfaces thoroughly before integration! Solution: Modified sandwich testing strategy 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 58!

59 Modified Sandwich Testing Strategy! Test in parallel:! Middle layer with drivers and stubs! Top layer with stubs! Bottom layer with drivers! Test in parallel:! Top layer accessing middle layer (top layer replaces drivers)! Bottom accessed by middle layer (bottom layer replaces stubs) Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 59!

60 Modified Sandwich Testing A! Test A! Test C! Test A,C! B! C! D! E! F! G! Test B! Test E! Test F! Test B, E, F! Test! A, B, C, D,! E, F, G! Test D! Test G! Test D,G! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 60!

61 Continuous Testing! Continuous build:! Build from day one! Test from day one! Integrate from day one!! System is always runnable! Requires integrated tool support:! Continuous build server! Automated tests with high coverage! Tool supported refactoring! Software configuration management! Issue tracking Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 61!

62 Continuous Testing Strategy A! Spread" SheetView! Layer I! B! Data" Model! C! Calculator! D! Currency! Converter! Layer II! E! F! BinaryFile" Storage! XMLFile" Storage! G! Currency" DataBase! Layer III! Sheet View! + Cells! + Addition! + File Storage! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 62!

63 Steps in Integration Testing 1. Based on the integration strategy, select a component to be tested. Unit test all the classes in.! the component. 2. Put selected component together; do any preliminary fix-up necessary to make the integration test operational (drivers, stubs) 3. Test functional requirements: Define test cases that exercise all uses cases with the selected component 4. Test subsystem decomposition: Define test cases that exercise all dependencies 5. Test non-functional requirements: Execute performance tests 6. Keep records of the test cases and testing activities. 7. Repeat steps 1 to 7 until the full system is tested. The primary goal of integration testing is to identify failures with the (current) component configuration Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 63!

64 System Testing! Functional Testing! Validates functional requirements! Performance Testing! Validates non-functional requirements! Acceptance Testing! Validates clients expectations 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 64!

65 Functional Testing Goal: Test functionality of system! Test cases are designed from the requirements analysis document (better: user manual) and centered around requirements and key functions (use cases)! The system is treated as black box! Unit test cases can be reused, but new test cases have.! to be developed as well Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 65!

66 Performance Testing Goal: Try to violate non-functional requirements! Test how the system behaves when overloaded.! Can bottlenecks be identified? (First candidates for redesign in the next iteration)! Try unusual orders of execution! Call a receive() before send()! Check the system s response to large volumes of data! If the system is supposed to handle 1000 items, try it with 1001 items.! What is the amount of time spent in different use cases?! Are typical cases executed in a timely fashion? 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 66!

67 Types of Performance Testing! Stress Testing! Stress limits of system! Volume testing! Test what happens if large amounts of data are handled! Configuration testing! Test the various software and hardware configurations! Compatibility test! Test backward compatibility with existing systems! Timing testing! Evaluate response times and time to perform a function! Security testing! Try to violate security requirements! Environmental test! Test tolerances for heat, humidity, motion! Quality testing! Test reliability, maintainability & availability! Recovery testing! Test system s response to presence of errors or loss of data! Human factors testing! Test with end users Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 67!

68 Acceptance Testing! Goal: Demonstrate system is ready for operational use! Choice of tests is made by client! Many tests can be taken from integration testing! Acceptance test is performed by the client, not by the developer.! Alpha test:! Client uses the software at the developer s environment.! Software used in a controlled setting, with the developer always ready to fix bugs.! Beta test:! Conducted at client s environment (developer is not present)! Software gets a realistic workout in target environment 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 68!

69 Testing has many activities Establish the test objectives! Design the test cases! Write the test cases! Test the test cases! Execute the tests! Evaluate the test results! Change the system! Do regression testing! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 69!

70 Test Team Analyst! Professional! Tester! Programmer! too familiar! with code! User! Test! Team! System! Designer! Configuration! Management! Specialist! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 70!

71 The 4 Testing Steps 1. Select what has to be tested! Analysis: Completeness of requirements! Design: Cohesion! Implementation: Source code 2. Decide how the testing is done! Review or code inspection! Proofs (Design by Contract)! Black-box, white box,! Select integration testing strategy (big bang, bottom up, top down, sandwich) 3. Develop test cases! A test case is a set of test data or situations that will be used to exercise the unit (class, subsystem, system) being tested or about the attribute being measured 4. Create the test oracle! An oracle contains the predicted results for a set of test cases! The test oracle has to be written down before the actual testing takes place Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 71!

72 Guidance for Test Case Selection! Use analysis knowledge about functional requirements (black-box testing):! Use cases! Expected input data! Invalid input data! Use design knowledge about system structure, algorithms, data structures (white-box testing):! Control structures! Test branches, loops,...! Data structures! Test records fields, arrays,...! Use implementation knowledge about algorithms and datastructures:! Force a division by zero! If the upper bound of an array is 10, then use 11 as index Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 72!

73 Summary! Testing is still a black art, but many rules and heuristics are available! Testing consists of! Unit testing! Integration testing! System testing! Acceptance testing! Design patterns can be used for integration testing! Testing has its own lifecycle 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 73!

74 Additional Reading! Lecture: Prozesse und Methoden beim Testen von Software, Dr. Stefan Kriebel! Andreas Spillner, Tilo Linz, Basiswissen Softwaretest, dpunkt.verlag, 2004! JUnit Website J. Thomas, M. Young, K. Brown, A. Glover, Java Testing Patterns, Wiley, 2004! D. Saff and M. D. Ernst, An experimental evaluation of continuous testing during development Int. Symposium on Software Testing and Analysis, Boston July 12-14, 2004, pp "! A controlled experiment shows that developers using continuous testing were three times more likely to complete the task before the deadline than those without Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 74!

75 Object-Oriented Software Engineering! Using UML, Patterns, and Java! Testing, Part 3 Model-based Testing

76 Outline of the Lectures on Testing "!Terminology "!Types of errors "!Approaches for dealing with errors "!Testing activities "!Unit testing "!Integration testing "!System testing! Model-based Testing 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 76!

77 Generating Tests from a System Model! There are many different ways to "derive" tests from a system model"! Manual generation"! Automatic generation"! Because testing is usually experimental and based on heuristics, there is no one best way to do this. "! It is common to consolidate all test related decisions into a package that is often known as "test requirements or "test package."! This test package can contain e.g. information about the part of the system model that should be the focus for testing, or about the conditions where it is correct to stop testing (test stopping criteria)." 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 77!

78 Model Driven Architecture (MDA)! Recall: MDA focuses on forward engineering, i.e. producing code from a system model! Note: The term architecture in MDA does not mean the architecture of the system being modeled, but refers the architecture of the various standards and models that serve as the technology basis for MDA! UML: Unified Modeling Language! MOF: Meta-Object Facility! XMI: Metadata Interchange! EDOC: Enterprise Distributed Object Computing! SPEM: Software Process Engineering Metamodel! CWM: Common Warehouse Metamodel An important objective of MDA is executable UML Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 78!

79 Executable UML! Executable UML means that given a UML system model the following steps can be performed automatically:! Code generation! Simulation Validation! Test generation ( Model based Testing ) 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 79!

80 Model Based Testing! Definition Model Based Testing: All testing is based on a system model which is used to (manually) generate test cases and oracles! Advantage:! Increased effectiveness of testing! Decreased costs! Reuse of artifacts such as analysis and design models! Requirement:! The system model is formalized (specification model) and described in UML Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 80!

81 Model Driven Testing! Model Driven Testing is a special case of model based testing:! All the test cases are automatically generated from the system model! Advantage:! Automatic generation of tests suites! Better test thoroughness! Reduces the cost of test execution Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 81!

82 Observations on Model Based Testing! A system model is an abstract representation of the system s desired behavior! The test cases derived from this model are functional tests on the same level of abstraction as the model! These test cases are known as the abstract test suite! Because test suites are derived from models - not from source code - model-based testing is a form of black-box testing"! The abstract test suite cannot be executed! An executable test suite must be derived from the abstract test suite which communicates with the system under test (SUT)! This is done by mapping the abstract test suite to concrete test cases suitable for execution.! Mapping is not part of this lecture Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 82!

83 U2TP! UML-2 Testing Profile (U2TP):! A UML extension that makes UML applicable to software testing.! U2TP allows the collection of all information required for the model-based testing process! Deriving a test model from the system model.! Test Model:! A set of test cases derived from the system model Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 83!

84 General Definitions used in U2TP! Test: An attempt to create a difference between the observed behavior and the specified behavior of a system a planned way! Test data: The structures and meaning of values to be processed in a test! System under test (SUT)! A system, subsystem or component! The SUT is treated as black-box. This means:! The SUT can be accessed only via a public interface! No information on the internal structure of the SUT is available for use in the specification of test cases.! Test case: Describes a specified behavior, that is, a behavior specified in the system model (normative behavior) of the SUT Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 84!

85 Test Objective! Test objective! Find any difference between the observed behavior and the specified behavior, that is, the behavior specified in the system model.! UML Example! The test objective in the diagram below specifies that the objective of the ValidWithdrawal test case is to validate the WithdrawMoney use case. <<testcase>>! ValidWithdrawal! <<objective>>! <<usecase>>! WithdrawMoney! 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 85!

86 Test Architecture Definitions! Test component: An object within a test system! Test architecture: The test components and their relationships in a test system! Testing configuration: The combination of a test system with a SUT! Test control: A ordering of the execution of the tests performed on the SUT! Test context: A description of! A set of test cases! A test configuration! A test control Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 86!

87 Example: Test System for a Vending Machine! Vending Machine with 2 use cases:! The user inserts money! A bottle is returned, when the inserted money equals the price of the bottle Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 87!

88 Model-based Testing of MoneyBox Test System! SUT! The test system MoneyUnitTest is developed with the purpose of unit testing the Moneybox, which is part of the system under test MoneyBox must be imported by MoneyUnitTest to be accessible Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 88!

89 UML Model of the MoneyBox Now we need to test that the machine correctly counts the money inserted by the user into the Moneybox -> MoneyUnitTest 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 89!

90 Model-based Testing of MoneyBox Test System! Test Context! SUT! Test Case! MoneyBox is the system under test MoneyUnitTest is the test system MoneyTest is the test context addmoneytest is the test case 2009 Bernd Bruegge!!!! Software Engineering II, Lecture 3: Testing!! 90!