Chapter 14 Software Testing Techniques

Transcription

1 Software Engineering: A Practitioner s s Approach, 6/e Chapter 14 Software Testing Techniques copyright 1996, 2001, 2005 R.S. Pressman & Associates, Inc. For University Use Only May be reproduced ONLY for student use at the university level when used in conjunction with Software Engineering: A Practitioner's Approach. Any other reproduction or use is expressly prohibited. 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Testability Operability it it operates cleanly Observability the the results of each test case are readily observed Controllability the the degree to which testing can be automated and optimized Decomposability testing testing can be targeted Simplicity reduce complex architecture and logic to simplify tests Stability few changes are requested during testing Understandability of the design 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

2 What is a Good Test? A good test has a high probability of finding an error A good test is not redundant. A good test should be best of breed A good test should be neither too simple nor too complex 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Test Case Design "Bugs lurk in corners and congregate at boundaries..." Boris Beizer OBJECTIVE CRITERIA to uncover errors in a complete manner CONSTRAINT with a minimum of effort and time 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

3 Exhaustive Testing loop < 20 X 14 There are 10 possible paths! If we execute one test per millisecond, it would take 3,170 years to test this program!! 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Selective Testing Selected path loop < 20 X 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

4 Software Testing white-box methods black-box methods Methods Strategies 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, White-Box Testing... our goal is to ensure that all statements and conditions have been executed at least once... 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

5 Why Cover? logic errors and incorrect assumptions are inversely proportional to a path's execution probability we often believe that a path is not likely to be executed; in fact, reality is often counter intuitive typographical errors are random; it's likely that untested paths will contain some 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Basis Path Testing First, we compute the cyclomatic complexity: number of simple decisions + 1 or number of enclosed areas + 1 In this case, V(G) = 4 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

6 Cyclomatic Complexity A number of industry studies have indicated that the higher V(G), the higher the probability or errors. modules V(G) modules in this range are more error prone 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Basis Path Testing Next, we derive the independent paths: Since V(G) = 4, there are four paths Path 1: 1,2,3,6,7,8 Path 2: 1,2,3,5,7,8 Path 3: 1,2,4,7,8 Path 4: 1,2,4,7,2,4,...7,8 7 8 Finally, we derive test cases to exercise these paths. 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

7 Basis Path Testing Notes you don't need a flow chart, but the picture will help when you trace program paths count each simple logical test, compound tests count as 2 or more basis path testing should be applied to critical modules 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Graph Matrices A graph matrix is a square matrix whose size (i.e., number of rows and columns) is equal to the number of nodes on a flow graph Each row and column corresponds to an identified node, and matrix entries correspond to connections (an edge) between nodes. By adding a link weight to each matrix entry, the graph matrix can become a powerful tool for evaluating program control structure during testing 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

8 Control Structure Testing Condition testing a test case design method that exercises the logical conditions contained in a program module Data flow testing selects test paths of a program according to the locations of definitions and uses of variables in the program 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Loop Testing Simple loop Nested Loops Concatenated Loops Unstructured Loops 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

9 Loop Testing: Simple Loops Minimum conditions Simple Loops 1. skip the loop entirely 2. only one pass through the loop 3. two passes through the loop 4. m passes through the loop m < n 5. (n-1), n, and (n+1) passes through the loop where n is the maximum number of allowable passes 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Loop Testing: Nested Loops Nested Loops Start at the innermost loop. Set all outer loops to their minimum iteration parameter values. Test the min+1, typical, max-1 and max for the innermost loop, while holding the outer loops at their minimum values. Move out one loop and set it up as in step 2, holding all other loops at typical values. Continue this step until the outermost loop has been tested. Concatenated Loops If the loops are independent of one another then treat each as a simple loop else* treat as nested loops endif* for example, the final loop counter value of loop 1 is used to initialize loop 2. 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

10 Black-Box Testing requirements output input events 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Black-Box Testing How is functional validity tested? How is system behavior and performance tested? What classes of input will make good test cases? Is the system particularly sensitive to certain input values? How are the boundaries of a data class isolated? What data rates and data volume can the system tolerate? What effect will specific combinations of data have on system operation? 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

11 Graph-Based Methods To understand the objects that are modeled in software and the relationships that connect these objects object #1 Undirected link Directed link (link weight) object #3 (a) Parallel links object #2 Node weight (value) In this context, we consider the term objects in the broadest possible context. It encompasses data objects, traditional components (modules), and object- oriented elements of computer software. new file is represented as menu select generates (generation time < 1.0 sec) allows editing of document text (b) contains document window Attributes: background color: white text color: default color or preferences 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Equivalence Partitioning user queries mouse picks output formats prompts FK input data 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

12 Sample Equivalence Classes Valid data user supplied commands responses to system prompts file names computational data physical parameters bounding values initiation values output data formatting responses to error messages graphical data (e.g., mouse picks) Invalid data data outside bounds of the program physically impossible data proper value supplied in wrong place 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Boundary Value Analysis user queries mouse picks output formats prompts FK input data input domain output domain 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

13 Comparison Testing Used only in situations in which the reliability of software is absolutely critical (e.g., human-rated systems) Separate software engineering teams develop independent versions of an application using the same specification Each version can be tested with the same test data to ensure that all provide identical output Then all versions are executed in parallel with real-time comparison of results to ensure consistency 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Orthogonal Array Testing Used when the number of input parameters is small and the values that each of the parameters may take are clearly bounded Z Z Y X One input item at a time Y X L9 orthogonal array 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

14 Object-Oriented Testing 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, OOT Test Test Case Design Berard [BER93] proposes the following approach: 1. Each test case should be uniquely identified and should be explicitly associated with the class to be tested, 2. The purpose of the test should be stated, 3. A list of testing steps should be developed for each test and should contain [BER94]: a. a list of specified states for the object that is to be tested b. a list of messages and operations that will be exercised as a consequence of the test c. a list of exceptions that may occur as the object is tested d. a list of external conditions (i.e., changes in the environment external to the software that must exist in order to properly conduct the test) e. supplementary information that will aid in understanding or implementing the test. 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

15 Testing Methods Fault-based testing The tester looks for plausible faults (i.e., aspects of the implementation of the system that may result in defects). To determine whether these faults exist, test cases are designed to exercise the design or code. Class Testing and the Class Hierarchy Inheritance does not obviate the need for thorough testing of all derived classes. In fact, it can actually complicate the testing process. Scenario-Based Test Design Scenario-based testing concentrates on what the user does, not what the product does. This means capturing the tasks (via use-cases) that the user has to perform, then applying them and their variants as tests. 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, OOT Methods: Random Testing Random testing identify operations applicable to a class define constraints on their use identify a minimum test sequence an operation sequence that defines the minimum life history of the class (object) generate a variety of random (but valid) test sequences exercise other (more complex) class instance life histories 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

16 OOT Methods: Partition Testing Partition Testing reduces the number of test cases required to test a class in much the same way as equivalence partitioning for conventional software state-based partitioning categorize and test operations based on their ability to change the state of a class attribute-based partitioning categorize and test operations based on the attributes that they use category-based partitioning categorize and test operations based on the generic function each performs 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, OOT Methods: Inter-Class Testing Inter-class testing For each client class, use the list of class operators to generate a series of random test sequences. The operators will send messages to other server classes. For each message that is generated, determine the collaborator class and the corresponding operator in the server object. For each operator in the server object (that has been invoked by messages sent from the client object), determine the messages that it transmits. For each of the messages, determine the next level of operators that are invoked and incorporate these into the test sequence 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

17 OOT Methods: Behavior Testing The tests to be designed should achieve all state coverage [KIR94]. That is, the operation sequences should cause the Account class to make transition through all allowable states open empty acct dead acct setup Accnt balance credit accntinfo close set up acct deposit (initial) working acct withdrawal (final) nonworking acct deposit withdraw Figure 14.3 State diagram for Account class (adapted from [KIR94]) 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001, Testing Patterns Pattern name: pair testing Abstract: A process-oriented pattern, pair testing describes a technique that is analogous to pair programming (Chapter 4) in which two testers work together to design and execute a series of tests that can be applied to unit, integration or validation testing activities. Pattern name: separate test interface Abstract: There is a need to test every class in an object-oriented system, including internal classes (i.e., classes that do not expose any interface outside of the component that used them). The separate test interface pattern describes how to create a a test interface that can be used to describe specific tests on classes that are visible only internally to a component. [LAN01] Pattern name: scenario testing Abstract: Once unit and integration tests have been conducted, there is a need to determine whether the software will perform in a manner that satisfies users. The scenario testing pattern describes a technique for exercising the software from the user s s point of view. A failure at this level indicates that the software has failed to meet a user visible requirement. [KAN01] 6/e and are provided with permission by R.S. Pressman & Associates, Inc., copyright 1996, 2001,

18 Åben Uddannelse Test i XP Automatisk test Software testing Testing is the process of executing a program with the intent of finding errors. Glen Myers Test i XP 2 Software Engineering, Efteråret

19 Åben Uddannelse Testing You couldn t possibly write all those tests? It would take too much time. Programmers won t write tests! Unless The design is as simple as it can be You are programming with a partner, so if you couldn t think of another test your partner can You feel good when you see the tests all running Your customer feels good about the system when they see all of their tests running Test i XP 3 Test-First Development Tests are specified before software is written Users specify inputs and outputs as part of test Outside-in perspective improves the software design Ensures software delivers expected results Writing tests is part of s/w design process Detailed test knowledge captured in tests Functional Tests specify requirements in detail Unit Tests developed as soon as component/class identified; defines software interface Tests drive the development process Software is only written because tests are failing. Ensures all software is tested Ensures all requirements are captured as tests Test i XP 4 Software Engineering, Efteråret

20 Åben Uddannelse Timelines Acceptance need to be available in the same iteration as the story is scheduled The only way to know that the stories are done is that the acceptance tests are passed Programmers have the right to know what is needed Therefore insist on automated functional tests Customers have the right to see progress in a running system Proven to to work by automated tests that you specify Test i XP 5 Functional Tests Specified by the user Implemented by users, developers, and/or test team Automated Run at least daily Part of the specification Test i XP 6 Software Engineering, Efteråret

21 Åben Uddannelse Test Metrics Key: Tests failed Tests passed Test i XP 7 Unit Tests Testing bit by bit Tests made by the programmers for the programmers Test i XP 8 Software Engineering, Efteråret

22 Åben Uddannelse Unit Tests A unit test is written BEFORE THE CODE IS WRITTEN for each piece of the system. When you create your tests first, before the code, you will find it much easier and faster to create your code. The combined time it takes to create a unit test and create some code to make it pass is about the same as just coding it up straight away. But, if you already have the unit tests you don't need to create them after the code saving you some time now and lots later. Eases regression testing JUnit (for Java), SUnit (for Squeak) allow for running all unit tests. When a bug is found, a new unit test is written to catch for that bug in the future Test i XP 9 Unit Tests Written by developers Always run at 100% Support design, coding, refactoring, and quality. Test i XP 10 Software Engineering, Efteråret

23 Åben Uddannelse Testing steps Create a test class Name it like ClassNameTestCase Code a setup Write a test Run the test Modify the model code and repeat steps four and five until the test runs successfully Release the code and the test case Test i XP 11 Automated Testing Continuous Automated Testing throughout system lifecycle during development & integration before delivery, after installation Tests are rerun many times every day Low cost regression testing since most tests are automated Higher initial cost recouped before initial delivery; free thereafter one button testing, anytime, anywhere Improves Developer Morale Developers enjoy testing, because it is coding Removes developers fear of breaking something Reduces maintenance cost significantly 12 Software Engineering, Efteråret

24 Åben Uddannelse What is JUnit? A small testing framework written in Java. A series of extensible classes that do much of the testing grunt work for us. (i.e. counting and reporting errors and failed tests, running tests in batch, etc.) Very handy for Extreme Testing Developed by Kent Beck, kentbeck@compuserve.com Erich Gamma, erich_gamma@acm.org Test i XP 13 Essential Classes junit.framework.testcase Allows running multiple tests Does all counting and reporting of errors junit.framework.assert A set of assert methods Test fails if the assert condition does not hold If a test fails it is counted and reported junit.framework.testsuite A collection of tests Uses Java introspection to find all the methods start with "test and have void parameters. TestSuite's run method executes all the tests Test i XP 14 Software Engineering, Efteråret

25 Åben Uddannelse Test Runners Classes that run tests for a test case Can be Text or UI based Text Based: Accepts the name of a TestCase class as an argument If this class defines a static suite method it will be invoked and the returned test is run. Otherwise all the methods starting with "test" having no arguments are run. AWT UI Based: A handy tool for running test cases Runs test cases and reports failures and errors Test i XP 15 Class diagram Test i XP 16 Software Engineering, Efteråret

26 Åben Uddannelse JUnit How-To (1) Define a test case (of course) Create a Class which extends TestClass Define the fixture for the test class, these are all variables required to carry out the tests Override the setup and teardown methods of TestCase to initialize members or free up resources, respectively Test i XP 17 JUnit How-To (2) Define the test methods required for the test case. Additional methods may be added at any time Define a suite method which creates a TestSuite from the methods of the test class Define a main method which uses some test runner to run the tests in the TestSuite Simply run the test case class Test i XP 18 Software Engineering, Efteråret

27 Åben Uddannelse JUnit How-To (3) Multiple test cases my be run by creating a class which contains a suite of TestCase subclasses These suites can be run in batch or individually by simply running the whole class containing the suite or by calling run with the name of the test case you want to run. Each test case constructor can accept a string which is the name of the test case Test i XP 19 Junit User Interface Test i XP 20 Software Engineering, Efteråret

28 Åben Uddannelse A Quick Example (1) Class Person Person contains a first and a last name Also contains the method hashcode methods for computing hashcodes for first and last public class Person { public Person() { first = ""; last = ""; }... public int hashcode() { int result = 0; result += first.hashcode(); result += last.hashcode(); return result; }... } Test i XP 21 A Quick Example (2) Class PersonTest extends the Junit class TestCase The fields first and last make up the fixture PersonTest contains a test which test if the hashcode method returns the proper values The method suite is inherited from TestCase Test i XP 22 Software Engineering, Efteråret

29 Åben Uddannelse A Quick Example (3) The PersonTest Constructor public PersonTest(String name) { super(name); } Accepts the name of the test case and passes it to the super class Test i XP 23 A Quick Example (4) SetUp of one PersonTest private static final Person personthree = new Person("First3", "Last3"); Test i XP 24 Software Engineering, Efteråret

30 Åben Uddannelse A Quick Example (5) The PersonTest test method testhashcode public void testhashcode() { int code = personthree.hashcode(); Person test = new Person("First3","Last3"); assertequals(code,test.hashcode()); test = new Person( "First1", "Last1"); asserttrue((code == test.hashcode()) == false); } Test i XP 25 A Quick Example (6) The PersonTest suite method public static Test suite ( ) { return new TestSuite (PersonTest.class)); } Creates a new TestSuite containing the tests defined in the PersonTest class Test i XP 26 Software Engineering, Efteråret

31 Åben Uddannelse A Quick Example (7) Run multiple test suites (class AllTests) Define new suite - TestSuite Add suites to TestSuite public static Test suite ( ) { TestSuite suite= new TestSuite("All JUnit Tests"); suite.addtest(new TestSuite(PersonTest.class)); suite.addtest(new TestSuite(LinkedListTest.class)); return suite; } Test i XP 27 A Quick Example (8) The public class AllTests public static void main (String[] args) { /** junit.textui.testrunner.run (suite());*/ junit.swingui.testrunner.run (AllTests.class); } Creates a new TestSuite that runs all the tests defined in the PersonTest and LinkedListTest classes Test i XP 28 Software Engineering, Efteråret