What is Testing? Table of Contents

Transcription

1 Testing Table of Contents Testing - From Hating to... Fishing What is Testing? Validation. Verification. Testing The Testing Process Testing vs. Debugging Testing Techniques Black-box Testing Glass-box Testing Testing Data Abstractions Object Integration Testing XUnit - The Testing Framework Writing a "Hello World" Test Case Running the Tests Adding More Tests Suites and Fixtures Homework Testing - From Hating to... Fishing We will start this tutorial with an excellent apology for testing, by A.Hunt and D.Thomas from their book "The Pragmatic Programmer": Most developers hate testing. They tend to test gently, subconsciously knowing where the code will break and avoiding the weak spots. Pragmatic Programmers are different. We are driven to find our bugs now so we don't have to endure the shame of others finding our bugs later. Finding bugs is somewhat like fishing with a net. We use fine, small nets (unit testing) to catch the minnows, and big, coarse nets (integration tests) to catch the killer sharks. Sometimes the fish manage to escape, so we patch any holes that we find, in hope of catching more and more slippery defects that are swimming about in our project pool. Therefore... What is Testing? One of the key issues in developing software is ascertaining whether or not a program works as we hope it will and discovering why not when it does not. Most of us have an image of what testing us, but unfortunately this image is either distorted or incomplete. Therefore, the goal of this section is to put some order among the key concepts in testing, so that we make sure we use a consistent vocabulary when talking about testing. In this context we will briefly discuss the testing process and its phases. Validation. Verification. Testing Following definitions are proposed by B.Liskov in her book Program Development in Java: Validation is a process designed to increase our confidence that a program works as intended. Validation can be done through verification or testing. Verification is a formal or informal argument that a program works on all possible inputs. Testing is the process of running on a set of test cases and comparing the actual results with expected results. In the rest of this tutorial we will focus exclusively on testing, as a form of validation. The Testing Process The Five Stages of Testing The figure below shows a five-stage testing process where system components are tested, the integrated system is tested and, finally, the system is tested with the customer's data.

2 Ideally, component defects are discovered early in the process and interface problems when the system is integrated. However, as defects are detected the program must be debugged and this may require other stages in the testing process to be repeated. The process is therefore an iterative one with information being fed from later stages to earlier parts of the process. Unit Testing. Individual components are tested to assure that they operate correctly. Each compoenent is tested independently, without other system components. We say that the component is tested in isolation. Module Testing. A module is a collection of dependent components such as an abstract data type, a class, or some looser collection of procedures and functions. A module encapsulates related components, so it can be tested without other system modules. Sub-system Testing This phase involves testing collections of modules which have been integrated into sub-systems. The most common problems which arise in large software systems are interface mismatches. The sub-system test process should therefore concentrate on the detection of module interface errors by rigorously exercising the interfaces. System Testing. This process is concerned with finding errors that result from unanticipated interactions between sub-systems and sub-system interface problems. System Testing is also used for validating that the system meets its functional and non-functional requirements. Acceptance Testing. This is the final testing stage in which the system is tested with data supplied the system customer rather than simulated test data. It may reveal errors and omissions in the system requirements definition because the real data exercise the system in different ways from the test data. Acceptance testing is sometimes called alpha testing. The Four Phases of Testing The goal of defect testing is to expose latent defects in a software system before the system is delivered. This contrasts with validation testing which is intended to demonstrate that a system meets its specification. Validation testing requires the system to perform correctly using given acceptance tests. A succesful defect testing is a test which causes the system to perform incorectly and hence exposes a defect. Testing demonstrates the presence, not the absence, of program faults!! In the figure below you see the phases of a defect testing process. Test cases are specifications of the inputs to the test and the expected output from the system, plus a statement of what is being tested. Test Data are the inputs that are devised to test the system. Test data can be sometimes generated automatically. Exhaustive testing is impractical. Therefore, testing has to be based on a subset of possible test cases and in order choose these test cases each company has to develop testing policies. These policies might be base on general testing policies, such as: "all program statements should be executed at least once". Alternatively, they may be based

3 on experience of system usage and may focus on testing the features of the operational system. For example: All system functions that are accessed through menus should be tested. Combinations of functions that are accessed through same menu must be tested. Where user input is provided, all functions must be tested both for correct and incorrect input. Testing vs. Debugging Testing tells us that something is wrong with a program, but knowing the symptom is still far from knowing its cause. Debugging is the process of understanding and correcting errors. When debugging, we try to narrow the scope of the problem by looking for simple test cases that manifest the bug and by looking at intermediate values to locate the responsible region in the code. As we collect evidence about the bug, we formulate hypotheses and attempt to refute them by running further tests. When we think we understand the cause of the bug, we study the appropriate region of code to find and correct the error. Testing Techniques Black-box Testing Black-box testing is an approach to testing where the test are derived from the program or component specification. The tester presents inputs to the component or the system and examines the corresponding outputs. Equivelance partitioning The key problem for the tester is to select the inputs that have a high probability of exposing erroneous values in the output set. The input data to a program usually fall into a number of different classes, that have common characteristics. We call them partitions or domains. Once you have identified a set of partitions, you then choose test cases from each of these partitions. Use the following guidelines: Consider also invalid inputs as partitions. Choose test-cases on the boundaries of the partitions plus cases close to the mid-point of the partition. If you test programs that work on collections (sequences) of elements, consider the following guidelines: Test software with sequences that have no values and sequences with only a single value. Use different sequences of different sizes in different tests. Derive tests so that the first, middle and last element of the sequence are accessed. Glass-box Testing The objective of glass-box testing is to assure that each independent program path is executed at least once. An independent path is one that traverses at least one new edge in the flow graph. Both true and false branches of all conditions must be tested! Here are a summary of the major guidelines for glass-box testing: Glass-box tests should supplement the black-box tests so that the tests are path-complete: every path in the code is exercised by at least one test. For loops with a fixed amount of iteration, tests should include one and two iterations of the loop. For loops with a variable amount of iteration, tests should include zero, one and two iterations of the loop. For recursion, the tests should include no recursion and one recursive call. Testing Data Abstractions When testing classes, complete test coverage should include: the testing in isolation of all operations associated with the object. the setting and interogation of all attributes associated with the object. the exercise of the object in all possible states. This means that all events that cause a state change in the object should be simulated. If inheritance is used, this makes it more difficult to design tests for classes. Where a superclass provides operations that are inherited by a number of subclasses, all of these subclasses should be tested with all inherited operations. The reason for this is that the inherited operation may make assumptions about other operations that have been changed when inherited. Equally, when a superclass operation is overriden, the overwriting operation must be tested. Object Integration Testing When object-oriented systems are developed, the levels of integration are less distinct. Because of that, there is no direct equivalency to module testing in object-oriented systems. However, some authors suggest that groups of classes which act in combination to provide a set of services should be tested together. They call it cluster testing. The most effective approach to integration (or cluster) testing is the use-case or scenario-based testing, because use-cases describe one mode of using the system. Testing can be based on these scenario descriptions and object clusters created to support the use-cases that relate to that mode of use. XUnit - The Testing Framework When we talk about testing we cannot pass by the XUnit testing framework. Well now, please don't panic! Things will get clearer as you'll read forward. JUnit is the XUnit version for Java and was the first testing framework created by Kent Beck and Erich Gamma. For those of you who are "addicted" to C++, there is also some good news: JUnit was ported also for C++ and it can be even integrated with the MS Development Studio. You can also download CppUnit from our site. Writing a "Hello World" Test Case "Bfffaagh, what can you possibly test at a "Hello World" program!?". Hmmm, let's see. First this section is not intended to be a comprehensive Cookbook. If you find yourself too valuable to follow the steps presented here you can find the original Cookbook in the junit3.8.1.zip. For the rest of you who continued to stay with us, we would like to introduce the HelloClass: package hello; public class HelloClass { public String sayhello() { return "Hello World"; package test.hello; import hello.*;

4 public class TestHelloClass extends TestCase { public TestHelloClass(String s) { super(s); //this is mandatory public void testsayhello() { assertequals(helloobject.sayhello(), "Hello World"); Running the Tests Now, you have the HelloClass and the associated TestClass. All you have to do is to run it. There are three possiblities to do that. 1. The first possibility is to make use of the JUnit text oriented interface and run from command line. The output is as follows: 2. C:\new\tmp>java junit.textui.testrunner test.hello.testhelloclass Time: OK (1 tests) Another possibility is to write a main method. In this case, the code you have to add would look like: 9. package test; import test.hello.*; public class AllTests { 15. public static void main(string[] args) { 16. junit.textui.testrunner.run(testhelloclass.class); Of course, we will not cover the starting of a Java program in this section. 21. Of course, there also is a nice Swing user interface. You can take advantage of the beautiful landscape by typing: java junit.swingui.testrunner (of course don't forget to press <ENTER> :-)). The beauty of the friendly interface will rapidly come and gently touch your eyes. We do not want to spoil the surprise with a static picture, so please try it yourself. Caution Did you, by chance, get the following output? Exception in thread "main" java.lang.noclassdeffounderror: junit/textui/testrunner If you did then you probably forgot to place the junit.jar into the CLASSPATH. Adding More Tests The next step of our little Cookbook starts with the wish of Gigi who just told you to make the HelloClass to support multiple languages. How do you do that? The first solution that strikes you is to have a HashMap with languages as keys. So, the HelloClass would look like: package hello; import java.util.*; public class HelloClass { private Map hellomap = null; public HelloClass() { hellomap = new HashMap(); hellomap.put("english", "Hello World"); hellomap.put("romana", "Buna ziua lume"); hellomap.put("deutsch", "Aloha"); //You just made a mistake :). Try reading a German Conversational Guide public String sayhello() { return "Hello World"; public String sayhello(string language) { if (hellomap.containskey(language) == true) return (String)helloMap.get(language); else return ""; //you know its a String so you downcast Of course, you want to test this class. If you haven't already done that (even before you wrote the HelloClass!) you should write some tests. So, The TestHelloClass would look like: package test.hello; import hello.*; public class TestHelloClass extends TestCase { public TestHelloClass(String s) { super(s);

5 public void testsayhello() { assertequals(helloobject.sayhello(), "Hello World"); public void testsayhelloinromanian() { assertequals(helloobject.sayhello("romana"), "Buna ziua lume"); public void testsayhelloingerman() { assertequals(helloobject.sayhello("deutsch"), "Hallo Welt"); Good. Now we have the code and the associated tests. So we run again the tests: C:\new\tmp>java test.alltests...f Time: 0.02 There was 1 failure: 1) testsayhelloingerman(test.hello.testhelloclass)junit.framework.assertionfailederror: expected:<aloha> but was:<hallo Welt> at test.hello.testhelloclass.testsayhelloingerman(testhelloclass.java:23) at sun.reflect.nativemethodaccessorimpl.invoke0(native Method) at sun.reflect.nativemethodaccessorimpl.invoke(unknown Source) at sun.reflect.inflatablemethodaccessorimpl.invoke(unknown Source) at test.alltests.main(alltests.java:8) FAILURES!!! Tests run: 3, Failures: 1, Errors: 0 Oops! One test just failed on your code. What test exactly? Obvoiusly the "testsayhelloingerman". We will let the debugging pleasure to you. You should stop debugging only when all the tests are running. Suites and Fixtures One thing you should notice is that in every test method you have a common code line: As we don't like you to repeat yourself we would like to put it in one place (a method for example) and use it many times. That is what the fixtures are for. From this perspective the new TestHelloClass would look like: package test.hello; import hello.*; public class TestHelloClass extends TestCase { private HelloClass helloobject; //the fixture public TestHelloClass(String s) { super(s); //this is the method that initializes the fixtures public void setup() { helloobject = new HelloClass(); public void testsayhello() { assertequals(helloobject.sayhello(), "Hello World"); public void testsayhelloinromanian() { assertequals(helloobject.sayhello("romana"), "Buna ziua lume"); public void testsayhelloingerman() { assertequals(helloobject.sayhello("deutsch"), "Hallo Welt"); Now, supose you haven't found out why the "testsayhelloingerman" failed and the boss is on your back. In that case you would like to appear cool and would like not to run the "testsayhelloingerman" test. You have the Suite in your aid. So you modify the AllTests class: package test;

6 import test.hello.*; public class AllTests { public static void main(string[] args) { junit.textui.testrunner.run(suite()); //the method that creates the wanted suite public static Test suite() { TestSuite theworkingsuite = new TestSuite("Non German Languages Tests"); theworkingsuite.addtest(new TestHelloClass("testSayHello")); theworkingsuite.addtest(new TestHelloClass("testSayHelloInRomanian")); return theworkingsuite; Your boss leaves happy as the screen shows: C:\new\tmp>java test.alltests.. Time: 0.01 OK (2 tests) Caution Even if you passed your boss' vigilence, keep in mind another golden rule from the "Pragmatic Programmer":