Jtest Tutorial. Tutorial

Transcription

1 Jtest Jtest Welcome to the Jtest. This tutorial walks you through how to perform common Jtest tasks using example files. Please note that although the four types of tests (static analysis, white-box testing, black-box testing, and regression testing) are discussed separately, Jtest can perform all of these tests with just one click of the Start button. This tutorial contains the following lessons: Lesson 1 - Performing White-Box Testing on page 2 Lesson 2 - Performing Static Analysis on page 13 Lesson 3 - Performing Black-Box Testing on page 23 Lesson 4 - Performing Regression Testing on page 51 Lesson 5 - Testing a Set of Classes on page 54 Lesson 6 - Using User-Defined Stubs on page 59 Lesson 7- Using JUnit Test Classes with Jtest on page 69 Lesson 8- Using Jtest in a Multi-User Environment on page 79 1

2 Lesson 1 - Performing White-Box Testing Lesson 1 - Performing White-Box Testing White-box testing checks that the class is structurally sound. It doesn t test that the class behaves according to the specification, but instead ensures that the class doesn t crash and that it behaves correctly when passed unexpected input. White-box testing involves looking at the class code and trying to find out if there are any possible class usages that will make the class crash (in Java this is equivalent to throwing an uncaught runtime exception). Jtest completely automates white-box testing by automatically generating and executing test cases designed to fully test the class. During the test, Jtest executes the class using a symbolic virtual machine, then searches for uncaught runtime exceptions. For each uncaught runtime exception that is detected, Jtest reports an error and provides the stack trace and the calling sequence that led to the problem. This lesson contains the following sections: Performing White-Box Testing: Overview on page 2 Performing White-Box Testing: Example on page 3 Viewing Automatically Generated Test Cases on page 6 Suppressing Exceptions on page 7 Viewing a Text or HTML Format Report on page 7 Setting an Object to a Certain State on page 8 Performing White-Box Testing: Overview To perform white-box testing, just tell Jtest what class or set of classes to test then click the Start button. If you test a project, results will be displayed in the Class Name> Errors Found> Uncaught Runtime Exceptions branch of the Project Testing UI's Results panel. 2

3 Lesson 1 - Performing White-Box Testing If you test a single class, results will be displayed in the Uncaught Runtime Exceptions branch of the Class Testing UI s Errors Found panel. Performing White-Box Testing: Example 1. Go to Jtest s Class Testing UI. (This UI opens by default when you launch Jtest). 2. If a class is already loaded into the Class Testing UI (i.e., if you see a class name in the Class Name field), click the New button to clear the previous test. 3. Browse to Simple.class (in <jtest_install_dir>/examples/eval) using the Browse button in the Class Name panel. 4. Click the Start button in the tool bar. Jtest will perform static and dynamic analysis on the class. A dialog box will open to notify you when testing is complete. Information on test progress will be displayed in the Test Progress panel. Problems found will be reported in the Errors Found panel. 3

4 Lesson 1 - Performing White-Box Testing You will see two types of errors reported in the Errors Found panel. For this example, we re going to focus on the uncaught runtime exception found. The Errors Found panel will list the following uncaught runtime exception: This error message reveals that there is some input for which the class will throw an uncaught runtime exception at runtime. This could cause the application running this class to crash. To see a stack trace like the one the Java virtual machine would give if this uncaught runtime exception were thrown, expand this branch. 4

5 Lesson 1 - Performing White-Box Testing To see an example usage of this class that would lead to the reported uncaught runtime exception, expand the Test Case Input branch. The error shows us that the "startswith" method is implemented incorrectly. The method should return false for the argument "" and "0" instead of throwing a runtime exception. If the error is not fixed, any application using this class will eventually crash or give incorrect results. To view the source code of the class (with the problematic line of the stack trace highlighted), double-click the node containing the exception s file/line information. 5

6 Lesson 1 - Performing White-Box Testing Viewing Automatically Generated Test Cases To see a selection of the test cases that Jtest automatically generated for this test, click the View button in the tool bar. The View Test Cases window will open. In the View Test Cases window, expand the Automatic Test Cases branch. This will display a list of this class s methods. Click any method with a number greater than zero by it, and open the Test Cases branch, or press Control and click your right mouse button, then choose Expand Children from the shortcut menu that opens. The inputs that Jtest generated for each method are now displayed. 6

7 Lesson 1 - Performing White-Box Testing Suppressing Exceptions You can tell Jtest to suppress uncaught runtime exceptions that you do not expect to occur or that you are not concerned with by adding Design by Contract tags to your code. To have Jtest suppress errors for inputs that you do not expect to occur, use tag to specify what inputs are permissible. To have Jtest suppress expected exceptions, use tag to specify what exceptions you want Jtest to ignore. For example, if you wanted to suppress reports of an expected Negative- ArraySizeException that occurs when a negative index is used as an index to an array, you might enter the following comment above the appropriate method: java.lang.negativearraysizeexception */ This not only tells Jtest to ignore the exception, but it also makes code easier to understand and maintain. If you use tag to indicate valid method inputs, and then use ParaSoft s Jcontract to check Design by Contract contracts at runtime, you will automatically be alerted to instances where the system passes this method any unexpected inputs. Viewing a Text or HTML Format Report Jtest automatically creates a report for each test. You can view this report by clicking the Report button. By default, this report is formatted in text (ASCII) format. If you would like Jtest to generate HTML reports (e.g., if you want to post the report on your development intranet), choose Prefer- 7

8 Lesson 1 - Performing White-Box Testing ences> Configuration Options> Report Format> HTML before you click the Report button.. By default, the report file will be saved in <jtest_install_dir>/u/username/results. To prompt Jtest to save reports to a different location, modify the Common Parameters> Directories> Results parameter at the Global, Class, or Project level. Setting an Object to a Certain State In some cases, you may want to set up an initial state prior to testing a class. For example, suppose that a class is used as a global object con- 8

9 Lesson 1 - Performing White-Box Testing taining static member variables accessible by any other project within the application. When Jtest tests an object that uses this static member variable, a NullPointerException will result because the variable has not been set. This problem can be solved by giving Jtest initialization code. For example, the file ExecGlobal.java (in <tutorial_install_dir>/lesson1/execglobal.java) contains a static variable that is assigned by its ancestor Exec, which is defined in the Exec.java file (also in <tutorial_install_dir>/lesson1/exec.java). In this example, the instantiation on the Exec object is performed by the Exec object s "main" method, which is defined in the Exec.java file. When Jtest tests the ExecTest object s dotest method, it does not know that a certain state is assumed by the dotest method. Thus, initialization needs to be performed. To see why initialization is necessary: 1. Go to Jtest s Class Testing UI. (This UI opens by default when you launch Jtest). 2. If a class is already loaded into the Class Testing UI (i.e., if you see a class name in the Class Name field), click the New button to clear the previous test. 3. Use the Browse button in the Class Name panel to choose ExecTest.class (in <tutorial_install_dir>/lesson1/exectest.class) 4. Test the class by clicking the Start button in the Class Testing UI tool bar. A NullPointerException is reported because the assumed state is not available. You will need to perform static initialization on the ExecTest class to ensure the objects referenced by this class are in the assumed 9

10 Lesson 1 - Performing White-Box Testing state. To perform the necessary initialization: 1. Click the Class button to open the Class Test Parameters window. 2. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> Common. 3. Double-click the Static Class Initialization node. 10

11 Lesson 1 - Performing White-Box Testing The Static Class Initialization window will open. 4. Enter the following initialization code in the Static Class Initialization window. new Exec (); This will create an instance of the "Exec" object before the class "ExecTest" is tested. 11

12 Lesson 1 - Performing White-Box Testing 5. (Optional) If you want to check the method, choose Save & Check from the Options menu. 6. Choose Options> Save. 7. Choose Options> Quit. 8. Click Start in the Class Testing UI tool bar to retest the class. You no longer receive a NullPointerException because an instance of Exec is available when testing the ExecTest object. 12

13 Lesson 2 - Performing Static Analysis Lesson 2 - Performing Static Analysis During static analysis analyzes source code to expose violations of coding standards, language specific rules that help you prevent errors. Jtest enforces the following types of coding standards: Traditional coding standards: Traditional coding standards are rules that apply to constructs within the class under test. A traditional coding standard might test whether or not a file s source code contains a construct that has a high probability of resulting in an error. For example, one traditional coding standard checks that you use equals instead of == when comparing strings (writing == when you should have used equals causes the program to check if two strings are identical, rather than check if two strings have the same value). Global coding standards: Global coding standards are rules that ensure that projects use fields, methods, and classes wisely. A global coding standard might check that a project does not contain logical flaws and unclear code such as unused or overly-accessible fields, methods, and classes; such problems increase the probability of an error being introduced into the code, and may also make the code less object-oriented. Metrics: Metrics are measures of the size and complexity of code. When Jtest performs static analysis, it also measures your class s and (if applicable) project s metrics; it reports all metrics in the Metrics window, and reports a static analysis violation for any class metric that is outside of the bounds that have been set for that metric. Custom coding standards: Custom coding standards are rules specially tailored to your own or your group s unique coding style. You can create such customized rules using Jtest s RuleWizard feature. Static analysis is performed automatically when you test a class or project. 13

14 Lesson 2 - Performing Static Analysis When enforcing all static analysis rules except for global static analysis rules, Jtest statically analyzes the classes by parsing the.java source and applying a set of rules to it, then alerting you to any rule violations found. When enforcing global static analysis rules, Jtest scans all of the project s.class files to collect global usage information, uses this information to check if each class violates any rules, then alerts you to any violations in the classes under test. Because Jtest uses.class files (rather than.java files) to enforce global static analysis rules, you can perform global static analysis even when.java files are not available. Note: Global static analysis can only be performed while testing a project. This lesson contains the following sections: Performing Static Analysis: Overview on page 14 Performing Static Analysis: Example on page 14 Viewing Metrics on page 17 Customizing Static Analysis on page 18 Performing Static Analysis: Overview To perform static analysis, just Jtest what class or set of classes to test then click the Start button. If you test a project, results will be displayed in the Class Name> Errors Found> Static Analysis branch of the Project Testing UI's Results panel. If you test a single class, results will be displayed in the Static Analysis branch of the Class Testing UI s Errors Found panel. Important: Because Jtest s static analysis tests at the source level, Jtest will only perform static analysis on classes whose.java files are available. Performing Static Analysis: Example 1. Go to Jtest s Class Testing UI. (This UI opens by default when you launch Jtest). 14

15 Lesson 2 - Performing Static Analysis 2. If a class is already loaded into the Class Testing UI (i.e., if you see a class name in the Class Name field), click the New button to clear the previous test. 3. Browse to Simple.class (in <jtest_install_dir>/examples/eval) using the Browse button in the Class Name panel. 4. Click the Start button in the tool bar. Jtest will perform static and dynamic analysis on the class. Information on test progress will be displayed in the Test Progress panel. Errors found will be reported in the Errors Found panel. You will see two types of errors reported in the Errors Found panel. For this example, we re going to focus on the static analysis violations. The Static Analysis branch of the Errors Found panel should list the following violations: 15

16 Lesson 2 - Performing Static Analysis To see more information about a violation, expand the violation s branch. For example, expand the violation of the PB.TLS rule. This violation message reveals that the developer inadvertently wrote "case10" instead of "case 10". If the class is not fixed, it will give incorrect results when it is passed the value 10. To learn more about any rule that Jtest enforces, right-click the topmost branch of the error violation that you would like explained, then choose View Rule Description from the shortcut menu. A dialog box that contains a description and example of the selected rule violation, as well as a suggested repair, will open. You can also view the source code responsible for the violation; to do this, simply double-click the node containing the violation's file/line information, then view the code in the source code viewer that opens. 16

17 Lesson 2 - Performing Static Analysis Viewing Metrics To view this class s metrics, click the Metrics button. A metrics window containing this class s metrics will then open. 17

18 Lesson 2 - Performing Static Analysis To view the description of a certain metric, right-click that metric and choose View Rule Description from the shortcut menu. Customizing Static Analysis You can change which rules Jtest enforces by enabling or disabling specific rules, or entire rule categories. Note: You can find a description of each rule, as well as an example violation and suggested repair, in the Reference section of the Jtest User s Guide. Enabling/Disabling Specific Rules One way to control what rules are enforced is to enable/disable individual rules. If you do this, be aware that Jtest looks for violations of a rule only if the rule is enabled and its severity level is enabled. 1. Determine which rules are currently enabled/disabled. 18

19 Lesson 2 - Performing Static Analysis a. Open the Global Test Parameters window by clicking the Global button. b. In the Global Test Parameters window, open Static Analysis> Rules. c. Open the Built-in Rules node or User Defined Rules node. d. Open the node for the category of rules you want to view. 2. Enable/disable rules by selecting or clearing the button next to the rule name. For example, to prevent Jtest from enforcing the NAMING.IFV rule (Use upper-cases for interface member fields), open the Global Test Parameter s Static Analysis> Rules> Built-in Rules> Naming Conventions node, then clear the Use uppercase letters for interface fields option. 19

20 Lesson 2 - Performing Static Analysis Enabling/Disabling Rule Categories Each rule has two categories: The descriptive category that describes the type of rule (for example, Servlets, Design by Contract, Initialization, User Defined Rules, etc.) The severity category that describes the severity of a violation of the rule (violations of coding standards that are most likely to cause an error are level 1; violations of coding standards that are least likely to cause an error are level 5). By default, Jtest reports violations of all coding standards with a severity level of 1, 2, or 3. A rule s severity is indicated by the final character in each rule code, as indicated in the Global Test Parameters window s Static Analysis> Rules> Built-in Rules node. For example, PB.SBC has a severity level of 1, and OOP.IIN has a severity level of 5. To enable/disable all rules in a certain descriptive category: 1. Open the Global Test Parameters window by clicking Global. 2. Right-click the Static Analysis> Rules> <name_of_rule_category> node. A shortcut menu will open. 3. Choose Enable All or Disable All from the shortcut menu. For example, if you want did not want to enforce Enterprise JavaBean rules for your current project, you would open the Global Test Parameter s Static Analysis> Rules> Built in-rules branch, right-click the Enterprise JavaBeans node, then choose Disable All from the shortcut menu. Or, if you wanted to enable the Design by Contract rules, you would open the Global Test Parameter s Static Analysis> Rules> Built in-rules branch, right-click the Design by Contract node, then choose Enable All from the shortcut menu. 20

21 Lesson 2 - Performing Static Analysis To enable/disable rule all rules in a certain severity categories: 1. Open the Global Test Parameters window by clicking Global. 2. Open the Severity Levels Enabled node (beneath Static Analysis> Rules). 3. Enable/disable categories by right-clicking a category whose status you want to change and choosing Enable or Disable from the shortcut menu. For example, if you want Jtest to enforce rules from severity levels 1-4 (instead of 1-3), open the Global Test Parameter s Static Analysis> Rules> Severity Levels Enabled branch, then enable the Level 4 option. 21

22 Lesson 2 - Performing Static Analysis You can also enable/disable severity categories at the class and project level. You might want to do this, for example, if you generally want to enforce only the most severe rules, but you want to enforce all rules for a specific class. To enable/disable rules at the class or project level, set the Severity flags in Class Test Parameters - Static Analysis (if you are testing a single class) or Project Test Parameters - Static Analysis (if you are testing a set of classes). 22

23 Lesson 3 - Performing Black-Box Testing Lesson 3 - Performing Black-Box Testing Black-box (functionality) testing checks a class s functionality by determining whether or not the class s public interface performs according to specification. This type of testing is performed without paying attention to implementation details. If your class contains Design by Contract (DbC)-format specification information, Jtest completely automates the black-box testing process. If not, Jtest makes the black-box testing process significantly easier and more effective than it would be if you were creating test cases by hand. Jtest reads specification information built into the class with the DbC language, then automatically develops test cases based on this specification. Jtest designs its black-box test cases as follows: If the code has postconditions, Jtest creates test cases that verify whether the code satisfies those conditions. If the code has assertions, Jtest creates test cases that try to make the assertions fail. If the code has invariant conditions (conditions that apply to all of a class s methods), Jtest creates test cases that try to make the invariant conditions fail. If the code has preconditions, Jtest tries to find inputs that force all of the paths in the preconditions. If the method under test calls other methods that have specified preconditions, Jtest determines if the method under test can pass non-permissible values to the other methods. Jtest also helps you create black-box test cases if you do not use Design by Contract. You can use Jtest s automatically-generated set of test cases as the foundation for your black-box test suite, then extend it by adding your own test cases. Test cases can be added in a variety of ways; for example, test cases can be introduced by adding: 23

24 Lesson 3 - Performing Black-Box Testing Method inputs directly to a tree node representing each method argument. Constants and methods to global or local repositories, then adding them to any method argument. JUnit-format Test Classes for test cases that are too complex or difficult to be added as method inputs. If a class references external resources, you can enter your own stubs or have Jtest call the actual external method. When the test is run, Jtest uses any available stubs, automatically executes the inputs, and displays the outcomes for those inputs in a simple tree representation. You can then view the outcomes and verify them with the click of a button. Jtest automatically notifies you when specification and regression testing errors occur on subsequent tests of this class. This tutorial covers the following main topics: Adding the Path to Your JDK on page 25 Performing Automatic Black-Box Testing With Design by Contract on page 25 Using Test Classes on page 28 Using Test Classes to Test Methods That Accept Object Type Inputs on page 32 Using Test Classes to Test a Specific Calling Sequence on page 37 Using Test Classes to Validate the State of an Object on page 39 Using JUnit Test Classes on page 45 Adding Primitive Inputs to Static and Instance Methods on page 46 24

25 Lesson 3 - Performing Black-Box Testing Adding the Path to Your JDK To perform black-box testing with Jtest, you must have the JDK compiler on your PATH. The javac command is the compiler; it is usually located under the bin directory of most JDK installations. To tell Jtest where your Java compiler is located: 1. Click the Global button in the current Jtest UI. 2. In the Global Test Parameters window, double-click the Common Parameters> Path to JDK directory, then enter the path to your JDK in the dialog box that opens. 4. Click OK. Performing Automatic Black-Box Testing With Design by Contract This example demonstrates how Jtest performs automatic black-box testing on Example.class, a class file that contains Design by Contract-format specification information. To test this class, perform the following steps: 1. Go to Jtest s Class Testing UI. (This UI opens by default when you launch Jtest). 2. If a class is already loaded into the Class Testing UI (i.e., if you see a class name in the Class Name field), click the New button to clear the previous test. 3. Browse to Example.class (in <jtest_install_dir>/examples/eval) using the Browse button in the Class Name panel. 4. Click the Start button in the tool bar. 25

26 Lesson 3 - Performing Black-Box Testing Jtest will perform both static and dynamic on the class. Because the code s specification is incorporated into the code using Design by Contract comment tags, Jtest can fully automate black-box (functionality) testing as well as white-box (construction) testing. Jtest will automatically create and execute black-box test cases that verify the functionality described in the class s postcondition contract. It will also create and execute white-box test cases that check how the class handles a wide range of inputs. A dialog box will open to notify you when testing is complete. Information on test progress will be displayed in the Test Progress panel. This test uncovers one static analysis error, one Design by Contract violation and one uncaught runtime exception. The following Design by Contact violation will be reported in the Design by Contract Violations branch of the Errors Found panel. 26

27 Lesson 3 - Performing Black-Box Testing This violation indicates that one of the class s methods did not function as described in the specification. To see more information about this violation, expand the violation s branch. To open the source code of the class in an editor, right-click the Source button, then choose Edit Source from the shortcut menu. The source file reveals that the code contract (postcondition) requires the method to return the value of a+b. However, as Jtest revealed, it does not function as specified: the method actually returns the value of a-b. If this were your own class, you would now fix the problem, recompile the class, then retest it to check if your modifications fixed the problem. To see a sample of the test cases that Jtest automatically created to test this class s functionality, click the View button to open the View Test Cases window. In the View Test Cases window, Control-right-click the Automatic Test Cases node, then choose Expand Children from the shortcut menu. Next, go to the uncaught runtime exception found (located in the Uncaught Runtime Exceptions branch of the Errors Found panel) and expand its branches. 27

28 Lesson 3 - Performing Black-Box Testing This error message shows that a NegativeArraySizeException occurs when a negative index is used as an index to an array. This is an expected exception. If this were your code, you would want to document this exception in your code by adding the following Design by Contract Javadoc comment above the method: java.lang.negativearraysizeexception */ By adding this comment, you make the code easier to maintain. Someone looking at the code later on will immediately know that the method is throwing an exception because the code is supposed to throw an exception, not because the code has a bug. In addition, you tell Jtest to suppress future occurrences of this exception. Using Test Classes Test Classes let you test cases that are too complex or difficult to be added as method inputs. For example, you might want to use Test Classes if you want to: Use objects as inputs for static and instance methods. Test a calling sequence and check the state of the object using asserts. Create complicated test cases that depend upon a specific calling sequence. Validate the state of an object. A Test Class is a class that extends jtest.testclass and is used to specify test cases that Jtest should use to test the class. For information about jtest.testclass, see the Jtest API javadoc (you can access this documentation by choosing Help> Jtest API). The jtest.testclass file is in the jtest.zip file located in <jtest_install_dir>/classes. 28

29 Lesson 3 - Performing Black-Box Testing Test Classes give you an easy way to write complicated test cases and to verify the state of the objects under test. You can write your own Test Classes using any Java development environment, or you can integrate JUnit classes into your Jtest tests. This example demonstrates how to add and execute the test cases contained in the CounterTest Test Class located in <jtest_install_dir>examples/dynamic/testclasses to test the Counter class contained in that same directory. Before you start performing tests, open the CounterTest s source code in your preferred source editor. Note the following things about this class: It extends jtest.testclass in <jtest_install_dir>/classes/jtest.zip. It has two test cases, method testadd, method testsubstract, and a test case method called testdividethrowsarithmeticexception. Each of its test case methods begins with the word test. It uses asserts to validate that the calling sequence was performed. To test the Counter class with the CounterTest Test Class, perform the following steps: 1. In the Class Testing UI, browse to and open <jtest_install_dir>/examples/dynamic/testclasses/counter.class. 2. Click the Class button to open the Class Test Parameters window. 3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined. You will see that Jtest has automatically detected the examples.dynamic.testclasses.countertest Test Class. (Jtest automatically detects Test Classes if they are named <name_of class_under_test>test (for example, footest ) and if they are located in the same package as the class under test. 29

30 Lesson 3 - Performing Black-Box Testing 4. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI. 5. When the test is complete, view the results of the user defined test cases. a. Click View to open the View Test Cases window. b. Open the Test Case branches located under User Defined Test Cases> Test Classes. This shows you that Jtest executed the test cases contained in the Test Class. 30

31 Lesson 3 - Performing Black-Box Testing In the Errors Found panel, notice that Jtest finds one Uncaught Runtime Exception from the automatic test cases and one specification error from the testsubstract user-defined test case in CounterTest. 31

32 Lesson 3 - Performing Black-Box Testing Using Test Classes to Test Methods That Accept Object Type Inputs The following examples use Test Classes to specify test cases for static methods and instance methods that accept object type arguments. Testing a Static Method that Accepts an Object Type Input This example demonstrates how to use a Test Class for a static method that accepts a String argument. This example uses the following Test Class (located in <tutorial_install_dir>/lesson3/staticexampletest.class). import jtest.*; public class StaticExampleTest extends TestClass { 32

33 Lesson 3 - Performing Black-Box Testing } public static void testreturnstaticstring() { assert( "String should be hello", StaticInstanceExample.returnStaticString( "hello" ) == "hello"); } Note the following things about this class: It extends jtest.testclass (located in <jtest_install_dir>/classes/jtest.zip. It contains one test case (method testreturnstaticstring). It uses an assert to validate whether the correct object is returned. (To learn more about the assert method, choose Help> Jtest API). To test the StaticInstanceExample class with the StaticExampleTest Test Class, perform the following steps. 1. In the Class Testing UI, browse to and open <tutorial_install_dir>/lesson3/staticinstanceexample.class. 2. Click the Class button to open the Class Test Parameters window. 3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined. 4. Right-click Test Classes, then choose Add Test Class from the shortcut menu. 5. Type StaticExampleTest in the text field, then press Enter. 33

34 Lesson 3 - Performing Black-Box Testing 6. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu. 7. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI. 8. When the test is complete, view the results of the user defined test case. a. Click View to open the View Test Cases window. b. Open User Defined Test Cases> Test Classes> Test Case 1. 34

35 Lesson 3 - Performing Black-Box Testing Testing an Instance Method that Accepts an Object Type Input This example demonstrates how to use a Test Class for an instance method that accepts a String argument. The example Test Class used for this example is located in <tutorial_install_dir>/lesson3/instanceexampletest.class). To run this test, perform the following steps: 1. If you have not already done so, browse to and open <tutorial_install_dir>/lesson3/staticinstanceexample.class. 2. Click the Class button to open the Class Test Parameters window. 3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined. 4. Right-click Test Classes, then choose Add Test Class from the shortcut menu. 35

36 Lesson 3 - Performing Black-Box Testing 5. Type InstanceExampleTest in the text field, then press Enter. 6. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu. 7. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI. 8. When the test is complete, view the results of the user defined test case. a. Click View to open the View Test Cases window. b. Open User Defined Test Cases> Test Classes> Test Case 1. 36

37 Lesson 3 - Performing Black-Box Testing Using Test Classes to Test a Specific Calling Sequence Test Classes can be used to test a calling sequence and check the state of the object using asserts. They can also be used to create complicated test cases that depend upon a specific calling sequence. This example demonstrates how to use Test Classes to test a calling sequence. This example uses the following Test Class (located in <tutorial_install_dir>/lesson3/callingsequencetest.class). import jtest.*; public class CallingSequenceTest extends TestClass{ } public static void testsequence() { CallingSequence m = new CallingSequence( 0 ); // ensure the total after the following sequence is 20 m.add (10); m.mult (2); m.add (2); m.sub (2); assert ("total should be 20", m.gettotal () == 20); } 37

38 Lesson 3 - Performing Black-Box Testing To test the CallingSequence class with the CallingSequenceTest Test Class, perform the following steps: 1. In the Class Testing UI, browse to and open <tutorial_install_dir>/lesson3/callingsequence.class. 2. Click the Class button to open the Class Test Parameters window. 3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined. 4. Right-click Test Classes, then choose Add Test Class from the shortcut menu. 5. Type CallingSequenceTest in the text field, then press Enter. 6. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu. 38

39 Lesson 3 - Performing Black-Box Testing 7. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI. 8. When the test is complete, view the results of the user defined test case. a. Click View to open the View Test Cases window. b. Open User Defined Test Cases> Test Classes> Test Case 1. Using Test Classes to Validate the State of an Object This example demonstrates how to use a Test Class to validate the state of an object in the java.util.vector class from the JDK distribution s rt.jar file. The example Test Class is located in <tutorial_install_dir>\lesson3\myvectortest.java): import java.util.vector; public class ThisVectorTest extends jtest.testclass { public static void test1 () { Vector vector = new Vector (); 39

40 Lesson 3 - Performing Black-Box Testing } vector.addelement ("name"); assert ("should be 1", vector.size () == 1); public static void test2 () { Vector vector = new Vector (); vector.addelement ("name"); vector.addelement ("address"); vector.addelement ("phone number"); assert ("size should be 3", vector.size () == 3); } /* public static void testsetsizethrowsexceptiononnegative () { Vector vector = new Vector (); try { vector.setsize (-1); } catch (ArrayIndexOutOfBoundsException e) { return; } // should not reach this point in the test case assert ("should have thrown exception: size is negative", false); } */ } Note the following things about this class: It extends jtest.testclass in <jtest_install_dir>/classes/jtest.zip. It has two test cases, method test1 and method test2, and a commented test case method called 'testsetsizethrowsexceptiononnegative'. It uses asserts to validate the state of the vector after operations have been performed on it. Each test case begins with test. To test the class using this Test Class. perform the following steps: 1. In the Class Testing UI, click New, then enter java.util.vector in the Class Name field. 2. Click the Class button to open the Class Test Parameters window. 40

41 Lesson 3 - Performing Black-Box Testing 3. In the Class Test Parameters window, open Common Parameters> java/javac-like parameters> -cp. 4. Right-click -cp, then choose Edit from the shortcut menu. 5. Browse to and select <tutorial_install_dir>/lesson3/, then click Add. 6. In the Class Test Parameters window, open Dynamic Analysis> Test Case Execution. 7. Clear the Execute Automatic option. 8. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined. 9. Right-click Test Classes, then choose Add Test Class from the shortcut menu. 10. Type ThisVectorTest in the text field, then press Enter. 41

42 Lesson 3 - Performing Black-Box Testing 11. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu. 12. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI. If you see a dialog box that asks you to set a path to Vector.java, click Cancel. The source code for java.util.vector is not required for this exercise. 42

43 Lesson 3 - Performing Black-Box Testing 13. When the test is complete, two errors should be reported. To view the results of the user defined test cases: a. Click View to open the View Test Cases window. b. Open Test Case branches found under User Defined Test Cases> Test Classes. If you look in each test case s Outcomes node, you will see that no errors were found for these test cases. To add a new test case that will throw an exception as an outcome, perform the following steps: 43

44 Lesson 3 - Performing Black-Box Testing 1. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined> Test Classes. 2. Right-click ThisVectorTest, then choose Edit Test Class from the shortcut menu. 3. In the text editor that contains the Test Class, uncomment the testsetsizethrowsexceptiononnegative method, then save and close the Test Class file. 4. In the Class Test Parameters window, right-click ThisVectorTest, then choose Compile Test Class from the shortcut menu. 5. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu. 6. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI. To see what happens when an exception fails, perform the following steps: 44

45 Lesson 3 - Performing Black-Box Testing 1. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined> Test Classes. 2. Right-click ThisVectorTest, then choose Edit Test Class from the shortcut menu. In the text editor that contains the Test Class, comment out the return statement of the catch block, then save and close the Test Class file. 3. Right-click ThisVectorTest, then choose Compile Test Class from the shortcut menu. 4. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu. 5. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI. An exception will appear in the Specification and Regression Errors branch of the Errors Found panel. Using JUnit Test Classes Jtest can use JUnit Test Classes as well as Jtest Test Classes. For a demonstration of how to integrate your JUnit Test Classes into Jtest, see Lesson 7- Using JUnit Test Classes with Jtest on page

46 Lesson 3 - Performing Black-Box Testing Adding Primitive Inputs to Static and Instance Methods This section contains examples that demonstrate how to add primitive inputs to static and instance methods. Adding Primitive Inputs to a Static Method This example demonstrates how to add primitive input values to static methods and how to validate the outcomes for these inputs. 1. In the Class Testing UI, browse to and open <tutorial_install_dir>/lesson3/staticinstanceexample.class. 2. Click the Class button to open the Class Test Parameters window. 3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined> Method Inputs> return- StaticInt. 4. Right-click int ARG1, then choose Add Input Value from the shortcut menu. 5. Type 3 in the text field, then press Enter. 46

47 Lesson 3 - Performing Black-Box Testing 6. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu. 7. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI. 8. When the test is complete, view the results of the user defined test case, then validate the outcome. a. Click View to open the View Test Cases window. b. Open User Defined Test Cases> Method Inputs> returnstaticint> Test Case 1. c. Right-click RETVAL=3, then choose Mark as Correct to indicate that this is the correct outcome for the given input. Adding Primitive Inputs to Instance Methods This example demonstrates how to add primitive inputs to instance methods and how to validate the outcomes for these inputs. 47

48 Lesson 3 - Performing Black-Box Testing 1. In the Class Testing UI, browse to and open <tutorial_install_dir>/lesson3/staticinstanceexample.class. 2. Click the Class button to open the Class Test Parameters window. 3. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined> Method Inputs> return- InstanceInt. 4. Right-click StaticInstance Example THIS, then choose Add Input Value from the shortcut menu. 5. Type new StaticInstanceExample () in the text field, then press Enter. 6. Right-click int ARG1, then choose Add Input Value from the shortcut menu. 7. Type 5 in the text field, then press Enter. 8. To check the test cases that you just added, press the <control> key and right-click the User Defined node, then select Check 48

49 Lesson 3 - Performing Black-Box Testing from the shortcut menu. 9. Run the test by closing the Class Test Parameters window, then clicking Start in the Class Testing UI. 10. When the test is complete, view the results of the user defined test case, then validate the outcome. a. Click View to open the View Test Cases window. b. Open User Defined Test Cases> Method Inputs> retuninstanceint> Test Case 1. c. Right-click RETVAL=5, then choose Mark as Correct to indicate that this is the correct outcome for the given input. 49

50 Lesson 3 - Performing Black-Box Testing 50

51 Lesson 4 - Performing Regression Testing Lesson 4 - Performing Regression Testing Regression testing checks that class behavior doesn t change. Regression testing gives you the peace of mind of knowing that the class doesn t break when the code is modified. Jtest provides automatic regression testing. Even if you don t specify what the correct outcomes are, Jtest remembers the outcomes from previous test runs, compares outcomes every time the class is tested, and reports an error for any outcome that changes. This lesson contains the following sections: Performing Regression Testing: Overview on page 51 Performing Regression Testing: Example on page 52 Performing Regression Testing: Overview To perform automatic regression testing: 1. Restore previously saved test parameters by choosing File> Open, then select the.ctp or.ptp file that you want to restore. Note: You can also open recently used parameters by choosing File> Open Recent> [File Name]. 2. Run the test by clicking the Start button. If you tested a project, results are displayed in the Class Name> Errors Found> Specification and Regression Errors branch of the Project Testing UI's Results panel. If you tested a single class, results are displayed in the Specification and Regression Errors branch of the Class Testing UI s Errors Found panel. 51

52 Lesson 4 - Performing Regression Testing Performing Regression Testing: Example Jtest doesn t display any regression errors on the first run through a class because it is impossible to detect a regression error the first time a class is tested. Regression testing checks that class outcomes don t change, so it always needs a first run for reference. To see how regression testing works, test Simple.class as described in Lesson 1 - Performing White-Box Testing on page 2 or Lesson 2 - Performing Static Analysis on page 13, introduce an error into Simple.class, and retest Simple.class. To introduce an error and rerun the test: 1. Copy Simple_re.java into Simple.java (both classes are located in <jtest_install_dir>/examples/eval). 2. Recompile Simple.java. 3. Retest Simple.class. Now, along with the other errors, Jtest reports the following regression errors in the Errors Found panel: Expand the error messages to see the inputs for which these regression errors occur. The first error tells us that the code has changed and that the method "add" is now returning 3 instead of 0 for the input 0, 0. The second error reveals that the method "add" is now returning 17 instead of 14 for the input 7,7. 52

53 Lesson 4 - Performing Regression Testing 53

54 Lesson 5 - Testing a Set of Classes Lesson 5 - Testing a Set of Classes At this point, you ve seen how the main types of testing can be performed using the Class Testing UI. Now we will describe how you might apply various testing techniques while testing one of your own projects using the Project Testing UI. This lesson includes the following sections: Beginning the Test on page 54 Fully Automatic Testing on page 54 Running User-Defined Test Cases on page 57 Regression Testing on page 58 Beginning the Test Before you start a test, you need to open the Project Testing UI. You can open this UI by clicking the Project button in the Class Testing UI. Next, click the Browse button and choose the directory, jar file, or zip file that you want to test. Finally, choose File> Save to save your project so that you can easily restore it later. Fully Automatic Testing You might want to start your testing of this project by performing static analysis, white-box testing, and black-box testing (if your classes contain Design by Contract comments) on all of your classes. To perform these tests, simply click the Start button. Errors found will be reported in the [Class Name]> Errors Found branch of the Results panel in the Project Testing UI. 54

55 Lesson 5 - Testing a Set of Classes Static Analysis Results If you want to focus on the static analysis results, you can view only the results for classes that have static analysis errors by right-clicking the [Class Name]> Errors Found> Static Analysis node in the upper results window, then selecting Show Results for this Category from the shortcut menu. Note: To see results for all classes in the lower Results window, right-click the All Classes node in the upper Results panel, then choose Show Results for this Category from the shortcut menu. To get more information about a particular error, go to the lower Results panel and fully expand the Class Name> Errors Found> Static Analysis> [error message] node. To view a description of the rule that was violated, right-click the static analysis error message, then choose View Rule Description from the shortcut menu. To view the source with the line containing the violation highlighted, double-click the node that contains the file/line information for the violation. To enable or disable a rule or rule category, click the Global button and modify static analysis parameters in the Global Test Parameters window. To view project and average class metrics, click the Metrics button. To view a graph of a certain project metric: 1. Click the History button to open the Test History window. 2. Right-click the Test History window s History for Test node. 3. Choose Draw Metrics Graph> [Desired Type of Graph] from the shortcut menu. White-Box Testing Results To focus on the uncaught runtime exceptions found, go to the upper Results window, right-click the [Class Name]> Errors Found> Uncaught Runtime Exceptions node, and choose Show Results from this Category from the shortcut menu. To get detailed information about an error, go to the lower Results window and expand any node. The exception s stack trace is contained in the [Class Name]> Errors Found> Uncaught Runtime Exceptions> [error message] node, and the calling sequence 55

56 Lesson 5 - Testing a Set of Classes is contained in the [Class Name]> Errors Found> Uncaught Runtime Exceptions> [error message]> Test Case Input node. To see the test cases that Jtest automatically generated for a class, right-click the [Class Name] node in the lower Results window, and choose View Test Cases from the shortcut menu. This will open the View Test Cases window that contains the test case inputs and outcomes used for this test. Black-Box (Design by Contract) Results To focus on the functionality problems indicated by Design by Contract violations, go to the upper Results window, right-click the [Class Name]> Errors Found> Design by Contract Violations node, and choose Show Results from this Category from the shortcut menu. To get detailed information about an error, go to the lower Results window and expand any node. The exception s stack trace is contained in the [Class Name]> Errors Found> Design by Contract Violations> [type_of_violation]> [error message] node, and the calling sequence is contained in the [Class Name]> Errors Found> Design by Contract Violations> [type_of_violation]> [error message]> Test Case Input node. To see the test cases that Jtest automatically generated for a class, right-click the [Class Name] node in the lower Results window, and choose View Test Cases from the shortcut menu. This will open the View Test Cases window that contains the test case inputs and outcomes used for this test. To fix the problems found, determine whether each problem is caused by incorrect code or an incorrect contract, then make the necessary modifications to the code or contract. Coverage Information To see the coverage this initial test achieved, open the [Class Name]> Test Progress> Total Coverage node. Focusing on a Single Class s Results If you want to focus on a single class s results, you can open the class in the Class Testing UI; if a class is open in this UI, you will see only the 56

57 Lesson 5 - Testing a Set of Classes errors related to the selected class. To open a class s results in the Class Testing UI, right-click the [Class Name] node in the lower Results window, then choose Load in Class Testing UI from the shortcut menu. Running User-Defined Test Cases Now, if you are not using Design by Contract in your code, you might want to add user-defined test cases to a few of the classes in the project. This type of black-box testing involves three main steps: 1. Adding test cases. 2. Running the test. 3. Validating outputs. Adding Test Cases When you are testing a project, you add test cases for each class individually, via the Class Test Parameters window. One way to edit class test parameters from the Project UI is by performing the following steps: 1. In the Project Testing UI, click Project to open the Project Test Parameters. The Project Test Parameters window will open. 2. Open the Classes in Project branch. 3. Right-click the node that corresponds to the class whose parameters you want to modify. A shortcut menu will open. 4. Choose Edit Class Test Parameters from the shortcut menu. The Class Test Parameters window associated with that class will open. Note: When a class has been tested in the Project Testing UI, you can open its class test parameters by right-clicking the [Class Name] node in the Results panel, then choosing Edit Class Test Parameters from the shortcut menu. Once you are in the Class Test Parameters window, you can add test cases using either Test Classes or method inputs. You can also set the state of objects. Procedures for performing these actions are described in Lesson 3 - Performing Black-Box Testing on page

58 Lesson 5 - Testing a Set of Classes Running the Test To run the test, simply click the Start button in the Project Testing UI. Jtest will then perform all of the tests it performed in previous test runs, plus it will run the user-defined test cases that you added. Validating Outcomes You can see and evaluate each class s test case inputs and outcomes in the View Test Cases window. To open this window from the Project Testing UI, right-click the appropriate [Class Name] node, then choose View Test Cases from the shortcut menu. At this point, you can validate outcomes as described in Lesson 3 - Performing Black-Box Testing on page 23. The next time you run Jtest, all available tests (white-box testing, static analysis, black-box testing, regression testing) will be performed. If any specification or regression errors occur, they will be reported in [Class Name]> Errors Found> Specification and Regression Errors. Regression Testing After you modify one or more of the classes in your project, you will probably want to check whether or not your changes introduced any problems. You can gather such information by performing regression testing. To perform regression testing, simply restore the project test parameters by selecting File> Open, choosing the appropriate.ptp file, then clicking the Start button. Any specification and regression testing errors that occur will be reported in [Class Name]> Errors Found> Specification and Regression Errors. 58

59 Lesson 6 - Using User-Defined Stubs Lesson 6 - Using User-Defined Stubs Stubs are basically replacements for references to external methods. For example, you could use stubs to specify that when the method stream.readint() is invoked, Jtest should use the value 3 instead of actually invoking the readint method. Stubs are mainly used: To isolate a class and test it independently of other classes, or To test a class before the external classes it uses are available. You can enter your own stubs for both automatic and user-defined test cases. When you configure Jtest to use your own stubs, you have complete control over what values or exceptions an external method returns to the class under test-- without having to have the actual external method completed and/or available. Creating and Using User-Defined Stubs There are 5 basic steps involved in creating and using user-defined stubs: Step 1: Create a Stubs Class Step 2: Enable user-defined stubs Step 3: Indicate the stub s location Step 4: Run the test Step 5: View the stubs Step 1: Create a Stubs Class Standard Procedure The first step in using user-defined stubs is creating a Stubs Class. If you create a Stubs Class named (name_of_class_under_test_stubs) and 59

60 Lesson 6 - Using User-Defined Stubs save it in the same directory as the class under test, you will not have to indicate the stub s location. You can use a class with a different name or location as long as you indicate the stub s location (as described in Step 3: Indicate the stub s location). The main way to specify stubs is to create a Stubs Class: a class that contains one or more stubs() methods which define what (if any) return values or exceptions should be used for a certain input. Stubs Classes extend jtest.stubs. For information about jtest.stubs, see the Jtest API javadoc (you can access this documentation by choosing Help> Jtest API). Each Stubs Class should implement a method of the form: public static Object stubs ( Method method // external method being invoked, Object _this // "this" if instance method, Object[] args // arguments to the method // invocation, Method caller_method // method calling "method", boolean executing_automatic // true if executing //automatic Test Case ); Important: Only the first parameter is required; all others are optional. For example, you could define a stubs() method of the form Object stubs (Method method). Whenever the class under test invokes a method method external to the class, Jtest will call the stubs() method. The "stubs()" method should declare what (if any) return values or exceptions should be returned for certain inputs. Use the following table to determine what type of return values or exceptions should be used for each possible stub type: If you want the external method to return this... no stub void Have the stubs() method return this... NO_STUBS_GENERATED VOID 60

61 Lesson 6 - Using User-Defined Stubs a value an exception The value. If the value is a primitive type, it should be wrapped in an Object. For example, if stubs() decides that a given external call should return the integer 3, stubs() should return new Integer (3). The exception that you want the stub to throw. For example, you could use something like: throw new IllegalArgumentException ( time is: + time); Important: The stubs() method can only return an Object. To specify a return value of a primitive type, you need to wrap that type in an Object. For example, if you wanted to specify that the method int usermethod() returned 3, you should have the "stubs ()" method return return new Integer (3) To define stubs for constructor invocations, define a stubs() method whose first parameter is a constructor (instead of a method). If some stubs() method is not defined, no stubs will be used for those members (method or constructor). Stub Objects Stub objects are very useful when writing user-defined stubs. A stub object is similar to any other object, with the following differences: The stub object can be an instance of an interface. For example, the following creates an instance of "Enumeration": Enumeration enum = makestubobject (Enumeration.class); Any method invocation or field reference is a stub even if no stub has been defined for it. If no stub has been defined, Jtest will use a default stub returning the default initialization value for the method return type or field type (for example, "null" for Object, "0.0d" for double,etc.). 61

62 Lesson 6 - Using User-Defined Stubs You need to use stub objects if you want to test classes that use interfaces for which an implementation has not yet been written. They can be used whenever an object of the interface class needs to be created. Stub objects can also be used whenever you want to create an object of a given type without having to call a specific constructor. For example, instead of using "new java.io.fileinputstream ("what to put here?)", you could use: "(FileInputStream) JT.makeStubObject (java.io.fileinput- Stream.class)"; this creates a FileInputStream object. For an example of how to use stub objects, assume that you have a class called StubObjects that uses the Interface Toaster. The Interface Toaster does not have an implementation. Stub Objects.java import java.lang.reflect.*; public class StubObjects { public static int count (Toaster t1) { Toaster t2 = t1.copy (); int num = 0; if (t1.done ()) num += t1.numslices (); if (t2.done ()) num += t2.numslices (); return num; } } Toaster.java package examples.dynamic.stubs.userdef; interface Toaster { boolean done (); 62

63 Lesson 6 - Using User-Defined Stubs } int numslices (); Toaster copy (); Before you test the StubObjects class, you should call the Stubs Class called StubObjectsStubs because the methods copy(), numslices(), and done() are not yet implemented in Toaster. StubObjectsStubs.java class StubObjectsStubs extends jtest.stubs { public static Object stubs (Method method) { String method_name = method.getname (); if (method_name.equals ("copy")) return makestubobject (Toaster.class); if (method_name.equals ("done")) return Boolean.TRUE; return NO_STUB_GENERATED; } } Note that the makestubobject()' method has been used in two places in this example: 1. In the "stubs()" method. 2. As the user-defined input for argument "t1" of method "count". To test the StubObjects class, perform the following steps: 1. In the Class Testing UI, open StubObjects.ctp (located in <jtest_install_dir>/examples/dynamic/stubs/userdef). 2. Look at the user-defined input for argument t1. a. Click Class to open the Class Test Parameters. b. In the Class Test Parameters window, open Dynamic Analysis> Test Case Generation> User Defined> 63

64 Lesson 6 - Using User-Defined Stubs Method Inputs> count> examples.dynamic.stubs.userdef.toaster ARG1. Note that StubObject needed to be invoked as: JT.makeStubObject (...)". 3. Close the Class Test Parameters window. 4. Run the test and view the stubs as described in Step 4: Run the test and Step 5: View the stubs. Note that even though we did not define stubs for the "numslices" method, Jtest automatically generated a stub because the Toaster object was an object created with "makestubobject()". 64

65 Lesson 6 - Using User-Defined Stubs Using Stubs in Test Classes If you are using a Test Class, you can define specific stubs for each test method by defining a "stubs()" methods within the Test Class. To specify the stubs for a test case defined by a "testxyz" method, define a method of the form: Object stubsxyz (Method method,...); in that Test Class. For example, assume that you wanted to use a Test Class to add inputs to MyFile. MyFile package examples.dynamic.stubs.userdef; import java.util.*; public class MyFile { public int add (int i, int j) { return i + j; } } If you wanted to include user-defined stubs in your Test Class, you might create a Test Class like MyFileTest. MyFileTest package examples.dynamic.stubs.userdef; import jtest.*; import java.lang.reflect.*; import java.util.*; class MyFileTest extends TestClass { public static void testadd () { MyFile mt = new MyFile(); Integer Iop1 = (Integer) makestubobject(inte- 65

66 Lesson 6 - Using User-Defined Stubs ger.class); Integer Iop2 = (Integer) makestubobject(integer.class); int op1 = Iop1.parseInt ("20"); int op2 = Iop2.parseInt("10"); int total = mt.add (op1, op2); } assert ("total should be 30", total == 30); public static Object stubsadd (Method method, Object[] args) { String method_name = method.getname(); if (method_name.equals("parseint")) { String tempstring = (String) args[0]; if (tempstring.equals("10")) return new Integer(10); if (tempstring.equals("20")) return new Integer(20); return new Integer(0); } return NO_STUB_GENERATED; } } MyFile If a Test Class does not define a "stubs()" method, or if it does not return any stubs, Jtest will apply the Class and Project Test Parameters "stubs()" methods. Step 2: Enable user-defined stubs Jtest will not use your user-defined stubs unless it is configured to do so. By default, user-defined stubs are enabled for user-defined test cases, but disabled for automatically-generated test cases. 66

67 Lesson 6 - Using User-Defined Stubs You can enable user-defined stubs at the global, project, or class level. To do so, open the appropriate parameters window, then: 1. Open Dynamic Analysis> Test Case Execution> Stubs. 2. Open either Options for Automatic Test Cases or Options for User Defined Test Cases. 3. Enable the Use User Defined Stubs option. Step 3: Indicate the stub s location By default, when Jtest detects that the class under test references an external method, it searches for and uses Stubs Classes that are: In the same directory as the class under test, and Named <name_of_class_under_test>stubs (for example, foostubs.class). If you do not change the default setting (Class Test Parameters Dynamic Analysis> Test Case Execution> Stubs> Stubs Class value set to $DEFAULT ), your Stubs Classes are named correctly, and your Stubs Classes are located in the same directory as the class under test, you do not need to indicate the stub s location. If you need to indicate the Stubs Class location, right-click the Class Test Parameters Dynamic Analysis> Test Case Execution> Stubs> Stubs Class node, choose Edit, then enter the location of the Stubs Class. Important: If you specified your stub via a Test Class, you do not need to indicate the location of a Stubs Class. Step 4: Run the test If you have performed the above steps, Jtest will automatically use your stubs when you test the class or project in the normal manner. Step 5: View the stubs After you test a class using a user-defined stub, you can view the stubs in the User Defined Test Cases> Method Inputs> Method Name> Test Case> Test Case Input branch of the View Test Cases tree. User-defined stubs will be marked with a small black box. Each stub branch displays 67

68 Lesson 6 - Using User-Defined Stubs the method invoked as well as the value or exceptions returned by the stub. Expand the stub s branch to see the stack trace where the invocation occurred. If the stub s values resulted in an error, the above information would also be displayed in the Errors Found Panel (if you were testing a class) or the Results panel (if you were testing a project). Note: For additional examples, look at the following example files located in <jtest_install_dir>/examples/dynamic/stubs/userdef: FileExampleStubs.java, FileExample.java ConstructorExample.java InterpreterTest.java, Interpreter.java (example of use of stubs within a TestClass). 68

69 Lesson 7- Using JUnit Test Classes with Jtest Lesson 7- Using JUnit Test Classes with Jtest Jtest supports Jtest Test Classes as well as JUnit (version 3.5) Test Classes. JUnit is a simple framework to write repeatable tests. This lesson demonstrates how to use JUnit Test Classes in conjunction with Jtest. This lesson contains the following sections: Installing JUnit 3.5 on page 69 Running JUnit's Sample Test Class (junit.samples.vectortest.class) in Jtest on page 70 Testing a Class Using Only a JUnit Test Class (Black-Box Testing) on page 71 Testing a Class Using Only Jtest s Automatic Test Cases (White-Box Testing) on page 73 Testing a Class Using JUnit's Test Class and Jtest s Automatic Test Cases on page 75 Important: Before starting the exercises in this lesson, you need JUnit 3.5. If you do not already have JUnit 3.5, follow the directions in Installing JUnit 3.5 below; otherwise, skip that section. You will also need to use the latest release of Jtest to use JUnit3.5 Test Classes. When this lesson was written, Jtest 4.0 was the latest release. Installing JUnit 3.5 You can download the latest version of JUnit (3.5) from JUnit is distributed as a Zip file. You should extract Junit3.5.zip into c:\ (a junit3.5 directory will be created in c:\ when the file is unzipped). On a Windows NT/2000 system, you will not need Administrator privileges. For more information on installing JUnit, see the Installation section at the JUnit Web site. 69

70 Lesson 7- Using JUnit Test Classes with Jtest This lesson will assume that JUnit is installed in c:\junit3.5. If not, change the paths as appropriate when you are following the exercises. Important: Note that the tests are not contained in junit.jar but in the installation directory directly. Running JUnit s Sample Test Class (junit.samples.vectortest.class) in Jtest Jtest will automatically execute a Test Class if it is loaded into the Class Testing UI. After running the Test Class, Jtest will provide you with information on the number of test cases found and the amount of coverage. To run the sample JUnit Test Class (junit.samples.vectortest), perform the following steps: 1. In the Class Testing UI, click New to clear any previous tests, then load the JUnit Test Class (junit.samples.vectortest) in the Class Testing UI by clicking the Browse button in the Class Name panel, then selecting VectorTest.class from c:\junit3.5\junit\samples directory. 2. Click the Class button to open the Class Test Parameters window. 3. In the Class Test Parameters window, open Common Parameters> java/javac-like parameters branch. 4. Right-click -cp, then choose Edit from the shortcut menu. 5. Add the junit.jar file to the -cp list. a. Browse to and select c:\junit3.5\junit.jar. b. Click the Add button. c. Click OK to close the -cp option window. 6. Click the Start button on the tool bar. Jtest will begin running the VectorTest Test Class (this is similar to running java on VectorTest 70

71 Lesson 7- Using JUnit Test Classes with Jtest from the command line). When Jtest is done running the Test Class, an error dialog will appear. Click OK in this dialog box. 7. Look at the coverage results obtained from this Test Class run by opening the Test Progress panel s Dynamic Analysis> Total Coverage branch. Notice that the amount of coverage is 70.8%. 8. To view the test cases that Jtest found in this Test Class, click the View button on the tool bar. In the Test Cases window, open the User Defined Test Cases> Test Classes branch. Notice that Jtest found 6 test cases for this Test Class. Open each Test Case branch to verify the test case method names of the junit.samples.vectortest Test Class. Testing a Class Using Only a JUnit Test Class (Black-Box Testing) If you want to use a JUnit Test Class for user-defined test cases, you need to: 1. Include the junit.jar file on your CLASSPATH. 2. Compile your junit classes with "junit.jar." 3. Manually indicate the location of your Test Class After you perform these steps, Jtest will use the JUnit Test Class when you test a class in the normal manner. Important: To perform black-box testing with Jtest, you must have the JDK compiler on your PATH. The 'javac' command is the compiler; it is usually located under the bin directory of most JDK installations. To test the java.util.vector class from the JDK distribution (included in rt.jar) with JUnit's Test Class junit.samples.vectortest, perform the following steps: 1. In the Class Testing UI, click New to clear any previous tests, then enter java.util.vector in the Class Name field. 71

72 Lesson 7- Using JUnit Test Classes with Jtest 2. Click the Class button to open the Class Test Parameters window. 3. In the Class Test Parameters window, open the Common Parameters> java/javac-like parameters branch. 4. Right-click -cp, then choose Edit from the shortcut menu. 5. Browse to and select the JUnit installation directory (c:\junit3.5), then click Add. 6. Add the junit.jar file to the -cp list. a. Browse to and select c:\junit3.5\junit.jar. b. Click the Add button again. c. Click OK to close the -cp option window. Note: You could also add junit.jar to the -cp option in the Global Test Parameters window. 7. In the Class Test Parameters window, open the Dynamic Analysis> Test Case Execution branch and set the Execute Automatic option to false. 8. In the same window, open the Static Analysis branch, and set the Perform Static Analysis option to false. Since the source code is not available for java.util.vector, you can skip all Static Analysis tests. 9. In the same window, open the Dynamic Analysis> Test Case Generation> User Defined branch. 10. Right-click the Test Classes node, then choose Add Test Class from the shortcut menu. 11. In the text field, type junit.samples.vectortest, then press Enter. The number 6 should appear in brackets before the Test Classes folder and the Test Class name (the name may appear abbreviated). 12. To check the Test Class, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu. If no message is displayed, you are ready to perform testing on the Vector class. 72

73 Lesson 7- Using JUnit Test Classes with Jtest 13. To start black-box testing on the java.util.vector class, close the Class Test Parameters window and click the Start button in the Class Testing UI. When the test is complete, a dialog will appear. Click OK to confirm the number of errors found. 14. Look at the coverage results obtained from this black-box test by opening the Test Progress panel s Dynamic Analysis> Total Coverage branch. Notice that the amount of coverage is 28.1%. 15. To view the test cases executed on the Vector class, click the View button on the tool bar. In the Test Cases window, open the User Defined Test Cases> Test Classes branch. Notice that Jtest executed the 6 test cases from the VectorTest Test Class. 16. Open each Test Case branch to view and verify the outcomes. If you look in each test case's Outcomes node, you will see that no errors were found for these test cases. Testing a Class Using Only Jtest s Automatic Test Cases (White-Box Testing) To test the java.util.vector class from the JDK distribution (included in rt.jar) with Jtest s automatic tests and without JUnit's Test Class, perform the following steps: 1. If java.util.vector is already loaded in the Class Name field, go to step 2 below. Otherwise, in the Class Testing UI, click New to clear any previous tests), then enter java.util.vector in the Class Name field. 2. Click the Class button to open the Class Test Parameters window. 3. If you have already added the junit directory and jar file to your -cp list, proceed to step 7. If not, in the Class Test Parameters window, open the Common Parameters> java/javac-like parameters branch. 4. Right-click -cp, then choose Edit from the shortcut menu. 73

74 Lesson 7- Using JUnit Test Classes with Jtest 5. Browse to and select the junit installation directory (c:\junit3.5), then click Add. 6. Add the junit.jar file to the -cp list. a. Browse to and select c:\junit3.5\junit.jar. b. Click the Add button again. c. Click OK to close the -cp option window. 7. In the Class Test Parameters window, open Dynamic Analysis> Test Case Execution and set the Execute Automatic option to true. Set the Execute User Defined option to false. 8. In the same window, open the Static Analysis branch, and set the Perform Static Analysis option to false. Since the source code is not available for java.util.vector, you can skip all static analysis tests. 9. To start white-box testing on the java.util.vector class, close the Class Test Parameters window and click the Start button in the Class Testing UI. When the test is complete, a dialog will appear. Click OK to confirm the number of errors found. Nine errors should be reported. 10. Look at the coverage results obtained from this white-box test by opening the Test Progress panel s Dynamic Analysis> Total Coverage branch. Notice that the amount of coverage achieved by this completely automatic mode of testing is 77.7%. 11. To view the automatically-created test cases executed on the Vector class, click the View button on the tool bar. In the Test Cases window, open the Automatic Test Cases branch. Notice that Jtest automatically generated and executed 129 test cases. 12. Open 10 different Test Case branches to see the outcomes. If you look in each test case's Outcomes node, you might see some return values (RETVAL), and some Exceptions. Notice that the test cases from the VectorTest Test Class were not executed because we turned that option off in step 7. 74

75 Lesson 7- Using JUnit Test Classes with Jtest Testing a Class Using JUnit s Test Class and Jtest s Automatic Test Cases To test the java.util.vector class (from the JDK distribution) using JUnit s Test Class, junit.samples.vectortest as well as Jtest s automatic test cases, perform the following steps: 1. If java.util.vector is already loaded in the Class Name field, go to step 2 below. Otherwise, in the Class Testing UI, click New to clear any previous tests), then enter java.util.vector in the Class Name field. 2. Click the Class button to open the Class Test Parameters window. 3. If you have already added the junit directory and jar file to your -cp list, proceed to step 7. If not. in the Class Test Parameters window, open the Common Parameters> java/javac-like parameters branch. 4. Right-click -cp, then choose Edit from the shortcut menu. 5. Browse to and select the junit installation directory (c:\junit3.5), then click Add. 6. Add the junit.jar file to the -cp list. a. Browse to and select c:\junit3.5\junit.jar. b. Click the Add button again. c. Click OK to close the -cp option window. 7. In the Class Test Parameters window, open Dynamic Analysis> Test Case Execution, set the Execute Automatic option to true, then set the Execute User Defined option to true. 8. In the same window, open the Static Analysis branch, and set the Perform Static Analysis option to false. Since the source code is not available for java.util.vector, you can skip all static analysis tests. 75

76 Lesson 7- Using JUnit Test Classes with Jtest 9. In the same window, open the Dynamic Analysis> Test Case Generation> User Defined branch. If you have already selected the junit.samples.vectortest Test Class, then skip steps Right-click the Test Classes node, then choose Add Test Class from the shortcut menu. 11. In the text field, type junit.samples.vectortest, then press Enter. The number 6 should appear in brackets before the Test Classes folder and the Test Class name (the name may appear abbreviated). 12. To check the Test Class, press the <control> key and right-click the User Defined node, then select Check from the shortcut menu. If no message is displayed, you are ready to perform testing on the Vector class. 13. To start black-box and white-box testing on the java.util.vector class, close the Class Test Parameters window and click the Start button in the Class Testing UI. When the test is complete, a dialog will appear. Click OK to confirm the number of errors found. Nine errors should be reported. 14. Look at the coverage results obtained from these white-box and black-box tests by opening the Test Progress panel s Dynamic Analysis> Total Coverage branch. Notice that the amount of coverage achieved is now 83.7%. 15. To view the automatically-created test cases executed on the Vector class, click the View button on the tool bar. In the Test Cases window, open the Automatic Test Cases branch. Notice that Jtest automatically generated and executed 129 test cases. 16. Open 10 different Test Case branches to see the outcomes. If you look in each test case's Outcomes node, you might see some return values (RETVAL), and some Exceptions. 17. To view the user-defined test cases executed on the Vector class, click the View button on the tool bar. In the Test Cases window, open the User Defined Test Cases> Test Classes branch. Notice that Jtest executed the 6 test cases from the VectorTest Test Class as well as the automatically-generated test cases. 76

77 Lesson 7- Using JUnit Test Classes with Jtest 18. Open each Test Case branch to verify and see the outcomes. If you look in each test case s Outcomes node, you will see that no errors were found for these test cases. In summary, the JUnit Test Class alone contained 6 test cases, achieved 28.1% coverage, and found zero errors. When Jtest s automatic testing features were used in conjunction with the JUnit Test Class, 135 test cases were created, 83.7% coverage was achieved, and nine errors were exposed. Thus, a significant increase in test coverage and effectiveness was achieved by adding Jtest to the typical JUnit testing procedure. 77

78 Lesson 7- Using JUnit Test Classes with Jtest 78

79 Lesson 8- Using Jtest in a Multi-User Environment Lesson 8- Using Jtest in a Multi-User Environment This tutorial shows you how to use Jtest in a multi-user environment. This tutorial covers the following main topics: Sharing a Single Class s Test Settings on page 79 Sharing a Project s Test Settings on page 81 Sharing Global Test Settings on page 82 Sharing a Single Class s Test Settings To share a single class s test settings, you need to prepare the appropriate Class Test Parameters (.ctp) file for multi-user use. Then, you can check it into and out of source control and share it as you would share any other file. To prepare a.ctp file for multi-user use: 1. Load the class whose test settings you want to share into the Jtest Class Testing UI. 2. Check the settings in the Class Test Parameters tree s Common Parameters branch. 79

80 Lesson 8- Using Jtest in a Multi-User Environment If the directories used in the Directories and java/javac-like Parameters branches might not work on other systems, you should modify these parameters so that they use a Jtest variable or an environment variable. For information on the Jtest variables that can be used for the nodes in the Common Parameters branch, see the Context Sensitive Help for that area. 3. Save this class s Class Test Parameters (.ctp) file by choosing File> Save or File> Save As. You can then add this file and the related.java files into the source code repository. Team members will then be able to access and update the.java and Class Test Parameters files by checking them in and out of the source code repository in the normal manner. To open a saved class test parameters file in Jtest, choose File> Open in the Class Testing UI, then select the appropriate.ctp file from the file chooser. Note: It is important to use the most current version of the.ctp file when you perform regression testing. 80

81 Lesson 8- Using Jtest in a Multi-User Environment Sharing a Project s Test Settings To share a project s test settings, you need to prepare the appropriate Project Test Parameters (.ptp) file for multi-user use. Then, you can check this file and all related.ctp (Class Test Parameter) files into and out of source control and share them as you would share your other files. To prepare a.ptp file for multi-user use: 1. Load that project whose settings you want to share into the Jtest Project Testing UI. 2. Modify the Common Parameters> Directories> Class Test Parameters Root node so that it reflects a shared location. Jtest automatically creates a.ctp file for each class included in the specified project, and places each.ctp file in the location specified in the Class Test Parameters Root node. For example, if the $PARMS_DIR value is used, all the.ctp files will be placed in the same directory as the.ptp file. For information on the Jtest variables that can be used for the nodes in the Common Parameters branch, see the Context Sensitive Help for that area. 81