No Source Code : Software Testing Black-Box Testing Test-Driven Development No access to source code So test cases don t worry about structure Emphasis is only on ensuring that the contract is met Specification-Based Testing Advantages Specification Operation op Pre: X Post: Y Specification- Based Test Case Design Test Case 1 Input: x1 (sat. X) Exp. Output: y2 Test Case 2 Input: x2 (sat. X) Exp. Output: y2 Scalable; not dependent on size of code Testing needs no knowledge of implementation Tester and developer can be truly independent of each other Tests are done with requirements in mind Does not pardon inconsistencies in the spec Test cases can be developed in parallel with code
Disadvantages Test size will have to be small Specs must be clear, concise, and correct May leave many program paths untested Weighting of program paths is not possible Lot more research on white-box testing Equivalence Partitioning Input data for a program unit usually falls into a number of partitions e.g. all negative integers, zero, all positive numbers Each partition of input data makes the program behave in a similar way Two test cases based on members from the same partition is likely to reveal the same bugs By identifying and testing one member of each partition we gain 'good' coverage with 'small' number of test cases Testing one member of a partition should be as good as testing any member of the partition Test Case Design Examine pre-condition, and identify equivalence classes Enumerate possible inputs such that all classes are covered Apply the specification to input to write down expected output Equivalence Partitioning Example: for binary search the following partitions exist Inputs that conform to pre-conditions Inputs where the precondition is false Inputs where the key element is a member of the array Inputs where the key element is not a member of the array Pick specific conditions of the array The array has a single value Array length is even Array length is odd
Boundary Value Analysis Arises from the fact that most program fail at input boundaries Suppose system asks for a number between 100 and 999 inclusive The boundaries are 100 and 999 We therefore test for values 99 100 101 998 999 1000 Generating Test Cases for a Complete System Simple brute force generation of test cases is sub-optimal How do you know you have enough? How do you know if you re doing too much? Lower boundary upper boundary Example Test case 1: // Sequence public void Add(int pos, int item) /* Requires 0 <= pos <= self Ensures there exists a, b: String : #Self = a*b and a = pos and Self = a * <item> * b */ Input: s = <1, 1, 2, 5>, pos = 3, item = 3 Satisfies precondition: 0 <= pos <= s Expected output: s = <1, 1, 2, 3, 5> A Systematic Approach Modeling approach can vary Test case generation can be automated in some cases
Independently Testable Features Different test cases for different features Simplifies test generation Simplifies fault location Here s where modular design specs really help But the lines of modularity will be different Will generally include several units Test Case Spec Generation Suitably combining values for all inputs of the functional unit under test Cartesian product of enumerated values Combining properties of formal models Brute force ain t gonna work Five inputs, six values each 6 5 = 7776 Modeling and/or Representative Values If the spec is model-based, this is a trivial step Use equivalence partitioning in identifying representative values Two extremes Directly enumerating rep values based on informal spec Formal model that describes rep values based on properties xunit: Unit Testing Frameworks Several tools exist for behavioral testing Junit is one such - for unit testing Java programs Automatically generates stubs for methods you want to test Tester can then write test cases, and execute them
Test-Driven Development Keep the Bar Green! Test first strategy Test cases are written by the programmer Test cases are written as executable Java-code Test cases can be invoked by a single command All test cases should be 100% OK, before a change can be considered done The concept has been implemented in many languages Test cases from different sources can be combined TDD with xunit Keep the Bar Green!
Advantages of TDD Take small steps when writing software Producing test cases on-the-fly Produce documentation on the fly Writing test cases early forces loosely-coupled design The act of writing a unit test is more an act of design than of verification. It is also more an act of documentation than of verification. The act of writing a unit test closes a remarkable number of feedback loops, the least of which is the one pertaining to verification of function [Martin, Newkirk, and Koss, 2003] TDD: Code and Test in Either Order Analysis Design Code Tests Verification Analysis Design Code Tests Verification TDD: Test, Early, Often, and Automated TDD and XP Defects Testing TDD is an integral part of Extreme Programming XP requires the creation of Test Plan along with code Main form of documentation Ten-Minute Build Continuous Integration