Black Box Testing EEC 521: Software Engineering Software Testing Black-Box Testing Test-Driven Development Also known as specification-based testing Tester has access only to running code and the specification it is supposed to satisfy Test cases are written with no knowledge of internal workings of the code Focus is on ensuring that contract is met, and all legal invocations are handled correctly 11/12/09 EEC 521: Software Engineering 1 11/12/09 EEC 521: Software Engineering 2 Specification-Based Testing Specification Operation op Pre: X Post: Y Specification- Based Test Case Design Test Case 1 Input: x1 (sat. X) Exp. Output: y1 Test Case 2 Input: x2 (sat. X) Exp. Output: y2 No Source Code No access to source code So test cases don t worry about structure Emphasis is only on ensuring that the contract is met 11/12/09 EEC 521: Software Engineering 3 11/12/09 EEC 521: Software Engineering 4
Advantages Scalable; not dependent on size of code Testing needs no knowledge of implementation Tester and developer can be truly independent of each other Tests are done with requirements in mind Does not pardon inconsistencies in the spec Test cases can be developed in parallel with code Disadvantages Test size will have to be small Specs must be clear, concise, and correct May leave many program paths untested Weighting of program paths is not possible Lot more research on white-box testing 11/12/09 EEC 521: Software Engineering 5 11/12/09 EEC 521: Software Engineering 6 Test Case Design Examine pre-condition, and identify equivalence classes Enumerate possible inputs such that all classes are covered Apply the specification to input to write down expected output Equivalence Partitioning Input data for a program unit usually falls into a number of partitions e.g. all negative integers, zero, all positive numbers Each partition of input data makes the program behave in a similar way Two test cases based on members from the same partition is likely to reveal the same bugs By identifying and testing one member of each partition we gain 'good' coverage with 'small' number of test cases Testing one member of a partition should be as good as testing any member of the partition 11/12/09 EEC 521: Software Engineering 7 11/12/09 EEC 521: Software Engineering 8
Equivalence Partitioning Boundary Value Analysis Example: for binary search the following partitions exist Inputs that conform to pre-conditions Inputs where the precondition is false Inputs where the key element is a member of the array Inputs where the key element is not a member of the array Pick specific conditions of the array The array has a single value Array length is even Array length is odd 11/12/09 EEC 521: Software Engineering 9 Arises from the fact that most program fail at input boundaries Suppose system asks for a number between 100 and 999 inclusive The boundaries are 100 and 999 We therefore test for values 99 100 101 Lower boundary 998 999 1000 upper boundary 11/12/09 EEC 521: Software Engineering 10 Example Test case 1: // Sequence public void Add(int pos, int item) /* Requires 0 <= pos <= self Ensures there exists a, b: String : #Self = a*b and a = pos and Self = a * <item> * b */ Input: s = <1, 1, 2, 5>, pos = 3, item = 3 Satisfies precondition: 0 <= pos <= s Expected output: s = <1, 1, 2, 3, 5> Generating Test Cases for a Complete System Simple brute force generation of test cases is sub-optimal How do you know you have enough? How do you know if you re doing too much? 11/12/09 EEC 521: Software Engineering 11 11/12/09 EEC 521: Software Engineering 12
A Systematic Approach Modeling approach can vary Test case generation can be automated in some cases Independently Testable Features Different test cases for different features Simplifies test generation Simplifies fault location Here s where modular design specs really help But the lines of modularity will be different Will generally include several units 11/12/09 EEC 521: Software Engineering 13 11/12/09 EEC 521: Software Engineering 14 Modeling and/or Representative Values If the spec is model-based, this is a trivial step Use equivalence partitioning in identifying representative values Two extremes Directly enumerating rep values based on informal spec Formal model that describes rep values based on properties Test Case Spec Generation Suitably combining values for all inputs of the functional unit under test Cartesian product of enumerated values Combining properties of formal models Brute force ain t gonna work Five inputs, six values each 6 5 = 7776 11/12/09 EEC 521: Software Engineering 15 11/12/09 EEC 521: Software Engineering 16
xunit: Unit Testing Frameworks Several tools exist for behavioral testing Junit is one such - for unit testing Java programs Automatically generates stubs for methods you want to test Tester can then write test cases, and execute them 11/12/09 EEC 521: Software Engineering 17 Test-Driven Development Test first strategy Test cases are written by the programmer Test cases are written as executable Java-code Test cases can be invoked by a single command All test cases should be 100% OK, before a change can be considered done The concept has been implemented in many languages Test cases from different sources can be combined 11/12/09 EEC 521: Software Engineering 18 TDD with xunit Keep the Bar Green! 11/12/09 EEC 521: Software Engineering 19 11/12/09 EEC 521: Software Engineering 20
Keep the Bar Green! Advantages of TDD Take small steps when writing software Producing test cases on-the-fly Produce documentation on the fly Writing test cases early forces loosely-coupled design The act of writing a unit test is more an act of design than of verification. It is also more an act of documentation than of verification. The act of writing a unit test closes a remarkable number of feedback loops, the least of which is the one pertaining to verification of function [Martin, Newkirk, and Koss, 2003] 11/12/09 EEC 521: Software Engineering 21 11/12/09 EEC 521: Software Engineering 22 TDD: Test, Early, Often, and Automated TDD: Code and Test in Either Order Analysis Design Code Tests Verification Defects Testing Analysis Design Code Tests Verification 11/12/09 EEC 521: Software Engineering 23 11/12/09 EEC 521: Software Engineering 24
TDD and XP TDD is an integral part of Extreme Programming XP requires the creation of Test Plan along with code Main form of documentation Ten-Minute Build Continuous Integration 11/12/09 EEC 521: Software Engineering 25