Integration and Testing Uses slides from Lethbridge & Laganiere, 2001
Testing phases: V model Requirements Acceptance Testing Specifications System Testing Design Integration Testing Detailed Design Unit Testing Implementation A lifecycle view that shows relationships between development and test phases
Concurrent Test Development Design Impl Reqs Test Design Test Impl Unit Testing Integ Testing System Testing Testing is done after development, but test design & impl. often happens concurrently with development
Testing Phases Unit Testing Developer tests individual modules Usually glass box Integration testing Put modules together, try to get them working together Integration testing is complete when the different pieces are able to work together System testing Black-box testing of entire deliverable against specs Acceptance testing Testing against user needs, often by the user
Coverage Testing should try out all possibilities Exhaustive testing is impossible for non-trivial software Objective is to cover possibilities as well as possible Unit testing / glass-box testing Function coverage: Each function/method executed by at least one test case Statement coverage: Each line of code covered by at least one test case (need more test cases than above) Path coverage: Every possible path through code covered by at least one test case (need lots of test cases)
Coverage Black-box testing Requirements coverage: Each requirement / use case / user story covered by at least one test case Equivalence class coverage: Each equivalence class for each input covered by at least one test case Combination of equivalence classes coverage: Each legal combination of equivalence classes for different inputs covered may require huge number of test cases There are tools that will automatically analyze various forms of coverage and generate statistics Also tools to help generate test cases
System Testing Thorough black-box testing of entire software before delivery, based on requirements specification Final phase of testing before delivery Requirements coverage is a must, equivalence class coverage preferred For most commercial software, reasonably good coverage of combinations of equivalence classes is the norm Can automate system testing by creating test harnesses that feed test inputs and check outputs
Integration Testing Integration: Putting together the code created by different developers and trying to run an end-to-end test case Can be a major challenge, because every small misunderstanding /miscommunication between developers results in mismatches that need to be found and removed Each time it doesn t work, need to trace which code is the source of the problem Not uncommon to need to resolve dozens of bugs before that first test case will run through! Integration testing becomes easier if Unit testing is thorough Interfaces between modules are well-defined
Alternative Integration Strategies Big bang integration Put all the code together in one go, and try to get it all working together Works only for relatively small projects Incremental integration Test each individual module in isolation Start with one module, add one module at a time, work out problems among them before adding next More manageable, but more work Horizontal integration if there are multiple subsystems Vertical integration if there are multiple layers
Top down integration Start by testing just the user interface. The underlying functionality are simulated by stubs. Pieces of code that have the same interface as the lower level functionality. Do not perform any real computations or manipulate any real data. Then you work downwards, integrating lower and lower layers. Advantage: Easy to see the behavior at each stage The big drawback to top down testing is the cost of writing the stubs.
Bottom-up integration Start by testing the very lowest levels of the software. You needs drivers to test these lower layers of software. Drivers are simple programs designed specifically for testing that make calls to these lower layers. Drivers in bottom-up testing have a similar role to stubs in top-down testing, and are time-consuming to write. Advantage: Each piece being tested is complete in itself Problem: Need to write lots of drivers, functionality may not be easy to visualize
Sandwich testing Sandwich testing is a hybrid between bottom-up and top down testing. Test the user interface in isolation, using stubs. Test the very lowest level functions, using drivers. When the complete system is integrated, only the middle layer remains on which to perform the final set of tests.
Example of different integration strategies Top-down testing Bottom-up testing Sandwich testing UI Layer stub stub stub driver driver Database layer driver Network layer UI Layer stub stub stub UI Layer Functional layer stub stub stub driver driver Functional layer Database layer driver Network layer driver driver Database layer driver Network layer Fully integrated system UI Layer Functional layer Database layer Network layer
The test-fix-test cycle When a failure occurs during testing: Each failure report is entered into a failure tracking system. It is then screened and assigned a priority. Low-priority failures might be put on a known bugs list that is included with the software s release notes. Some failure reports might be merged if they appear to result from the same defects. Somebody is assigned to investigate a failure. That person tracks down the defect and fixes it. Finally a new version of the system is created, ready to be tested again.
The ripple effect There is a high probability that the efforts to remove the defects may have actually added new defects The maintainer tries to fix problems without fully understanding the ramifications of the changes The maintainer makes ordinary human errors The system regresses into a more and more failure-prone state
Regression testing It tends to be far too expensive to re-run every single test case every time a change is made to software. Hence only a subset of the previously-successful test cases is actually re-run. This process is called regression testing. The tests that are re-run are called regression tests. Regression test cases are carefully selected to cover as much of the system as possible. The law of conservation of bugs : The number of bugs remaining in a large system is proportional to the number of bugs already fixed
Deciding when to stop testing All of the level 1 ( critical ) test cases must have been successfully executed. Certain pre-defined percentages of level 2 and level 3 test cases must have been executed successfully. The targets must have been achieved and are maintained for at least two cycles of builds. A build involves compiling and integrating all the components. Failure rates can fluctuate from build to build as: Different sets of regression tests are run. New defects are introduced.
The roles of people involved in testing The first pass of unit and integration testing is called developer testing. Preliminary testing performed by the software developers who do the design. Independent testing may be performed by separate group. They do not have a vested interest in seeing as many test cases pass as possible. They develop specific expertise in how to do good testing, and how to use testing tools.
Test planning Decide on overall test strategy What type of integration Whether to automate system tests Whether there is an independent test team Decide on the coverage strategy for system tests Compute the number of test cases needed Identify the test cases and implement them The set of test cases constitutes a test suite May categorize into critical, important, optional tests (level 1, 2, 3) Identify a subset of the tests as regression tests
Testing performed by users and clients Alpha testing Performed by the user or client, but under the supervision of the software development team. Beta testing Performed by the user or client in a normal work environment. Recruited from the potential user population. An open beta release is the release of low-quality software to the general population. Acceptance testing Performed by users and customers. However, the customers do it on their own initiative.
Inspecting compared to testing Both testing and inspection rely on different aspects of human intelligence. Testing can find defects whose consequences are obvious but which are buried in complex code. Inspecting can find defects that relate to maintainability or efficiency. The chances of mistakes are reduced if both activities are performed.
Testing or inspecting, which comes first? It is important to inspect software before extensively testing it. The reason for this is that inspecting allows you to quickly get rid of many defects. If you test first, and inspectors recommend that redesign is needed, the testing work has been wasted. There is a growing consensus that it is most efficient to inspect software before any testing is done. Even before developer testing
Packaging for Delivery Software we deliver to the user must include Executable in a convenient format e.g. EXE, JAR file Release notes User documentation: instructions on usage Tutorials, user manuals, getting started instructions Installation instructions May create installables Compressed packages e.g. zip files, tar files Scripts that automate installation procedures