Verification, Testing, and Bugs

Transcription

1 Verification, Testing, and Bugs Ariane 5 Rocket First Launch Failure 2hwk

2 So What Happened? The sequence of events that led to the destruction of the Ariane 5 was as follows: 1. The INS attempted to convert a 64-bit number to a 16- bit number without checking that 16 bits was sufficient to hold the value. This caused a runtime error. 2. The runtime error caused the INS software to stop execution. 3. There was a backup computer executing the INS software, but (of course) it too encountered the same problem and terminated. 4. The INS hardware (of both computers) sent a report of the error to the main computer. 5. The main computer erroneously interpreted the report as extreme, but legal, data, and commanded the nozzle to fully deflect to one side. 6. The rocket turned at a sharp angle and was subjected to forces that it was not designed to withstand. It began to break up and was destroyed. Building Reliable Systems: Two Principles For a software system to be reliable: Each stage of development must be done well, with incremental verification and testing Testing and correction do not ensure quality, but reliable systems are not possible without thorough testing

3 Static and Dynamic Verification Static verification Techniques of verification that do not include execution of the software May be manual or use computer tools Dynamic verification Testing the software with trial data Debugging to remove errors Static Verification: Reviews Reviews are a form of static verification that is carried out throughout the software development process

4 Reviews Reviews are a fundamental part of good software development Concept: colleagues review each other s work Can be applied to any stage of software development, but particularly valuable for program design or code review Can be formal or informal Preparation The developer(s) provides colleagues with documentation (e.g., models, specifications, or design), or code listing. Participants study the materials in advance. Meeting The developer leads the reviewers through the materials, describing what each section does and encouraging questions. The Review Meeting A review is a structured meeting. Participants and their roles: Developer(s): person(s) whose work is being reviewed Moderator: ensures that the meeting moves ahead steadily Scribe: records discussion in a constructive manner Interested parties: other developers on the same project Outside experts: knowledgeable people who are not working on this project Client: representatives of the client who are knowledgeable about this part of the process

5 Benefits of Reviews Extra eyes spot mistakes, suggest improvements Colleagues share expertise; helps with training Incompatibilities between components can be identified Gives developers an incentive to tidy loose ends Helps scheduling and management control Successful Reviews To make a review a success: Senior team members must show leadership Good reviews require good preparation by everybody Everybody must be helpful, not threatening Allow plenty of time and be prepared to continue on another day.

6 Static Verification: Pair Design and Pair Programming Concept: achieve benefits of review by shared development Two people work together as a team: Design and/or coding Testing and system integration Documentation and hand-over Benefits include: Two people create better software with fewer mistakes Cross training Many software houses report excellent productivity using this model. Static Verification: Program Inspections Formal program reviews whose objective is to detect faults Code is read or reviewed line by line 150 to 250 lines of code in 2 hour meeting Use checklist of common errors Requires team commitment So effective that it is claimed that it can replace unit testing.

7 Static Verification: Analysis Tools Program analyzers scan the source of a program for possible errors and anomalies. Control flow: loops with multiple exit or entry points Data use: undeclared or uninitialized variables, unused variables, multiple assignments, array bounds Interface faults: parameter mismatches, non-use of functions results, uncalled procedures Storage management: unassigned pointers Defensive Programming Murphy s Law: if anything can go wrong, it will Defensive programming Write simple code Avoid risky programming constructs If code is difficult to read, rewrite it Incorporate redundant code to check system state after modifications Test implicit assumptions explicitly, e.g., check all parameters received from other routines Good programming practice eliminates all warnings from source code

8 Dynamic Verification: Stages of Testing Testing is most effective if divided into stages User interface testing (Carried out separately) Unit testing: unit test System testing Integration test Function test Performance test Installation test Acceptance testing (Carried out separately) Testing Strategies Bottom-up testing: each unit is tested with its own test environment Top-down testing: large components are tested with dummy stubs. Particularly useful for: user interfaces and client demos Stress testing: tests the system beyond its limits. Particularly useful for: real-time systems, transaction processing Most systems require a combination of all three strategies.

9 Methods of Testing Open box testing Testing is carried out by people who know the internals of what they are testing Example: tick marks in a graphing package Closed box testing Testing is carried out by people who do not know the internals of what they are testing Example: educational demonstration that was not foolproof Testing: Unit Testing Tests on small sections of a system, e.g., a single class Emphasis on accuracy of code against specification Test data is usually chosen by developers based on their understanding of the specification and knowledge of the unit Can be open or closed box: by the developers of the unit or by special testers If unit testing is not thorough, system testing becomes almost impossible. If you are working on a project that is behind schedule, do not rush unit testing.

10 Testing: System and Sub-System Testing Tests on components or complete system, combining units that have already been tested Emphasis on integration and interfaces Trial data that is typical of the actual data, and/or stresses the boundaries of the system Carried out systematically, adding components until the entire system is assembled Can be open or closed box System testing is finished faster if each component is completely debugged before assembling the next. Dynamic Verification: Test Design Testing can never prove the correctness of the system It can only show that either: A system is correct in a special case, or That it has an error The objective of testing is to find errors or demonstrate that program is correct in specific instances Testing is never comprehensive Testing is expensive

11 Test Cases Test cases are specific tests that are chosen because they are likely to find specific problems. Test cases are chosen to balance expense against chance of finding serious errors Cases chosen by the development team are effective in testing known vulnerable areas Cases chosen by experienced outsiders and clients will be effective in finding gaps left by the developers Cases chosen by inexperienced users will find other errors Variations in Test Sets A test suite is the set of all test cases that apply to a system or component of a system When running tests, there are some errors that occur only under certain circumstances, e.g., when certain other software is running on the same machine, when tasks are scheduled in specific sequences, with unusual data sets, etc. Therefore, it is customary for each test to vary some of the test cases systematically, and to change the order in which the tests are made, etc.

12 Incremental Testing (e.g., Daily Testing) [common in spiral process] Create a first iteration that has the structure of the final system and some basic functionality Create an initial set of test cases Check-in changes to the system on a daily basis, rebuild entire system daily Run a comprehensive set of test cases daily, identify and deal with any new errors Add new test cases continually Dynamic Verification: Regression Testing When software is modified, regression testing is used to check that modifications behave as intended and do not adversely affect the behavior of unmodified code After every change, however small, rerun the entire testing suite

13 Incremental Testing: Example A graphics package consisting of a pre-processor, a runtime package (Set of classes) and several device drivers Starting point: a prototype with a good overall structure, and most of the functionality, but hastily coded and not robust Approach on a daily cycle Design and code one small part of the package Integrate into prototype Create additional test cases if needed Regression test Documentation of Testing Every project needs a test plan that documents the testing procedures for thoroughness, visibility, and for future maintenance. It should include: Description of testing approach List of test cases and related bugs Procedures for running the tests Test analysis report

14 Fixing Bugs (debugging) Isolate the bug Intermittent -> repeatable Complex example -> simple example Understand the bug and its context Root cause Dependencies Structural interactions Fix the bug Design changes Documentation changes Code changes Create new test case Moving the Bugs Around Fixing bugs in an error-prone process When you fix a bug, fix its environment Bug fixes need static and dynamic testing Repeat all tests that have the slightest relevance (Regression testing) Bugs have a habit of returning When a bug is fixed, add the failure case to the test suite for future regression testing Persistence Most people work around a problem. The best people track down the root cause and fix it once and for all

15 Difficult Bugs Some bugs may be extremely difficult to track down and isolate. This is particularly true of intermittent failures. Examples: A large central computer stops a few times every month with no dump or other diagnostic. A database load dies after running for several days with no diagnostics. An image processing system runs correctly, but uses huge amounts of memory. Such problems may require months of effort to track down. For a fictional example, see: Ellen Ullman, The Bug: a Novel, (Doubleday 2003). Tracking Down a Difficult Bug (Image processing bug)

16 When Fixing a Bug Creates a Problem for Customers Sometimes customers will build applications that rely upon a bug. Fixing the bug will break the applications. The graphics package with rotation about the Z-axis in the wrong direction. An application crashes with an emulator, even though the emulator is bug free. (Compensating bug problem.) The 3-pixel rendering problem with Internet Explorer. With each of these bugs the code was easy to fix, but releasing it would have caused problems for exis6ng programs. A Note on User Interface Testing UI need several categories of testing During the design phase, user interface testing is carried out with trial users to ensure that the design is usable. Design testing is also used to develop graphical elements and to validate requirements During the implementation phase, the user interface goes through the standard steps of unit and system testing to check the reliability of the implementation Finally, acceptance testing is carried out with users, on the complete system