Software Quality Assurance (SQA) Software Quality Assurance Use of analysis to validate artifacts requirements analysis design analysis code analysis and testing Technical/Document reviews Control of changes configuration management
Costs of Poor Quality Increased cost to find and fi problems Increased cost to distribute modifications Increased customer support Product liability Failure in the marketplace Failure of the system Terminology correctness reliability testing debugging failure fault/defect error verification validation V&V IV&V
Terminology correctness--software (artifact) is consistent with its specification specification could be wrong or incomplete rarely is the specification correct rarely is software correct reliability-- (high) probability that the software conforms to its specification usually a statistical measure based on past performance e.g., mean time to failure need to know operational profile But, frequently not known Terminology (continued) testing--eecute the software on selected test cases evaluate the results (test oracle) evaluate performance evaluate ease of use testing can demonstrate the presence of bugs but can never prove their absence can rarely rely on ehaustive testing how should you select test cases? how do you know when to stop testing?
Testing is a sampling technique Terminology (Continued) failure-- an erroneous result produces incorrect outputs fails to meet a real-time constraint fault--the cause of one or more failures error -- incorrect concept e.g., design error, logic error debugging--the process of finding the cause of a failure and finding a way to fi the associated faults (without introducing additional faults)
Terminology (Continued) verification -- the process of proving, using mathematical reasoning, that a program is consistent with its specification validation -- the processes associated with demonstrating that the software performs reasonably well V&V -- verification and validation IV&V - independent V&V IV&V Psychologically difficult for programmers to test their own code thoroughly Inherently want to believe that it works and try to demonstrate this If the programmers had thought of all the devious cases that could arise, then these would be reflected in the program Want someone who is motivated to find problems Want someone who has fresh insight into how to eercise the system But, QA is often under pressure not to find problems so that a product can be released on time
Different kinds of testing unit testing-- test individual components use test stubs and drivers integration testing--combine components and test them big bang testing incremental integration build up to subsystems system testing--test whole system acceptance testing--testing to determine if the product will be accepted Different kinds of testing (continued) regression testing--retesting after some aspect of the system has been changed determine "old" test cases that must be re-eecuted determine what new test cases are required play and capture technology for UI centric systems
Different kinds of testing (continued) black bo or functional testing testing based on specifications white bo or structural testing testing based on looking at the artifact need to do both black bo and white bo testing Testing is hard work Typically 50% of software development effort goes to testing Up to 85% for life critical software Objectives Epose errors Good test case is one with a high probability of finding a new failure Successful test case is one that finds a new type of failure
Ehaustive testing Ehaustive testing requires testing all combination of input values Sorting an array of size 10 containing one each of 1..10 has 10! input combinations (3,628,800 cases) Testing usually sparsely samples the input space
Testing can: Uncover failures Show specifications are met for specific test cases Be an indication of overall reliability cannot: Prove that a program is fault-free or correct Testing Principles 80% of all errors will likely occur in 20% of the modules Ehaustive testing is not possible Each test case should be chosen to maimize likelihood of finding a failure Testing should also be done by someone other than the developers developers do original testing SQA (or IV&V) does independent testing usually black bo testing
Quality must be an on-going concern Can't build quality into software after the fact Tests should be planned long before testing begins requirements --> test plans --> functional test cases design --> enhanced test plans --> functional and structural test cases code --> enhanced structural (and functional) test cases maintenance --> enhanced functional and structural test cases Debugging Find the cause of a failure and fies it Debugging is difficult because Symptom may appear long after the fault occurs May be difficult to reproduce the symptom Symptom may be intermittent, indicating a timing or load dependent problem Unit testing (and incremental integration testing) simplifies debugging during development because it helps localize faults