Software Quality Assurance & Testing 1.Software Testing - An ISTQB-BCS Certified Tester Foundation Guide 3rd edition, 2015 Brian Hambling, Peter Morgan, Geoff Thompson, Peter Williams,Angelina Samaroo 2.Advanced Software Testing, Rex Black,2016 3. Introduction to Software Testing, Paul Ammann, Jeff Offutt, 2016
Question 1 Which of the following is a valid objective of testing? i)locating defects in the code. ii)preventing defects. iii)ensuring no defects are present. iv) Correcting defects.
Question 2 Which of the following are most important in determining the quality of test cases? i)referencing the current version of the requirements. ii)traceability to requirements via test conditions. iii)incorporating well-defined expected results. iv)referencing the test procedure and test execution schedule. v)identifying the author of the test case. a) i and ii b) ii and iii c) iii and iv d) ii and v
Testing and Debugging Testing and debugging are different kinds of activity Both are very important. Debugging is the process that developers go through to identify the cause of bugs or defects in code and undertake corrections. Testing is a systematic exploration of a component or system with the main aim of finding and reporting defects
Testing and Debugging Testing does not include correction of defects These are passed on to the developer to correct. Testing ensure that changes and corrections are checked for their effect on other parts of the component or system
Testing and Debugging Debugging does not give confidence that the component or system meets its requirements completely. Testing makes an examination of the behavior of a component or system and reports all defects found for the development team to correct. Testing repeats enough tests to ensure that defect corrections have been effective. Both are needed to achieve a quality result.
Specification-Based Testing Black-Box Testing To implement the behavioral testing, it is required to derive and select tests by analyzing the test basis. Test bases are the documents Test bases directly or indirectly explain how the component or system under test should and shouldn t behave what it is required to do how it is required to do it.
Equivalence Partitioning This is the most basic of specification-based test design techniques The method is about testing various groups that are expected to be handled the same way by the system and exhibit similar behavior. Those groups can be inputs, outputs, internal values, calculations, time values, valid and invalid groups.
We select a single value from each equivalence partition This allows us to reduce the number of tests. We can calculate coverage by dividing the number of equivalence partitions tested by the number identified The goal is to achieve 100% coverage by selecting at least one value from each partition.
Smoke Tests and Equivalance Partitioning The technique applicable at any test level, in any situation Those partitions are ideally independent, Since testing of some of the less-risky partitions frequently is omitted in smoke tests, this technique is useful in constructing smoke tests
Smoke Testing - Build Verification Testing It is type of software testing that comprises of a non-exhaustive set of tests The tests aim at ensuring that the most important functions work. The results of this testing is used to decide if a build is stable enough to proceed with further testing.
Equivalence Partitioning This technique will find primarily functional defects where data is processed improperly in one or more partitions. The key to this technique is to take care that the values in each equivalence partition are handled the same way Otherwise, it is possible to miss potentially important test values
Equivalent Classes Equivalent Partitions inputs, outputs, internal values, time relationships, calculations, just about anything else of interest These classes or partitions are called equivalent They should be handled the same way by the system
Valid and Invalid Equivalence Classes Some of the classes can be called valid equivalence classes they describe valid situations that the system should handle normally. Other classes can be called invalid equivalence classes they describe invalid situations The system should reject The system at least escalate to the user for correction or exception handling.
Visualizing Equivalence Partitioning.
Equivalence Partitioning Applied Iteratively
Equivalence Partitioning Errors
An Important Factor for the Equivalence Partitioning The equivalence partitioning process does not subtract; it divides The union of the subsets must be the same as the original set that was equivalence partitioned Equivalence partitioning does not generate spare subsets If this is not true, failing may occcur During the test of some important subset of inputs, outputs, configurations,.
Composing Tests The values may be independent and all valid, or dependent and all valid
Composing Tests with Invalid Values If we wanted to sub partition the invalid situation, we could also test with a start date in the present and one in the future, together with an end date before the start date, which would add two more tests.
Combinatorial Testing Identify a suitable subset of test combinations Testing process achieves a predetermined level of coverage when testing an object with multiple parameters where parameters themselves each have several values which gives rise to more combinations than are feasible to test in the time allowed. For example Test combinations of tasks and milestones (Test Case TC1) with past, present, and future start dates (Test Case TC2) and valid end dates (Test Case TC3). We check that illegal end dates are correctly rejected (Test Case TC4).
Equivalence Partitioning Exercise Screen prototype for one screen of the HELLOCARMS system
Equivalence Partitioning For the application-product field, the equivalence partitions are: #Partition 1Home equity loan 2Home equity line of credit 3Reverse mortgage
Equivalence Partitioning Example Explanation For each of two existing-account entries, the situation is best modeled as a single input field, which consists of two subfields. The first subfield is the Yes/No field. This subfield determines the rule for checking the second subfield, which is the account number If the first subfield is Yes, the second subfield must be a valid account number. If the first subfield is No, the second subfield must be blank.
The existing checking account information partitions are: #Partition 1Yes-Valid 2Yes-Invalid 3No-Blank 4No-Nonblank The existing savings account information partitions are: #Partition 1Yes-Valid 2Yes-Invalid 3No-Blank 4No-Nonblank For both of these, partitions 2 and 4 areinvalid partitions, while partitions 1 and 3 are valid partitions.
Test Cases for Equivalence Partitions
Product
Testing Levels
Basic Test Terminology An error is a mistake made by an engineer often a misunderstanding of a requirement or design specification A fault is a manifestation of that error in the code what we often call a bug A failure is an incorrect output/behavior that is caused by executing a fault The failure may occur immediately (crash!) or -much later in the execution Testing attempts to surface failures in our software systems Debugging attempts to associate failures with faults so they can be removed from the system If a system passes all of its tests, is it free of all faults?
Testing Goals Based on Test Process Maturity Bezier Testing Levels Level 0 There is no difference between testing and debugging Level 1 The purpose of testing is to show the correctness (software works). Level 2 The purpose of testing is to show that the software doesn t work. Level 3 The purpose of testing is not to prove anything specific, but to reduce the risk of using the software. Level 4 Testing is a mental discipline that helps all IT professionals develop higher quality software.
Testing Goals Based on Test Process Testing Levels (Bezier) Level 0 There is no difference between testing and debugging Level 1 The purpose of testing is to show the correctness (software works). Level 2 The purpose of testing is to show that the software doesn t work.(the purpose is to show failures) Level 3 Testing shows thepresence, not absence or failure. If we use software, we expose some risk. The risk may be small and unimportant, or the risk may be great and the consequences catastrophic, but risk is always there (The purpose of testing is to reduce the risk of software ; both testers and developers work together to reduce risk. Level 4 The purpose of testing is to improve the ability of the developers to produce high quality software. (Level 4 testing shows thattesters and developers are on the same team)
Bezier Testing Levels Once the testers and developers are on the same team, an organization can progress to real Level 4 testing. Level 4 testing means that the purpose of testing is to improve the ability of the developers to produce high quality software.
Software Bug A software bug is an error, flaw, mistake, failure, or fault in a computer program or system Bug is terminology of Tester Softare bug produces an incorrect or unexpected result, or causes it to behave in unintended ways
Nine Causes of Software Errors 1. Faulty requirements definition 2. Client developer communication failures 3. Deliberate deviations from software requirements 4. Logical design errors 5. Coding errors 6. Noncompliance with documentation and coding instructions 7. Shortcomings of the testing process 8. User interface and procedure errors 9. Documentation errors
Cause-and-effect relationship Faults can result in errors. Errors can lead to system failures Errors are the effect of faults. Failures are the effect of errors Bug in a program is a fault. Possible incorrect values caused by this bug is an error. Possible crush of the operating system is a failure
Origins of Faults Specification Mistakes incorrect algorithms, incorrectly specified requirements (timing, power, environmental) Implementation Mistakes poor design, software coding mistakes, Component Defects manufacturing imperfections, random device defects, components wear-outs External Factors operator mistakes, radiation, lightning,
class numzero { Program Example public static int numzero (int[] x) { // if x == null throw NullPointerException else return the number of occurrences // of 0 in x return } int count = 0; for (int i = 1; i < x.length; i++) if (x[i] == 0) count++; count; }
The Fault,Error and Failure in the Example The fault is that the program starts looking for zeroes at index 1 instead of index 0 For example, numzero ([2, 7, 0]) correctly results with 1, while numzero ([0, 7, 2]) incorrectly results with 0. In both of these cases the fault is executed. Both of the cases result in an error (because of next slide) But the first case results with correct value. Only the second case results in failure.
ErrorStates in the Example To understand the error states, we need to identify the state for the program. The state for numzero consists of values for the variables x, count, i and the program counter (PC). The state at the if statement on the first iteration of the loop is ( x = [2, 7, 0], count = 0, i = 1, PC = if) This state is in error, because the value of i should be 0 on the first iteration.
Error States in the Example To understand the error states, we need to identify the state for the program. The state for numzero consists of values for the variables x, count, i,program counter (PC). For the first case: The state at the if statement on the first iteration ( x = [2, 7, 0], count = 0, i = 1, PC = if) This state is in error because the value of ishould be zero on the first iteration. But the value of count is correct, the error state does not propagate to the output, and the software does not fail. In other words, a state is in error if it is not the expected state, even if all of the values in the state are acceptable.
However Error States in the Example (for the first input) The value of count is correct The error state does not affect the output, The software does not fail. Finally: A state is in error if it is not the expected state, even if all of the values in the state are acceptable.
Error States in the Example (for the second input) For the second input: The state for numzero consists of values for the variables x, count, i and the program counter (PC). Thestate is (x = [0, 7, 2], count = 0, i = 1, PC = if) The error affects the variable count The error is present in the return value of the method. The failure results.
Distinguish Testing from Debugging The definitions of fault and failure allow us to distinguish testing from debugging. Big difference is that debugging is conducted by a programmer and the programmer fix the errors during debugging phase. Tester never fixes the errors, but rather find them and return to programmer.
Testing versus Debugging Testing activity is carried down by a team of testers, in order to find the defect in the software. Test engineers run their tests on the piece of software and if they encounter any defect (i.e. actual results don't match expected results), they report it to the development team. Testers also have to report at what point the defect occurred and what happened due the occurrence of that defect. All this information will be used by development team to DEBUG the defect.
Testing versus Debugging Debugging is the activity which is carried out by the developer. After getting the test report from the testing team about defect(s), the developer tries to find the cause of the defect. He has to go through lines of code and find which part of code in causing that defect. After finding out the bug, he tries to modify that portion of code and then he rechecks if the defect has been finally removed. After fixing the bug, developers send the software back to testers.
What is Software Quality? According to the IEEE Software Quality is: The degree to which a system, component, or process meets specified requirements. The degree to which a system, component, or process meets customer or user needs or expectations.
Importance of Software Quality Software is a major component of computer systems (about 80% of the cost) used for communication (e.g. phone system, email system) health monitoring, transportation (e.g. automobile, aeronautics), economic exchanges (e.g. ecommerce), entertainment, etc. Software defects are extremely costly in term of money reputation loss of life
Software Quality Factors
Software Quality Factors Correctness accuracy, completeness of required output datedness, availability of the information Reliability maximum failure rate Efficiency resources needed to perform software function Integrity software system security, access rights Usability ability to learn, perform required task
Software Quality Factors Maintainability effort to identify and fix software failures (modularity, documentation, etc) Flexibility degree of adaptability (to new customers, tasks, etc) Testability support for testing (e.g. log files, automatic diagnostics, etc)
Software Quality Factors Portability adaptation to other environments (hardware, software) Reusability use of software components for other projects Interoperability ability to interface with other components/systems