T-76.5613 Software Testing and Quality Assurance Lecture 4, 20.9.2006 Black-box Testing Techniques SoberIT
Black-box test case design techniques Basic techniques Equivalence partitioning Boundary value analysis Function testing Testing heuristics State transition testing Specification testing Error guessing and Failure modes Scenario testing Soap opera testing 2
Equivalence partitioning and boundary value analysis (i.e. domain testing) To avoid unnecessary testing, partition (divide) the range of inputs into groups of equivalent tests. Then treat an input value from the equivalence class as representative of the full group. We treat two tests as equivalent if they are so similar to each other that it seems pointless to test both. Boundaries mark the point or zone of transition from one equivalence class to another. These are good members of equivalence classes to use because the program is more likely to fail at a boundary. invalid valid invalid 0 1 100 101 3
Why not test only the boundaries? If you do boundaries only, you have covered all the partitions as well technically correct and may be OK if everything works correctly boundaries can be analyzed wrongly, there can be hidden boundaries if the test fails, is the whole partition wrong, or is a boundary in the wrong place have to test mid-partition anyway testing only extremes may not give confidence for typical use scenarios (especially for business people or users) boundaries may be harder (more costly) to set up 4
Exercise: Design test cases Use equivalence classes and boundaries 5
Some test cases for print dialog All: just one slide, no slides, many slides Current slide: first, last, middle, selected, not selected Selection: no selection, one slide, many slides, discontinuous selection, all slides, Slides: One, first, last Semicolon separated, two, many, consecutive not consecutive Normal ranges, open ranges (e.g. -5 or 4- ), all slides, multiple ranges, duplicate and intersecting ranges Ranges that exceed the number of slides, etc. Invalid slide numbers Empty, invalid strings Slides per page: One, two, four, six, odd number, big even number Different amount of slides in the presentation (one, odd, even) 6
Function testing Testing one function at a time Functional unit testing Can be performed based on various information Going through the user interface or other interface commands User manual (draft) Functional specifications Various techniques can be utilized for testing each function Equivalence classes, boundary values, combinations, 7
Function testing tasks Build a list of functions Identify each function and sub-function Determine how to recognize correct behaviour Identify variables, equivalence classes, and boundaries. Identify environmental variables that may constrain the function under test Check that the function does what it's supposed to do and does not do what it's not supposed to do 8
Testing individual functions Variables Outputs of the function Possible scope of the function E.g. apply to selection, page, whole document Options of the function Settings, configuration, etc. Circumstances under which the function behaves differently Modes, or status of the system Different ways of performing the same function Other functions that interact with the tested function 9
Benefits of function testing Functional coverage Cover every feature and option Functional specification or function list Useful for initial testing of the product Introduction to the capabilities of the product Reveals serious problems that should be addressed early 10
Shortcomings of function testing as a primary method Misses interaction and relationships between functions Feature interactions misses exploration of the benefits offered by the program Is not necessarily powerful for validation Whether the software actually solves the customer s problem Whether the users can actually achieve the benefits Emphasizes isolated testing of individual items Load, performance, reliability, etc. not covered Easily focuses on the typical cases without considering boundaries, special cases or tests of special data 11
Test Heuristics A heuristic A rule of thumb A way of making an educated guess Useful rules that are not guaranteed to lead to the right result Heuristics can be used to generate ides for good tests Heuristics are suggestions that must be used wisely Do not blindly follow heuristics without understanding them Experienced testers collect heuristics that help them make good guesses when designing tests 12
Examples of general testing heuristics Test at the boundaries Test every error message Test configurations that are different from the programmers Run tests that are annoying to set up Avoid redundant tests 13
CICCUPP Consistency heuristics help recognizing defects in software Consistent with History: Present function behavior is consistent with past behavior. Consistent with our Image: Function behavior is consistent with an image that the organization wants to project. Consistent with Comparable Products: Function behavior is consistent with that of similar functions in comparable products. Consistent with Claims: Function behavior is consistent with what people say it s supposed to be. Consistent with User s Expectations: Function behavior is consistent with what we think users want. Consistent within Product: Function behavior is consistent with behavior of comparable functions or functional patterns within the product. Consistent with Purpose: Function behavior is consistent with apparent purpose. Suorce: Black Box Software Testing, Cem Kaner 14
State transition testing Uses a model that shows: states the software may occupy transitions between the states events which cause the transitions actions that result from the transitions Card inserted Ask for PIN Invalid PIN Ask for PIN Cancel Return card Wait for PIN Valid PIN Ask amount 15
State transition table Current state Event Action Next state Card inserted Ask for PIN Wait for PIN Cancel Return card Valid PIN Ask amount Wait for Amount Invalid PIN Ask for PIN 16
State transition testing coverage criteria Simple criteria Cover all states Cover all events Cover all transitions Cover all paths (infinite if loops possible) Switch coverage All single transitions (0-switch) Transition pairs (1-switch) Transition triples (2-switch) etc. A more complete test set will test for possible invalid transitions use state transition table to identify invalid transitions 17
Current state Event Action Next state Card inserted Ask for PIN Cancel State transition table with invalid transitions Valid PIN Invalid PIN Valid amount Sufficient balance Insufficient balance Card inserted Cancel Return card Valid PIN Ask amount Invalid PIN Ask for PIN Valid amount Sufficient balance Insufficient balance Card inserted Cancel Valid PIN Invalid PIN Valid amount Get amount Check balance Sufficient balance Insufficient balance Check balance Card inserted Check balance Check balance Cancel Check balance Check balance Valid PIN Check balance Check balance Invalid PIN Check balance Check balance Valid amount Check balance Check balance Sufficient balance Give money Dispense notes Check balance Insufficient balance Return card 18
Specification testing Testing based on some sort of specification document Requirements specification Functional specification User manual Etc. Discovering what claims are being made testing them against the product 19
Specification testing examples Using context-free questions Who? What? When? Where? How? Why? How does the spec say what it says Ambiguity Are multiple interpretations possible? Adequacy Does it provide enough information for programming, documentation and testing? Completeness To what extent does it cover the feature set, use cases, etc.? Whenever there is ambiguity, there is a strong opportunity for a defect 20
Error guessing and Failure model analysis Always worth including Complements systematic techniques Can find some faults that systematic techniques can miss Is based on experience and previous or typical defects 21
Error guessing: deriving test cases People are not that imaginative about committing bugs They tend to be the same bugs as before Learning from typical errors Consider Past failures Intuition Experience Brain storming What is the craziest thing we can do? Lists and failure modes in literature, defect catalogs Create a defect catalog of your own List the most typical defects in your context 22
Test Catalogs Test catalogs are refined defect catalogs Test catalog is a check list of typical tests for a certain situation Based on experience on typical errors that developers make Textual input/numbers the largest number that is supposed to be accepted the smallest number that is supposed to be accepted one larger than the largest number that is supposed to be accepted one smaller than the smallest number that is supposed to be accepted Machine boundaries 127 128 (signed byte overflow) 255 256 (unsigned byte overflow) 32767 (32 767) 32768 (32 768) (signed 16-bit overflow) 65535 (65 535) 65536 (65 536) (unsigned 16-bit overflow) 2147483647 (2 147 483 647) 2147483648 (2 147 483 648) (signed 32-bit word overflow) 4294967295 (4 294 967 295) 4294967296 (4 294 967 296) (unsigned 32-bit word overflow.) 23
Example of Failure Model Testing James A. Whittaker: How to Break Software User interface attacks Apply inputs that force all error messages Apply inputs that force the software to establish default values Overflow input buffers Force data structure to store too many or too few values Investigate alternative ways to modify internal data constraints Etc. System interface attacks Fill the system to its capacity Force media to be busy or unavailable Vary file permissions Vary file content or corrupt file Etc. 24
Scenario testing A scenario is a hypothetical story, used to help a person think through a complex problem or system. Scenario Testing is a testing approach that is based on scenarios. Testing scenarios are complicated and realistic stories of real usage of the system Scenarios are carefully designed and documented Scenarios can be executed manually or automated The goal is to focus on business needs and realistic use situations, instead of technical details and mechanical testing 25
Good Testing Scenarios Based on a story about how the program is used motivations of the people involved. Motivating A stakeholder with influence would push to fix a revealed defect Credible Not only could happen in the real world; stakeholders would believe that something like it probably will happen Complex Complex use of the program Complex environment Complex set of data. Results are easy to evaluate Especially important for scenarios because they are complex Cem Kaner. An Introduction to Scenario Testing, STQE, October, 2003. http://www.kaner.com/pdfs/scenariointrover4.pdf 26
Scenarios can be used for Learn the product Connect testing to documented requirements Bring requirements-related issues to the surface Expose failures to deliver desired benefits Explore expert use of a program Make a bug report more motivating Cem Kaner. An Introduction to Scenario Testing, STQE, October, 2003. http://www.kaner.com/pdfs/scenariointrover4.pdf 27
Soap Opera Testing Extreme Scenario Testing Build a scenario based on real-life experience Exaggerate each aspect of it for each variable, substitute a more extreme value if a scenario can include a repeating element, repeat it lots of times Make environment more hostile increase or decrease memory, printer resolution, video resolution, etc. Make the situation as complicated as possible Create a real-life story that combines all of the elements into a test case narrative. 28