Software Testing. 2. Models. 2. Models. Testing Approaches. Why Models? (in Testing) (Based on Part II: Models of Testing Object-Oriented Systems)

Transcription

1 2. Models (Based on Part II: Models of Testing Object-Oriented Systems) Software Testing 2. Models Models - Why? What? How? Combinational Models - Decision Tables: What? How? - Test Generation State Machines - What? (variants: Mealy & Moore) - State Transition Tables (State-to-state, Event-to-state) - The FREE State Model (State? Transition?) - Test Generation (N+) Chapter 2 2 (Smart) Poking Around Testing Approaches Brute Force Testing is a search problem Why Models? (in Testing) Models are what distinguishes craft from engineering! trillions and trillions of possible input and state combinations! only a few will reach, trigger and propagate bugs Systematic & Focused Solutions? Brute force does not work at this scale Poking around inefficient + false feeling of confidence. Model based = systematic, focused and automated - Systematic: we try every target combination (target! model) - Focused: where are bugs likely? (likely! fault model) - Automated: more tests; more repeatable test (tools! model) 3 4

2 What is a Model? (in Testing) Scope: Testing, Validation, Checking, Verification Covered in this course! ENGINEERING MODELS subject - e.g. airplanes, buildings, weather, economics TEST MODELS subject - e.g. implementation, component, under test point of view/theory - e.g. gravitation, air-pressure, law of demand and offer point of view/theory - required behaviour, where are bugs likely? representation - wire frame image, blueprint - mathematical equations representation - graphs (nodes & edges), - checklists technique - skill and expertise of modeler matters technique - experience with requirements, design, required behaviour Validation observed behaviour 5 Cartoons vs. Models CARTOONS sketching, refining, documentation - initial analysis, design, potentially (TEST) MODELS Specify precisely - what is allowed what is NOT allowed - complete - unambiguous - consistent - not integrated - different models (representing various perspectives) are coherent Completeness Checking Consistency Checking component representation yilit sab ting n spo Tes x] Re sed bo ba ck Verification (checklists, proofs) [no execution] a [Bl Implementationbased Testing [White box] component implementation 6 2. Models (Based on Part II: Models of Testing Object-Oriented Systems) Models - Why? What? How? Combinational Models - Decision Tables: What? How? - Test Generation State Machines necessarily - incomplete - ambiguous - inconsistent meta-model - What? (variants: Mealy & Moore) - State Transition Tables (State-to-state, Event-to-state) - The FREE State Model (State? Transition?) - Test Generation (N+) 7 8

3 Decision Table: example Decision Table: alternative views Annual renewal of a hypothetical car insurance policy Condition Section Variant # Claims # Insured Age Premium Increase Action Section Send Warning 1 0 <= No No 2 >= No No 3 1 <= Yes No 4 >= No No 5 2 to 4 <= Yes No 6 >= Yes No 7 5 or more Any 0 No Yes Implicit Assumptions Logical operator to interpret a row? (AND) Cancel Can a person of 15 really be insured? And what about 99? Decision Tree None 1 Number of Claims 2-4 >= 5 <= 25 Age >= 26 <= 25 Age >= 26 <= 25 Age >= 26 Cancel $50, no letter $25, no letter $100, send warning letter $50, no letter $400, send warning letter $200, send warning letter Conditi on Section Action Section Number of Claims Insured Age Premium increase Send warning easier to understand (an program) possibility to overlook a case Variant >= 5 <= 25 >= 26 <= 25 Column wise view >= 26 <= 25 >= Any No No Yes No Yes Yes No Cancel No No No No No No Yes 9 10 Combinational Models: Basic Approach Identify decision variable and conditions APPLICABLE? One of several distinct responses is to be selected according to distinct cases of input variables (finite!) - input cases are mutually exclusive response is independent of - order of input variables - prior input or output prefer small tables (combinational explosion) typical for business rules, simple protocols (locks), STEPS 1) identify the decision variables and conditions - n condition 2 n variants 2) identify the resultant actions to be selected or controlled 3) identify which actions should be produced in response to particular combinations of conditions 4) derive the logic function - via truth table - validate completeness and consistency 11 decision table with n conditions 2 n variants usually fewer variants, only explicit variants are listed Implicit variants? don t care: condition true or false, doesn t change the action - e.g.: when # claims >= 5, then age is don t care condition can t happen: condition which is assumed never to become true - e.g.: insured age < 16 or 18 (age when driving is allowed) - can sometimes occur when context changes (Ariane crash) - must be tested explicitly (sometimes is a surprise ) don t know: is an incomplete model - e.g. what happens with an age > 300? can t happen + don t know (+ don t care): typical source for bugs - specify resulting action anyway (defaults) - test for the result 12

4 Deriving the logic function: truth table Logic function: only boolean variables as input and output Expression consisting solely of AND, OR, XOR, NOT + conditions Convert decision table to truth table; truth table into expression Decision Variable Variant Condition Condition section Number of claims 0 T T F F F F F 1 F F T T F F F 2-4 F F F F T T F 5+ F F F F F F T Insured Age 25 or younger T F T F T F DC 26 or older F T F T F T DC Action Premium increase = 0 F F F F F F T section Premium increase = 25 F T F F F F F Premium increase = 50 T F F T F F F Premium increase = 100 F F T F F F F Premium increase = 200 F F F F F T F Premium increase = 400 F F F F T F F Send warning F F T F T T F Cancel F F F F F F T 13 Deriving the Logic Function: logic minimization reduce logic function to compact but equivalent expression - fewer test cases are needed - faster testing (automated) - may reveal omissions or inconsistencies Initial Development Logic Minimization Technique -- inspection s KV matrix s Cause-effect graph s Quine-McCluskey method s Starner-Dietmeyer s Exact minimization s Heuristic minimization upper limit, number of boolean variables 14 Incorrect value assigned to a decision variable Incorrect or missing operator in a predicate Incorrect or missing variable in a predicate Incorrect structure in a predicate - dangling else, misplaced semicolon, Incorrect or missing default case Incorrect or missing action(s) Extra actions Structural error in decision table implementation - falling through (i.e. a missing break in C++), incorrect value for an index or action selector Missing or incorrect class or method signature - when variants are implemented via polymorphism Generic errors Decision Table: Fault Model No empirical models about distribution of faults! - wrong version, incorrect or missing specification item, ambiguous All-Explicit Variants - Most intuitive approach - 1 test case for each explicit variant of all decision variables - example: 0 claims, 1 claim, 2-4 claims, 5+ claims, 25 or younger, 26 or older Applicable Test Generation: All-Explicit Variants - decision tables with non-binary variables - weak for undefined can t happen situations - weak when boundaries are undefined (=> don t know situation) Also apply non binary variable domain analysis 15 16

5 All-Variants, All-True, All-False, All-Primes Start from truth table (+ corresponding boolean function) All-Variants: every variant in the table is tested once - 2 n possibilities: works only for smaller tables All-True: every variant that produces TRUE - equivalent to testing all minterms of logical function - inappropriate when important actions follow from false variants All-False: every variant that produces FALSE - equivalent to testing complement of all minterms of logical function - inappropriate when important actions follow from true variants All-Primes: each prime implicant of boolean function at least once - i.e. each product term in a minimal sum-of-products form of boolean expression All-true, all-false, all-primes may miss critical bugs!! Each-Condition/All-Condition each variable is made true once, with all other variables being false + one test case where all variables are true (and logic) + one test case where all variables are false (or logic) Binary Decision Diagram Determinants Starts from the truth table - Produces a binary-decision diagram (see book p ) - Map the diagram into a determinant table (see book p ) Variable Negation Starts from sum-of-products boolean formula - unique true points: - near false points: Nonbinary Variable Domain Analysis More test generation techniques domain testing (inluding boundary values) on decision variables Decision Table Test Suite Size Inclusion Hierarchy Test Strategy Test size, Binary Decision Variables Test Size, Nonbinary Decision Variables Test Power All-Variants All-Variants 2 n 2 n * 4 n 100% Basic Variable Negation 0.1 * 2 n 0.1 * 2 n * 4n 97% Each Condition/All- Conditions m + " L(pi) (m + " L(pi)) * 4n? All-True All-False All BDD Determinants d d * 4n? All-True t t * 4n? All-False f f * 4n? Each-Condition/ All-Conditions Basic Variable Negation BDD Determinants All-Primes m m * 4n? All-Explicit x x * 4n? For non-binary decision variable we have an expression (A <= x <= B) All-Primes test x = A - 1, x = A + 1, x = B -1, x = B # tests may become smaller for ranges where x are adjacent 19 20

6 2. Models (Based on Part II: Models of Testing Object-Oriented Systems) Models - Why? What? How? Combinational Models - Decision Tables: What? How? - Test Generation State Machines - What? (variants: Mealy & Moore) - State Transition Tables (State-to-state, Event-to-state) - The FREE State Model (State? Transition?) - Test Generation (N+) = system whose output is determined by both current and past input (with Decision tables, only current input influences result) identical inputs are not always accepted identical inputs may produce different outputs State machine consists of 4 building blocks State: an abstraction that summarizes the past inputs Transition: an allowable two-state sequence - accepting and resulting state - Caused by an event; may result in an action! Guarded transition: event + guard predicate Event: an input (or an interval of time) Action: the result or output that follows an event What is a state machine? Special states: initial state, current state, final state MEALY State Machine transitions are active - may have output action - any output action may be used in more than one transition states are passive Mathematically equivalent Variants: Mealy and Moore MOORE State Machine transitions are passive states are active - may have output action - every output action has at least one state MEALY is preferred for engineering design UML allows both! - Hybrids are bad for test design! p1_winsvolley() [this.p1.score()< 20] / this.p1addpoint() Two-player Squash: Mealy State Machine p1_start()/ player1 served p1_winsvolley() [this.p1.score()== 20] / this.p1addpoint() p1_iswinner()/ return TRUE player1 won game started p1_winsvolley()/ / p2_start()/ player2 served player2 won [this.p2.score()< 20] / this.p2addpoint() [this.p2.score()== 20] / this.p2addpoint() p2_iswinner()/ return TRUE 23 24

7 Two-player Squash: Moore State Machine p1_start() Player1 serving Player1 won p1_addpoint() game started p1_winsvolley() Player1 continues p1_addpoint() p1_winsvolley() [this.p1_score() < 20] p1_winsvolley() p1_iswinner() p1_winsvolley() Answering Player1 won return TRUE; p1_winsvolley() [this.p1_score() == 20] p1_iswinner() p2_start() Player2 serving Player2 continues p2_addpoint() [this.p2_score() == 20] Player2 won p2_addpoint() p2_iswinner() Answering Player2 won return TRUE; [ ] p2_iswinner() State Transition Tables state-to-state format row = accepting state; column = resulting state cell = transition = event (with guard) + action event-to-state format row = event (with guard); column = accepting state cell = transition = action + resultant state expanded state-to-state format 2 state-to-state tables: 1 for events and 1 for actions separates the transition function and output function Example: State-to-state Example: Event-to-state Current State Game Started Player 1 served Player 2 served Player 1 won Player 2 won Game Started Resultant State/Event/Action Player 1 Served Player 2 Served Player 1 Won Player 2 Won p1_start() p1_winsvolley() [p1 score < 20] this.p1addpoint(), SimulateVolley() p1_winsvolley() p2_start() p2_winsvolley() p2_winsvolley() [p2 score < 20] this.p2addpoint(), SimulateVolley() p1_winsvolley() [p1 score == 20] p1_iswinner() return TRUE p2_winsvolley() [p2 score == 20] p2_iswinner() return TRUE Current State/Action/Next State Event Guard Game Started Player 1 Served Player 2 Served Player 1 Won Player 2 Won p1_start() p2_start() p2_winsv olley() p1_winsv olley() p1_iswinn er() DC (don t care) p2_score < 20 p2_score == 20 Player 1 Served Player 2 Served Player 2 Served this.p2addpoint(), Player 2 Served this.p2addpoint() Player 2 Served return TRUE Player 1 Won 27 28

8 Flattened Regular Expression (FREE) class under test is flattened - all inherited features are collapsed in class under test - assume type-substitution principle! instance of a class may be substituted by an instance of a subclass model behaviour as state machine restrictions to UML statechart notation few changes to UML statechart notation (e.g., alpha/omega states) - can be expressed in UML results in testable model - no ambiguities (among others precise definition of state ) - automatic generation of test cases The FREE State Model A particular subset of a class combinational value set! Represented by a State Invariant (predicate) Account.balance What is State? Class Account { private: AccountNumber number; Money balance; Date lastupdate} states Account.number Account.lastUpdate Open State Inactive State Implications What is a Transition? state invariant defines a subset of legal values for a class - state invariant is same or stronger than class invariant - method postconditions + state invariants = all states don t use hybrids (i.e. no mixing of Mealy and Moore) alpha state - null state: represent declared object before construction - accept only constructor, new, (alpha transition) - is NOT the same as the UML initial state (solid circle) omega state - null state: reached after deletion, destruction, out of scope - accepted explicit or implicit destructors (omega transition) - is NOT the same as the UML final state (bulls-eye) alpha-omega cycle Transition = 1 state invariant for accepting state 1 state invariant for resulting state 1 associated event - message sent to the class under test - interrupt or similar external control (timer, ) [optional guard expression] - predicate expression evaluated in the context of class under test 0 or more actions - message sent to a server object of the class under test - response provided by an object of the class under test 31 32

9 What should happen when a state machine in a given state receives an event not specified for this state? ignore - standard semantics for state machines - unacceptable for testing purposes omitted - incomplete specification: extend the state machine illegal (or impossible ) - illegal event = valid event, not acceptable for the current state (e.g. pop() on an empty stack) - if accepted results in an illegal transition Sneak path Unspecified Event/State Pairs - beware: guarded transitions, what if the guard is FALSE? - transition to an Illegal event exception state Response Matrix = Modified event-to-state table ROWS: list events + guards - unguarded event: 1 row - guarded event: 1 row for each unique event/guard combination! one row for each truth combination of subexpressions + additional column for all sub expressions! if event is sometimes guarded, then include a don t care column COLUMNS: list Accepting states CELLS: list responses - error codes for possible responses Response Matrix Response Matrix: Possible Responses 0. Accept (perform specified transition) 1. Queue (queue event for subsequent evaluation and ignore) 2. Ignore 3. Flag (return a non zero error code) 4. Reject (raise an exception) 5. Mute (disable the source of events and ignore) 6. Abend (invoke abnormal termination and halt process) Example: Response Matrix Accepting State/Expected Response Events Guards alpha A B C omega constructor event 1 x == 0 DC F T event 2 i > 10 k == max isreset() DC DC DC F F F F F T F T F destructor

10 s states, e events, a actions gives (s x a) es possible implementations! 5 states, 2 events and 2 actions: 10 billion possibilities! only one correct implementation Possible faults missing or incorrect transition missing or incorrect event missing or incorrect action an extra, missing or corrupt state State Based Testing: Fault Model a sneak path (a message is accepted when it shouldn t be) an illegal message failure (unexpected message causes a failure) trap door (implementation accepts undefined messages) No empirical models about distribution of faults! Verify the State Model structure check-list (Table 7.5) - structurally complete and consistent state name check-list (Table 7.6) - avoid weak, ambiguous or inappropriate state names guarded transition check-list (Table 7.7) - check logic and structure of guard expressions flattened machine check-list (Table 7.8) - free of level-to-level conflicts - consistent behaviour for inheritance tree - reuse of superclass test suites robustness check-list (Table 7.9) - safe and correct behaviour under failure modes Test Generation State Machine Coverage N+ Test Generation Features - exercise all implicit transitions to reveal sneaky paths - implementation must be able to report current state state reporter methods (available? encapsulation?) Steps - Develop a FREE model + Derive response matrix - Generate round-trip test cases via Transition Tree (see p. 249) cover all branches (= transitions) at least once identify parameter values, expected state and exception - Generate sneak path test cases cover all branches with unspecified transitions verify response + resulting state - Sensitize the transitions find input values for all guards Based on years of experience with state-based testing of hardware and telecomunications infrastructure All Transitions - every specified transition at least once All n-transition sequences - every specified transition sequence of n events All round-trip Paths - every sequence of specified transitions beginning and ending in the same state is exercised at least once M-Length Signature (when state reporter methods are absent) - signature = sequence of output actions unique for given state - verify whether we reached the given state by verifying signature 39 40

11 Conclusion Models - Why? What? How? Combinational Models - Decision Tables: What? How? - Test Generation State Machines - What? (variants: Mealy & Moore) - State Transition Tables (State-to-state, Event-to-state) - The FREE State Model (State? Transition?) alpha-omega cycle - Test Generation (N+) 41