Software Testing. 2. Models. 2. Models. Testing Approaches. Why Models? (in Testing) (Based on Part II: Models of Testing Object-Oriented Systems)

Transcription

1 2. Models (Based on Part II: Models of Testing Object-Oriented Systems) Software Testing 2. Models Models Why? What? How? Combinational Models Decision Tables: What? How? Test Generation State Machines What? (variants: Mealy & Moore) State Transition Tables (State-to-state, Event-to-state) The FREE State Model (State? Transition?) Test Generation (N+) Part III: Test Patterns Chapter 2 2 (Smart) Poking Around Testing Approaches Brute Force Why Models? (in Testing) Models are what distinguishes craft from engineering! Testing is a search problem trillions and trillions of possible input and state combinations only a few will reach, trigger and propagate bugs Systematic & Focused Solutions? Brute force does not work at this scale Poking around inefficient + false feeling of confidence. Model based = systematic, focused and automated Systematic: we try every target combination (target model) Focused: where are bugs likely? (likely fault model) Automated: more tests; more repeatable test (tools model) 3 4

2 What is a Model? (in Testing) ENGINEERING MODELS subject - e.g. airplanes, buildings, weather, economics point of view/theory - e.g. gravitation, air-pressure, law of demand and offer TEST MODELS subject - e.g. implementation, component, under test point of view/theory - required behaviour, where are bugs likely? representation - wire frame image, blueprint - mathematical equations technique - skill and expertise of modeller matters representation - graphs (nodes & edges), - checklists technique - experience with requirements, design, CARTOONS sketching, refining, documentation - initial analysis, design, coverage Scope: Testing, Validation, Checking, Verification required behaviour Validation observed behaviour Consistency Checking Verification (checklists, proofs) [no execution] a [Bl Implementationbased Testing [White box] - coherent 6 2. Models Models Why? What? How? Combinational Models Decision Tables: What? How? component representation yilit sab ting n spo Tes x] Re sed bo ba ck Different models representing various perspectives necessarily - complete - unambiguous - consistent (Based on Part II: Models of Testing Object-Oriented Systems) meta-model Completeness Checking potentially - incomplete - ambiguous - inconsistent (TEST) MODELS Specify precisely - what is allowed what is NOT allowed - not integrated 5 Covered in this course! Cartoons vs. Models component implementation 7 Test Generation State Machines What? (variants: Mealy & Moore) State Transition Tables (State-to-state, Event-to-state) The FREE State Model (State? Transition?) Test Generation (N+) Part III: Test Patterns 8

3 Decision Table: example Decision Table: alternative views Annual renewal of a hypothetical car insurance policy Condition Section Variant # Claims # Insured Age Premium Increase Action Section Send Warning 1 0 <= No No 2 >= No No 3 1 <= Yes No Cancel Decision Tree None 1 Number of Claims 2-4 >= 5 Age Age Age <= 25 >= 26 <= 25 >= 26 <= 25 >= 26 $50, no letter $25, no letter $100, send warning letter $50, no letter $400, send warning letter $200, send warning letter easier to understand (and program) possibility to overlook a case Column wise view 4 >= No No 5 2 to 4 <= Yes No Cancel Variant >= Yes No 7 5 or more Any 0 No Yes Condition Section Number of Claims Insured Age >= 5 <= 25 >= 26 <= 25 >= 26 <= 25 >= 26 Any Implicit Assumptions Logical operator to interpret a row? (AND) Can a person of 15 really be insured? And what about 99? Action Section Premium increase Send warning No No Yes No Yes Yes No Cancel No No No No No No Yes 9 10 Combinational Models: Basic Approach Identify decision variable and conditions APPLICABLE? One of several distinct responses is to be selected according to distinct cases of input variables (finite!) input cases are mutually exclusive response is independent of order of input variables prior input or output prefer small tables (combinational explosion) typical for business rules, simple protocols (locks), STEPS 1) identify the decision variables and conditions n conditions 2n variants 2) identify the resultant actions to be selected or controlled 3) identify which actions should be produced in response to particular combinations of conditions 4) derive the logic function via truth table validate completeness and consistency decision table with n conditions 2 n variants usually fewer variants, only explicit variants are listed Implicit variants? don t care: condition true or false, doesn t change the action - e.g.: when # claims >= 5, then age is don t care condition can t happen: condition which is assumed never to become true - e.g.: insured age < 16 or 18 (age when driving is allowed) - can sometimes occur when context changes (Ariane crash) - must be tested explicitly (sometimes is a surprise ) don t know: is an incomplete model - e.g. what happens with an age > 300? can t happen + don t know (+ don t care): typical source for bugs - specify resulting action anyway (defaults) - test for the result 11 12

4 Deriving the logic function: truth table Logic function: only boolean variables as input and output Expression consisting solely of AND, OR, XOR, NOT + conditions Convert decision table to truth table; truth table into expression Condition section Action section Decision Variable Variant Condition Number of claims 0 T T F F F F F 1 F F T T F F F 2-4 F F F F T T F 5+ F F F F F F T Insured Age 25 or younger T F T F T F DC 26 or older F T F T F T DC Premium increase = 0 F F F F F F T Premium increase = 25 F T F F F F F Premium increase = 50 T F F T F F F Premium increase = 100 F F T F F F F Premium increase = 200 F F F F F T F Premium increase = 400 F F F F T F F Send warning F F T F T T F Cancel F F F F F F T 13 Deriving the Logic Function: logic minimization reduce logic function to compact but equivalent expression - fewer test cases are needed - faster testing (automated) - may reveal omissions or inconsistencies Initial Development Logic Minimization Technique -- inspection s KV matrix s Cause-effect graph s Quine-McCluskey method s Starner-Dietmeyer s Exact minimization s Heuristic minimization upper limit, number of boolean variables 14 Incorrect value assigned to a decision variable Incorrect or missing operator in a predicate Incorrect or missing variable in a predicate Incorrect structure in a predicate - dangling else, misplaced semicolon, Incorrect or missing default case Incorrect or missing action(s) Extra actions Structural error in decision table implementation - falling through (i.e. a missing break in C++), incorrect value for an index or action selector Missing or incorrect class or method signature - when variants are implemented via polymorphism Generic errors Decision Table: Fault Model No empirical models about distribution of faults! - wrong version, incorrect or missing specification item, ambiguous All-Explicit Variants - Most intuitive approach - 1 test case for each explicit variant of all decision variables - example: 0 claims, 1 claim, 2-4 claims, 5+ claims, 25 or younger, 26 or older Applicable Test Generation: All-Explicit Variants - decision tables with non-binary variables - weak for undefined can t happen situations - weak when boundaries are undefined ( don t know situation) Also apply non binary variable domain analysis 15 16

5 All-Variants, All-True, All-False, All-Primes Start from truth table (+ corresponding boolean function) All-Variants: every variant in the table is tested once - 2 n possibilities: works only for smaller tables All-True: every variant that produces TRUE - equivalent to testing all minterms of logical function - inappropriate when important actions follow from false variants All-False: every variant that produces FALSE - equivalent to testing complement of all minterms of logical function - inappropriate when important actions follow from true variants All-Primes: each prime implicant of boolean function at least once - i.e. each product term in a minimal sum-of-products form of boolean expression All-true, all-false, all-primes may miss critical bugs!! Each-Condition/All-Condition each variable is made true once, with all other variables being false + one test case where all variables are true (and logic) + one test case where all variables are false (or logic) Binary Decision Diagram Determinants Starts from the truth table - Produces a binary-decision diagram (see book p ) - Map the diagram into a determinant table (see book p ) Variable Negation Starts from sum-of-products boolean formula - unique true points: - near false points: Non-binary Variable Domain Analysis More test generation techniques domain testing (including boundary values) on decision variables Decision Table Test Suite Size Inclusion Hierarchy Test Strategy Test size, Binary Decision Variables Test Size, Nonbinary Decision Variables Test Power All-Variants All-Variants 2 n 2 n * 4 n 100% Basic Variable Negation 0.1 * 2 n 0.1 * 2 n * 4n 97% Each Condition/All- Conditions m + L(pi) (m + L(pi)) * 4n? All-True All-False All BDD Determinants d d * 4n? All-True t t * 4n? All-False f f * 4n? Each-Condition/ All-Conditions Basic Variable Negation BDD Determinants All-Primes m m * 4n? All-Explicit x x * 4n? For non-binary decision variable we have an expression (A <= x <= B) All-Primes test x = A - 1, x = A + 1, x = B -1, x = B + 1 # tests may become smaller for ranges where x are adjacent 19 20

6 2. Models (Based on Part II: Models of Testing Object-Oriented Systems) Models Why? What? How? Combinational Models Decision Tables: What? How? Test Generation State Machines What? (variants: Mealy & Moore) State Transition Tables (State-to-state, Event-to-state) The FREE State Model (State? Transition?) Test Generation (N+) Part III: Test Patterns What is a state machine? = system whose output is determined by both current and past input (with Decision tables, only current input influences result) identical inputs are not always accepted identical inputs may produce different outputs State machine consists of 4 building blocks State: an abstraction that summarizes the past inputs Transition: an allowable two-state sequence accepting and resulting state Caused by an event; may result in an action Guarded transition: event + guard predicate Event: an input (or an interval of time) Action: the result or output that follows an event Special states: initial state, current state, final state MEALY State Machine transitions are active - may have output action - any output action may be used in more than one transition states are passive Mathematically equivalent Variants: Mealy and Moore MOORE State Machine transitions are passive states are active - may have output action - every output action has at least one state MEALY is preferred for engineering design UML allows both! - Hybrids are bad for test design! p1_winsvolley() [this.p1.score()< 20] / this.p1addpoint() Two-player Squash: Mealy State Machine p1_start()/ player1 served p1_winsvolley() [this.p1.score()== 20] / this.p1addpoint() p1_iswinner()/ return TRUE player1 won game started p1_winsvolley()/ / p2_start()/ player2 served player2 won [this.p2.score()< 20] / this.p2addpoint() [this.p2.score()== 20] / this.p2addpoint() p2_iswinner()/ return TRUE 23 24

7 p1_winsvolley() Two-player Squash: Moore State Machine p1_start() Player1 serving Player1 continues p1_addpoint() game started p1_winsvolley() [this.p1_score() < 20] p1_winsvolley() [this.p1_score() == 20] Player1 won p1_addpoint() p1_iswinner() p1_winsvolley() Answering Player1 won return TRUE; p1_winsvolley() p1_iswinner() p2_start() Player2 serving Player2 continues p2_addpoint() Player2 won p2_addpoint() p2_iswinner() Answering Player2 won return TRUE; [ ] [this.p2_score() == 20] p2_iswinner() State Transition Tables state-to-state format row = accepting state; column = resulting state cell = transition = event (with guard) + action event-to-state format row = event (with guard); column = accepting state cell = transition = action + resultant state expanded state-to-state format 2 state-to-state tables: 1 for events and 1 for actions separates the transition function and output function Example: State-to-state Example: Event-to-state Current State Game Started Player 1 served Player 2 served Player 1 won Player 2 won Game Started Resultant State/Event/Action Player 1 Served Player 2 Served Player 1 Won Player 2 Won p1_start() p1_winsvolley() [p1 score < 20] this.p1addpoint(), SimulateVolley() p1_winsvolley() p2_start() p2_winsvolley() p2_winsvolley() [p2 score < 20] this.p2addpoint(), SimulateVolley() p1_winsvolley() [p1 score == 20] p1_iswinner() return TRUE p2_winsvolley() [p2 score == 20] p2_iswinner() return TRUE Current State/Action/Next State Event Guard Game Started Player 1 Served Player 2 Served Player 1 Won Player 2 Won p1_start() p2_start() p2_winsv olley() p1_winsv olley() p1_iswinn er() DC (don t care) p2_score < 20 p2_score == 20 Player 1 Served Player 2 Served Player 2 Served this.p2addpoint(), Player 2 Served this.p2addpoint() Player 2 Served return TRUE Player 1 Won 27 28

8 The FREE State Model Flattened Regular Expression (FREE) class under test is flattened all inherited features are collapsed in class under test assume type-substitution principle instance of a class may be substituted by an instance of a subclass model behaviour as state machine What is State? A particular subset of a class combinational value set Represented by a State Invariant (predicate) Class Account { private: AccountNumber number; Money balance; Date lastupdate} Account.balance restrictions to UML statechart notation few changes to UML statechart notation (e.g., alpha/omega states) can be expressed in UML results in testable model no ambiguities (among others precise definition of state ) automatic generation of test cases states Account.lastUpdate Account.number Open State Inactive State state invariant defines a subset of legal values for a class state invariant is same or stronger than class invariant method postconditions + state invariants = all states don t use hybrids (i.e. no mixing of Mealy and Moore) alpha state null state: represent declared object before construction accept only constructor, new, (alpha transition) is NOT the same as the UML initial state (solid circle) omega state Implications null state: reached after deletion, destruction, out of scope accepted explicit or implicit destructors (omega transition) is NOT the same as the UML final state (bulls-eye) alpha-omega cycle What is a Transition? Transition = 1 state invariant for accepting state 1 state invariant for resulting state 1 associated event message sent to the class under test interrupt or similar external control (timer, ) [optional guard expression] predicate expression evaluated in the context of class under test 0 or more actions message sent to a server object of the class under test response provided by an object of the class under test 31 32

9 What should happen when a state machine in a given state receives an event not specified for this state? ignore standard semantics for state machines unacceptable for testing purposes omitted incomplete specification: extend the state machine illegal (or impossible ) illegal event = valid event, not acceptable for the current state (e.g. pop() on an empty stack) if accepted results in an illegal transition Sneak path Unspecified Event/State Pairs beware: guarded transitions, what if the guard is FALSE? transition to an Illegal event exception state Response Matrix = Modified event-to-state table ROWS: list events + guards unguarded event: 1 row guarded event: 1 row for each unique event/guard combination one row for each truth combination of subexpressions + additional column for all sub expressions if event is sometimes guarded, then include a don t care column COLUMNS: list Accepting states CELLS: list responses error codes for possible responses Response Matrix Response Matrix: Possible Responses 0. Accept (perform specified transition) 1. Queue (queue event for subsequent evaluation and ignore) 2. Ignore 3. Flag (return a non zero error code) 4. Reject (raise an exception) 5. Mute (disable the source of events and ignore) 6. Abend (invoke abnormal termination and halt process) Example: Response Matrix Accepting State/Expected Response Events Guards alpha A B C omega constructor event 1 x == 0 DC F T event 2 i > 10 k == max isreset() DC DC DC F F F F F T F T F destructor

10 s states, e events, a actions gives (s x a) es possible implementations 5 states, 2 events and 2 actions: 10 billion possibilities only one correct implementation Possible faults missing or incorrect transition missing or incorrect event missing or incorrect action an extra, missing or corrupt state State Based Testing: Fault Model a sneak path (a message is accepted when it shouldn t be) an illegal message failure (unexpected message causes a failure) trap door (implementation accepts undefined messages) No empirical models about distribution of faults! Verify the State Model structure check-list (Table 7.5) structurally complete and consistent state name check-list (Table 7.6) avoid weak, ambiguous or inappropriate state names guarded transition check-list (Table 7.7) check logic and structure of guard expressions flattened machine check-list (Table 7.8) free of level-to-level conflicts consistent behaviour for inheritance tree reuse of superclass test suites robustness check-list (Table 7.9) safe and correct behaviour under failure modes Test Generation State Machine Coverage N+ Test Generation Features exercise all implicit transitions to reveal sneaky paths implementation must be able to report current state state reporter methods (available? encapsulation?) Steps Develop a FREE model + Derive response matrix Generate round-trip test cases via Transition Tree (see p. 249) cover all branches (= transitions) at least once identify parameter values, expected state and exception Generate sneak path test cases cover all branches with unspecified transitions verify response + resulting state Sensitize the transitions find input values for all guards Based on years of experience with state-based testing of hardware and telecommunications infrastructure All Transitions every specified transition at least once All n-transition sequences every specified transition sequence of n events All round-trip Paths every sequence of specified transitions beginning and ending in the same state is exercised at least once M-Length Signature (when state reporter methods are absent) signature = sequence of output actions unique for given state verify whether we reached the given state by verifying signature 39 40

11 2. Models (Based on Part II: Models of Testing Object-Oriented Systems) Models Why? What? How? Combinational Models Decision Tables: What? How? Test Generation State Machines What? (variants: Mealy & Moore) State Transition Tables (State-to-state, Event-to-state) The FREE State Model (State? Transition?) Test Generation (N+) Part III: Test Patterns (Test) Patterns pattern = generalized solution to a specific recurring (design) problem capture the essence of proven solutions pattern template includes name (design discussions) context when to apply, when NOT to apply forces: different interests pull towards various directions trade-offs advantages and disadvantages in terms of forces known uses Test Pattern Template Test Documentation name intent context (incl. test scope) fault model what kind of faults are the target? motivate: why reach faults + trigger and propagate failures strategy how to design and implement the test suite procedure, oracle, automation example entry criteria preconditions for using this pattern when is the Implementation Under Test (IUT) test ready? exit criteria defines the results necessary to achieve an adequate test (See also IEEE standard 829) prepare a test plan what to test when deliverables stop criteria design the test suite test suite hierarchy (i.e. V-model) procedures for running the tests prepare test cases (using appropriate test pattern) test the component results for each test case: pass / no pass / unable to execute store results (and follow-up action) for later reference prepare test summary report Evaluate the operability of the IUT 43 44

12 Conclusion Models Why? What? How? Combinational Models Decision Tables: What? How? Test Generation State Machines What? (variants: Mealy & Moore) State Transition Tables (State-to-state, Event-to-state) The FREE State Model (State? Transition?) alpha-omega cycle Test Generation (N+) Part III: Test Patterns Patterns are NOT a good vehicle for Analytical comparison (and contrasting) Presenting a general model, theory, point of view Defining terms + conceptual relationship Lengthy and complex system of ideas I want you to teach some of the material & serve as opponent for your colleagues Will force you to acquire deep understanding In depth understanding of parts is preferred over broad understanding of the whole Patterns are not a Panacea Tips Preparing slides for a 2-hour lecture takes 8 hours! scope of the lecture is known message (what do I want the audience to remember?) story (how will I sequence the material from known to unknown?) Avoid death by bullet points use tables / schemas / animation / slides are a tool (not a goal in themselves) Passionate lecturing use personal anecdotes, examples, Examples? Industrial Guest Lectures! relate to your audience 47