Ch6 Protocol Testing. Outline. Outline. Protocol Conformance Testing. Introduction Concepts Fault models Related definitions General approach

Transcription

1 Outline VI. Protocol Testing Introduction Concepts Fault models Related definitions General approach Methodologies based on FSM T-Method (Transition Tour method) D-Method d(distinguishing i sequences) W-Method (Characterizing sequences) U-Method (Unique input/output sequences) Dept. Electrical & Information Engineering, Computer Engineering Lab 6- Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6- Dr. Junzhao Sun Outline Protocol Conformance Testing Introduction Concepts Fault models Related definitions General approach Methodologies based on FSM T-Method (Transition Tour method) D-Method d(distinguishing i sequences) W-Method (Characterizing sequences) U-Method (Unique input/output sequences) To confirm if an implementation conforms to its standard External tester applies a sequence of inputs to IUT and verifies its behavior Issue: preparation of conformance tests in coverage of IUT s all aspects Issue: time required to run test should not be unacceptably long Two main limitations Controllability: the IUT cannot be directly put into a desired state, usually requiring several additional state transitions Observability: prevents the external tester from directly observing the state of the IUT, which is critical for a test to detect errors Formal conformance testing techniques based on FSM Generate a set of input sequences that will force the FSM implementation to undergo all specified transitions Black box approach: only the outputs generated by the IUT (upon receipt of inputs) are observable to the external tester Dept. Electrical & Information Engineering, Computer Engineering Lab 6-3 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6-4 Dr. Junzhao Sun

2 Fault Models Fault Model for FSM A fault model is a hypothetical model of what types of faults may occur in an implementation Most fault models are structural, t i.e. the model is a refinement of the specification formalism (or of an implementation model) E.g. mutations of the specification or of a correct implementation It may be used to construct the fault domain used for defining what complete test coverage means E.g. single fault hypothesis (or multiple faults) gp Test suite development for given coverage objective Formalization of "test purpose" For existing test suite: coverage evaluation and optimization Diagnostics A fault model is useful for the following problems: Output fault: the machine provides an output different from the one specified by the output function Transfer fault: the machine enters a different state than that specified by the transfer function Transfer faults with additional states: number of states of the system is increased by the presence of faults, additional states is used to model certain types of errors Additional or missing transitions: one basic assumption is that the FSM is deterministic and completely defined (fully specified). So the faults occur when it turns out to be non-deterministic and/or incompletely (partially) specified Dept. Electrical & Information Engineering, Computer Engineering Lab 6-5 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6-6 Dr. Junzhao Sun Fault Models for FIFO Queue and Petri Nets FSM with several FIFO input queues Ordering fault: FIFO ordering is not preserved, or in case of multiple input queues, some input event enters a wrong input queue Maximum length fault: the maximum length implemented is less than the one specified, or if an input event gets lost while queue is not overflow Flow control fault: errors of ordering or of loss occur, in case the number of submitted input events overflows the maximum queue length specified Petri Nets Input or output arc fault: one of the input or output arcs is connected to the wrong place, missing, or exists in addition to those specified Missing or additional transition: the number of transitions is not the same as in the specification FSM Related Definitions (/) Directed graph G=(V, E) representing FSM M Set of vertices V = {v, v,, v n} represents the set of states S in M Directed edge (v i, v j) E represent a transition from state s i to state s j in M An edge in G is represented by a triple (v i, v j; L), L=a k/o l is the input/output operation corresponding to the transition from s i to s j in M Some other definitions & assumptions Deterministic FSM: predictable behavior in a given state for a given input Strongly connected: for each state pair (s i, s j) there is a transition path going from s i to s j, I.e. each state can be reached from any other state Fully specified: form each state it has a transition for each input symbol. Otherwise partially specified Minimal: the number of states of M is less than or equal to the number of states of any equivalent machine Dept. Electrical & Information Engineering, Computer Engineering Lab 6-7 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6-8 Dr. Junzhao Sun

3 FSM Related Definitions (/) Transition Level Approach Start state s 0 S, usually the state when power-up Often, there is a special input taking M to state s 0 from any other state with a single transition. In this case, M is said to have the reset capability and the input which performs the reset is denoted by ri Sequences for testing A test subsequence of M is a sequence of input symbols for testing either a state or a transition of M A β-sequence for M is a concatenation of test subsequences for testing all transitions of M A test sequence for M is a sequence of input symbols which can be used in testing conformance of implementations of M against the specification of M An optimize test sequence is a test sequence such that no subsequence of it is completely contained in any other subsequence So, the problem is how to obtain a optimize test sequence for M Dept. Electrical & Information Engineering, Computer Engineering Lab 6-9 Dr. Junzhao Sun The methods for protocol conformance test sequence generation Produce a test sequence which checks the correctness of each transition of the FSM implementation By no means exhaustive, I.e. no guarantee to exhibit correct behavior given every possible input sequence. The intent is to design a test sequence which guarantees beyond a reasonable doubt Three basic steps for checking a transition (s i, s j; L), L=a k/o l Step : The FSM implementation is put into state s i; (e.g. reset+transfer) Difficulty in realizing this is due to the limited controllability of the implementation Step : Input a k is applied and the output is checked to verify that it is o l, as expected; Step 3: The new state of the FSM implementation is checked to verify that it is s j, as expected Difficulty in verifying this is due to the limited observability of the implementation Dept. Electrical & Information Engineering, Computer Engineering Lab 6-0 Dr. Junzhao Sun Outline T-Method: Transition Tour Method Introduction Concepts Fault models Related definitions General approach Methodologies based on FSM T-Method (Transition Tour method) D-Method d(distinguishing i sequences) W-Method (Characterizing sequences) U-Method (Unique input/output sequences) For a given FSM S, a transition tour is a sequence which takes the FSM S from the initial state s 0, traverses every transition at least once, and returns to the initial state s 0. Straightforward and simple scheme New state of the FSM is not checked Fault detection power Detects all output errors There is no guarantee that all transfer errors can be detected The problem of generating a minimum-cost i ttest tsequence using the transition tour method is equivalent to the so-called Chinese Postman problem in graph theory First studied by Chinese mathematician Kuan Mei-Ko ( 管梅谷 ) in 96 Dept. Electrical & Information Engineering, Computer Engineering Lab 6- Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6- Dr. Junzhao Sun

4 T-Method Example T-Method Example The specification S. A transition tour is a,a,a,b,b,b The implementation I contains an output error. Our transition tour will detect it. The implementation I contains an transition error. Our transition tour will not detect it. 5 a/z 3 4 b/z A minimum-cost transition tour of the FSM is (including reset edges), starting from state : r, a, r, c, a, b, b, r, c, r, c, a, b, r, c, a, c, b, a, c, a, r, c, b, a, a Dept. Electrical & Information Engineering, Computer Engineering Lab 6-3 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6-4 Dr. Junzhao Sun T-Method Example 3 0 A/0 4 A/0 A/0 3 B/0 A transition tour sequence is : B, A, B, A, B, A, A, A, A, A, A, A, B, B 0, 0, 3, 3, 4, 0, 3, 4,,, 4,,, (not return to 0, but traverse all the transitions) D-Method: Distinguishing Sequences A sequence of inputs is a distinguishing sequence (DS) for an FSM S, if the output sequence produced by the FSM S in response to the input sequence is distinct for each initial state A DS is used as a state identification sequence A DS is a useful tool for achieving Step 3 in checking the new state Fault detection power Detects all output errors Detects all transfer errors Two severe drawbacks In practice, very few FSMs actually possess a DS Even if an FSM does have a DS, the upper bound on the length of the DS will be too large to be useful in general The requirement is too strong (leading to W- & U- methods ) Dept. Electrical & Information Engineering, Computer Engineering Lab 6-5 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6-6 Dr. Junzhao Sun

5 D-Method Example D-Method Example The specification S. A distinguishing sequence is: b,b If we apply it from: - state, we obtain y,y - state, we obtain y,x - state 3, we obtain x,y A test case which allow the detection of the transfer error is: a,b,b,b If we apply it from the initial state of: - the specification, we obtain x,x,y,y - the implementation, we obtain x,x,x,x Definitions to be used for other methods as well. The test cases (β -sequences) are: (no reset) state t : a,b,b bb b,b,b state : b,a,b,b b,b,b,b State 3: a,a,b,b a,b,b,b Test case structure corresponding to 3-step: preamble, tested transition, state identification Transfer sequence (Preamble): the minimumcost (shortest path) input sequence taking FSM fro one state to another. Dept. Electrical & Information Engineering, Computer Engineering Lab 6-7 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6-8 Dr. Junzhao Sun D-Method Example 3 D-Method Example 4 DS = a,c,a,, 3 The test cases (β -sequences) are: r,a,c,a(-) r,a,a,c,a r,c,a,c,a r,c,a,b,r,a,c,a (-) 5 b/z r,c,a,b,a,a,c,a r,c,a,b,b,a,c,a a/z r,c,b,r,a,c,a(3-) r,c,b,a,a,c,a r,c,r,a,c,a(4-) r,c,a,a,c,a 4 r,c,b,a,c,a r,c,a,r,a,c,a(5-) r,c,a,a,a,c,a r,c,a,b,a,c,a r,c,a,c,a,c,a Including the testing for reset capability 0 A/0 4 A/0 A/0 3 B/0 DS = B, B The β -sequences are: r, A, B, B r, B, B, B r, A, A, A, A, A, B, B r, A, A, A, A, B, B, B r, A, A, A, A, B, B r, A, A, A, B, B, B r, A, A, B, B r, A, B, B, B r, A, A, A, B, B r, A, A, B, B, B An optimized test sequence constructed from above is: raaaaabbraaaabbb raaabbbraabbbrabbbrbbb Dept. Electrical & Information Engineering, Computer Engineering Lab 6-9 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6-0 Dr. Junzhao Sun

6 W-Method: Characterizing Sequences W-Method Example For FSMs that do not possess a DS, W-Method defines partial DS each of which distinguishes a state si from a subset of the remaining states instead of from every state of the FSM The states of the FSM are first partitioned into blocks which can be distinguished by observing the sequence of outputs produced by a sequence of inputs Each block is subsequently partitioned into distinguishable sub-blocks, and so on, until each block consists of exactly one state Main idea is to iteratively find a DS for each subset To identify a state (for step 3) Applying an input sequence Returning to the state via a transfer sequence Applying a second input sequence, and so on The complete set of such input sequences for an FSM is called the characterizing set Attach each CS in the set to the end of each transfer sequence 0 A/0 4 A/0 A/0 3 B/0 For the input sequence Acs = A,A, the response is identical for states and 3 (0), but is distinct from that for states 0(00), (), 4(0). Another input sequence Acs = B is distinct for states (0) and 3(). Therefore, Acs is required to identify states 0,, 4, and two input sequences Acs and Acs, along with appropriate transfer sequences, are required to identify states and 3. I.e. W = {AA, B} Dept. Electrical & Information Engineering, Computer Engineering Lab 6- Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6- Dr. Junzhao Sun W-Method Example W-Method Example 3 /0 A D 0/ 0/0 0/0 /0 /0 0/ /0 B C No DS! For the input sequence Acs = 0,,0, the response is identical i for states C and D (0), but is distinct from that for states A(000) and B(00). Another input sequence Acs =, 0 is distinct for states C(00) and D(0). Therefore, Acs is required to identify states A and B, and two input sequences Acs and Acs, along with appropriate transfer sequences, are required to identify states C and D. I.e. W = {00, 0} Dept. Electrical & Information Engineering, Computer Engineering Lab 6-3 Dr. Junzhao Sun b/e c/e b/f a/e a/f 3 a/f c/e b/f c/f A characterizing set W = {a, b} - for state : a/e, b/f - for state : a/f, b/f - for state 3: a/f, b/e The β -sequences generated are: r,a r,b r,a,a r,a,b r,b,a r,b,b r,c,a, r,c,b, r,b,a,a r,b,a,b r,b,b,a r,b,b,b r,b,c,a r,b,c,b r,c,a,a r,c,a,b r,c,b,a r,c,b,b r,c,c,a r,c,c,b Dept. Electrical & Information Engineering, Computer Engineering Lab 6-4 Dr. Junzhao Sun

7 U-Method: Unique Input/Putout Sequences U-Method Example In DS and CS, requirement of state identification is too strong Answer the question of what is the current state of the implementation? For testing it is sufficient to know an error has been detected UIO sequence of a state of a FSM An I/O behavior that is not exhibited by any other state of the FSM Answer the question of is the implementation currently in state x? Advantages against DS & CS Cost is never more than DS and in practice is usually much less (shorter) Nearly all FSMs have UIO sequences for each state DS same for all states; UIO sequence normally different for each state To check state s by using UIO sequence of s Apply input part of UIO, compare output sequence with the expected one If the same, then the FSM is in the state s; otherwise, not in the state s If not in state s, no information about the identity of the actual state s The UIO sequences are: - state :, - state :, - state 3: The β -sequences generated by U-Method are: r,a,b r,a,a r,b,a,a r,b,a,a,b r,b,b,a r,a,a,a,a r,a,b,a,b Dept. Electrical & Information Engineering, Computer Engineering Lab 6-5 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6-6 Dr. Junzhao Sun U-Method Example U-Method Example 3 3 The UIO sequences are: state :, state : state 3:,a/z state 4: b/z state 5: The minimum-cost test sequence 5 b/z generated by U-Method are: a/z r,a,c,a, a,c,a, r,c,a, (5-) c,b, a,c,a, b,b, a,b, b,a,a, r,c,a, (-) a,c,b, r,c,a, (3-) a,c,r,c,a, (4-) a,c,a,b, 4 r,c,a, (-) a,c,b,a,a Including the testing for reset capability 0 A/0 4 A/0 A/0 3 B/0 The UIO sequences are: state 0: B/null state :, state : B/0 state 3:, state 4:,A/0 The β -sequences are: r,a,b,b r,b,b r,a,a,a,a,a,a,a r,a,a,a,a,b,b r,a,a,a,a,a,a r,a,a,a,b,b,b r,a,a,a,a r,a,b,b,b r,a,a,a,b r,a,a,b,b An optimized test sequence is: raaaaaaaraaaabbraaabbbraabbrab BBrBB Dept. Electrical & Information Engineering, Computer Engineering Lab 6-7 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6-8 Dr. Junzhao Sun

8 Analysis References Fault Testing Coverage Fault coverage for D-, W-, and U-methods is better than of T-method Fault coverage for D-, W-, and du-methods are the same Summary All of these four methods assume minimal, strongly connected and fully specified Mealy FSM model of protocol entities On average, T-method produces the shortest test sequence, W-method the longest. D- and U- methods generate test sequence of comparable lengths T-method test sequences are able to detect output faults but not transition D-, W-, and U-methods are capable of detecting all kinds of faults and give the same performance. U-method attracts more and more attentions and there are several approaches based on the basic idea with some improvements R. Dssouli and F. Khendek, Test development for distributed system, 000, R. Lai, A survey of communication protocol testing. Elsevier Journal of Systems and Software, 6, 00: -46 G.V. Bochmann and A. Petrenko, Protocol testing: review of methods and relevance for software testing, Proc. ACM ISSTA, Seattle Washington, USA, 994: 09-4 A.T. Dahbura, K.K. Sabnani, and M.U. Uyar, Formal methods for generating g protocol conformance test sequences, Proceedings of the IEEE, vol. 78, no. 8, 990: D.P. Sidhu and T.-K. Leung, Formal methods for protocol testing: a detailed study, IEEE Trans. Software Engineering, vol. 5, no. 4, 989: Dept. Electrical & Information Engineering, Computer Engineering Lab 6-9 Dr. Junzhao Sun Dept. Electrical & Information Engineering, Computer Engineering Lab 6-30 Dr. Junzhao Sun