Specification-based Testing for Refinement

Transcription

1 Specification-based Testing for Refinement T. Kahsai 1 M. Roggenbach 1 H. Schlingloff 2 1 Swansea University, Wales 2 Humboldt University Berlin / Fraunhofer FIRST, Germany SEFM London

2

3

4

5

6 Overview A specification exercise Colouring test processes Test execution Vertical development Example revisted

7 A specification exercise: Binary Calculator

8 Consequences out of the process part - I ccspec BCALC0 = data sort Number ops 0, 1 : Number; + : Number Number? Number channel Button, Display : Number process P 0 = (?x : Button P 0 ) (!y : Display P 0 ) end

9 Consequences out of the process part - I ccspec BCALC0 = data sort Number ops 0, 1 : Number; + : Number Number? Number channel Button, Display : Number process P 0 = (?x : Button P 0 ) (!y : Display P 0 ) end Button!0 Button!0 is left open behaviour.

10 Consequences out of the process part - ccspec BCALC0 = data sort Number ops 0, 1 : Number; + : Number Number? Number channel Button, Display : Number process II end P 1 =?x : Button!y : Display P 1

11 Consequences out of the process part - ccspec BCALC0 = data sort Number ops 0, 1 : Number; + : Number Number? Number channel Button, Display : Number process II end P 1 =?x : Button!y : Display P 1 Button!0 Button!0 is a unwanted behaviour.

12 Consequences out of the data part - I ccspec BCALC2 = data sort Number ops 0, 1 : Number; + : Number Number? Number channel Button, Display : Number process P 2 end =?x : Button Display!x?y : Button Display!(x + y) P 2

13 Consequences out of the data part - I ccspec BCALC2 = data sort Number ops 0, 1 : Number; + : Number Number? Number channel Button, Display : Number process P 2 end =?x : Button Display!x?y : Button Display!(x + y) P 2 Button!0 Display!0 Button!1 Display!1 is left open behaviour.

14 Consequences out of the data part - II ccspec BCALC2 = data CARDINAL [op WordLength = 1 : Nat] with sort CARDINAL Number channel Button, Display : Number process P 2 end =?x : Button Display!x?y : Button Display!(x + y) P 2

15 Consequences out of the data part - ccspec BCALC2 = data CARDINAL [op WordLength = 1 : Nat] with sort CARDINAL Number channel Button, Display : Number process P 2 end =?x : Button Display!x?y : Button Display!(x + y) P 2 II Button!0 Display!0 Button!1 Display!1 is intended behaviour.

16 Colouring test processes

17 Test case Given: (Sp, P) CSP-CASL specification A test case T is any CSP-CASL process in the signature of Sp.

18 Test case Given: (Sp, P) CSP-CASL specification A test case T is any CSP-CASL process in the signature of Sp. e.g. Button!0 Display!0 Button!1 Display!1 Remark: In the paper also variables.

19 Colouring test processes The colour of test T with respect to (Sp, P) is a value in {red, yellow, green}.

20 Formal definition of colouring Test processes colour(t) = green M Mod(Sp) and all variable evaluations ν : X M it holds that: 1. traces([t ] ν ) traces([p] : β(m) ) and 2. for all tr = t 1,... t n traces([t ] ν ) and for all 1 i n it holds that: ( t 1,..., t i 1, {t i }) / failures([p] : β(m) ) colour(t) = red iff for all models M Mod(Sp) and all variable evaluations ν : X M it holds that: traces([t ] ν ) traces([p] : β(m) ) colour(t) = yellow otherwise.

21 So far: Expected result of a test process with respect (Sp, P).

22 Test execution

23 How to execute a test case w.r.t. particular SUT?

24 Point of control and observation (PCO) Given: System under test (SUT) A PCO P = (A,..., D) of an SUT consists of: an alphabet A of primitive events a mapping... : A T Σ a direction D : A {ts2sut, sut2ts}.

25 Point of control and observation (PCO) Given: System under test (SUT) A PCO P = (A,..., D) of an SUT consists of: an alphabet A of primitive events a mapping... : A T Σ a direction D : A {ts2sut, sut2ts}. Calculator example : A = {button 0, button 1, display 0, display 1 } button 0 = Button.0, button 1 = Button.1... ts2sut : button events sut2ts display events.

26 Test Verdict A test case T is executable w.r.t. PCO P and (Sp, P) condition: unique coverage of the terms in the test by primitive events. The verdict of test T w.r.t. (Sp, P) and a particular SUT is a value in {pass, fail, inconclusive}. Pass increased confidence in SUT w.r.t. (SP, P) Fail violation of the intentions described in (Sp, P) Inconclusive neither increased nor destroyed confidence Remark: On the paper an algorithm to derive the test verdict.

27 Overall testing framework

28 Support for vertical development

29 Well-behaved refinement Let (Sp, P) (Sp, P ) be a refinement with certain properties is called well-behaved (w-b), iff : 1. colour(t) = green with respect to (Sp, P) implies colour(t) = green with respect to (Sp, P ), and 2. colour(t) = red with respect to (Sp, P) implies colour(t) = red with respect to (Sp, P ).

30 Colouring and verdict under w-b refinement

31 Well-behaved refinement relations Refinement relation Data refinement + Process refinement over T model Process refinement over F model Process refinement over N model Process refinement over R model Well behaved X + it works with F, N, R requires divergence-free processes.

32 Calculator example revisited

33 Test colouring P 0 = (?x : Button P 0 ) (!y : Display P 0 ) = Display!1 Stop T 0 colour(t 0 ) = yellow

34 Test colouring P 1 =?x : Button!y : Display P 1 = Display!1 Stop T 0 colour(t 0 ) = red

35 Test colouring sort Number ops 0, 1 : Number; + : Number Number? Number P 2 =?x : Button Display!x?y : Button Display!(x + y) P 2 = Button!0 Display!0 Button!1 Display!1 Stop T 1 colour(t 1 ) = yellow

36 Test colouring CARDINAL [op WordLength = 1 : Nat] with sort CARDINAL Number P 3 =?x : Button Display!x?y : Button Display!(x + y) P 3 = Button!0 Display!0 Button!1 Display!1 Stop T 1 colour(t 1 ) = green Colouring can be verified using:

37 Test colouring Screen-shot of CSP-CASL-Prover :

38 Summary and Future work

39 Summary Theory for the evaluation of test cases with respect to formal specification. Separation of test oracle & test evaluation by defining expected result and test verdict. 3-valued colouring scheme for expected result = specification transformation and refinement. 3-valued evaluation scheme for test verdict = tests at all stages in a system s design.

40 Future work Test generation and test coverage. Investigate Enhancement & formal testing. e.g. Re-use test suites for new products in a product line Case study of industrial strength.

41

42 Syntactic characterisation - (first condition) check T = ((P T)[[R s1 ]]... [[R sh ]] [A ] count(n)) \ s 1 \ \ s n count(n : Nat) = if n = 0 then Ok Stop else count(k 1)

43 Process ref. over T is not well behaved DOONEA T DONOTHING ccspec DOONEA = data sort s op a : s process a Stop end ccspec DONOTHING = data sort s op a : s process Stop end T = a Stop execution check(p DoOneA, T) = Ok Stop check(p DoNothing, T) Ok Stop verdict green red