Revisiting coverage criteria for Scade models Jean-Louis Colaço 7 December 2016

Transcription

1 Revisiting coverage criteria for Scade models Jean-Louis Colaço 7 December 2016

2 Context Code coverage is a measure that characterises how much a given test suite exercises a code, lots of criteria exist, avioncs standard (DO-178) requires MC/DC for the most critical application, in DO-178C (2011), suplement DO-331 about Model Based Design now requires model coverage. SCADE proposes model coverage for about 10 years: was based on ad hoc criteria defined by the user per operator, recent solution is inspired by work of Parissis et al. A. Lakehal and I. Parissis, Structural coverage criteria for LUSTRE/SCADE programs, in Software Testing, Verification and Reliablity, Wiley Interscience, 2009 J-L. Camus, C. Haudebourg and M. Schlickling Data Flow Model Coverage Analysis: Principles and Practice in Embedded Real Time Software and Systems, c ANSYS, Inc.

3 Why revisiting? current solution is based on Paths in the dataflow: quite complex objects; to study the relationship between model coverage and generated code coverage: paths are not well suited; to overcome some limitation of current implementation. 3 c ANSYS, Inc.

4 Why revisiting? current solution is based on Paths in the dataflow: quite complex objects; to study the relationship between model coverage and generated code coverage: paths are not well suited; to overcome some limitation of current implementation. The idea we had for the rework was actually nicely presented in: M. Whalen, G. Gay, Y. Dongjiang, M. P.E. Heimdahl and M. Staats Observable modified condition/decision coverage in Proceedings of the 35th International Conference on Software Engineering, c ANSYS, Inc.

5 Why revisiting? current solution is based on Paths in the dataflow: quite complex objects; to study the relationship between model coverage and generated code coverage: paths are not well suited; to overcome some limitation of current implementation. The idea we had for the rework was actually nicely presented in: M. Whalen, G. Gay, Y. Dongjiang, M. P.E. Heimdahl and M. Staats Observable modified condition/decision coverage in Proceedings of the 35th International Conference on Software Engineering, 2013 present work continues and extends it to full Scade 6 language. 3 c ANSYS, Inc.

6 Agenda Intuition Ideal definition of coverage Scade tagged semantics Tag based definition of coverage Static tag reduction Conclusion 4 c ANSYS, Inc.

7 Intuition Ideal definition of coverage Scade tagged semantics Tag based definition of coverage Static tag reduction Conclusion 5 Intuition c ANSYS, Inc.

8 Glossary flow or stream: infinite sequence of values. model: a Scade program and a root node. monitor: any construction that allows to observe a flow out of the model: (root node) outputs, probes,... outcome (of a test) values taken by all the monitors of the model when running a test. source designates any construction that introduces flow that that does not result from the combination of other flows. (root node) inputs, sensors, literal values, reference to constants. 6 Intuition c ANSYS, Inc.

9 The intuition Covering a stream occurrence s requires exhibiting a test that shows its ability to influence a monitor (red bubles); Covering a model requires covering all its streams occurrences. 7 Intuition c ANSYS, Inc.

10 Criterion 1: Influence A test T shows the influence of stream x of a model M if: T is such that x is in situation to influence an output of M i.e. T is such that modifying stream x in the execution of the test changes the outcome. A test suite T S covers a model M if for all stream x of M, T S contains a test T that covers stream x. 8 Intuition c ANSYS, Inc.

11 Criterion 2: OMC/DC A pair of tests (T 1, T 2 ) satisfies OMC/DC criterion for a Boolean stream b of a model M if T 1 and T 2 are such that: b takes different values in each test case and toggling b in both test cases changes the outcome. A test suite T S covers a model M in the sense of OMC/DC if for all Boolean stream b of M, T S contains two tests T 1 and T 2 such that satisfy the condition above. 9 Intuition c ANSYS, Inc.

12 Intuition Ideal definition of coverage Scade tagged semantics Tag based definition of coverage Static tag reduction Conclusion 10 Ideal definition of coverage c ANSYS, Inc.

13 Notations D n represent the set of stream prefix of size smaller or equal to n. If x is a stream prefix, x represents its size. If x is a stream prefix, (x) i where i x represents i th value. Let M be a Scade model and n in its number of inputs. A test case T of length n cycle is a tuple of n in components of D n. M(T ) represents the execution of test case T ; the outcome of this execution is itself a tuple of values in D n (one per monitor). If v is a stream prefix of a Boolean stream, i (v) represents the prefix with same length built from v by negating its i th value. A stream occurrence is represented as e k where k is an integer and e is a stream expression. 11 Ideal definition of coverage c ANSYS, Inc.

14 Occurrences identification Defined by function Streams (.): Streams (x1,..., xn = e;) def = Streams (e) def = def Streams (x) = { } x k def Streams (1) = { } 1 k Streams ( s;) def = { s k } Streams ( last s;) def = { last s k } Streams (op(e 1,..., e n)) def = { op(e 1,..., e n) k } Streams (e 1 ) def = Ideal definition of coverage c ANSYS, Inc.

15 Occurrences identification example Streams (o = x*x + pre (2*x) + 1;) = x 1, x 2, x 3, 2 4, 1 5, x 1 x 2 6, 2 4 x 3 7, pre( 2 4 x 3 ) 7 x 1 x (pre 2 4 x 3 7 x 1 x (pre 2 4 x , , 13 Ideal definition of coverage c ANSYS, Inc.

16 Stream occurrence mutation Let M be a model where: e k one of its stream occurrences: e k Streams (M), v is a finite stream prefixe: v D n, e and v are of same type, e is a stream expression with same clock as e: e e 0 e n e n+1 e n+2 v v 0 v n e v 0 v n e n+1 e n+2 M (v e k ) represents the model obtained by substituting e k in M by a e ; we called it a mutant of M for the occurrence e k. 14 Ideal definition of coverage c ANSYS, Inc.

17 Influence ideal definition Coverage of stream x by T : Influence(T, x, M) def = n > 0. v D n. M(T ) M (v x) (T ) Coverage of model M by a test suite T S : x Streams (M). T T S. Influence(T, x, M) 15 Ideal definition of coverage c ANSYS, Inc.

18 OMC/DC Ideal definition Coverage of stream x by (T 1, T 2 ): Omcdc(T 1, T 2, b, M) def = (i, j) N N. (b T1 ) i (b T2 ) j M(T1 ) M ( i (b T1 ) b) (T 1 ) M(T2 ) M ( j (b T2 ) b) (T 2 ) Coverage of model M by a test suite T S : b Streams (M). (T 1, T 2 ) T S T S. ( (b : bool) Omcdc(T 1, T 2, b, M) ) 16 Ideal definition of coverage c ANSYS, Inc.

19 Limit of the ideal definition Not really implementable: based on the exitence of mutants without giving a way to build them (it is a guess); requires both executions on original model and on the mutant; needs one mutant per stream occurrence. 17 Ideal definition of coverage c ANSYS, Inc.

20 Intuition Ideal definition of coverage Scade tagged semantics Tag based definition of coverage Static tag reduction Conclusion 18 Scade tagged semantics c ANSYS, Inc.

21 Tagged semantics Tagged semantics: based on tagged values; defines tag propagation rules. provides primitives for tag introduction; 19 Scade tagged semantics c ANSYS, Inc.

22 Tagged values The values used in a tagged Scade model M # are in V # n,m defined by: V # 0,m V # n+1,m def { } = (bool numeric declared enum values ) P(Tags) def = V n,m # { [v # 1,..., v p # ] } 1 i p m, v # i V n,m # P(Tags) { {l 1 :v # 1,..., lp:v p # } } 1 i p m, v # i V n,m # P(Tags) where Tags is a finite set of tags 20 Scade tagged semantics c ANSYS, Inc.

23 Tag propagation of combinatorial operators For most operators input tags propagate to the outputs: op # ((v 1, τ 1 ),..., (v n, τ n )) = (op(v 1,..., v n ), i [1..n] τ i) 21 Scade tagged semantics c ANSYS, Inc.

24 Tag propagation of temporal operators Behave as usual but on tagged streams: (a, τ a ) (a 0, τ0 a) (a 1, τ1 a) (a 2, τ2 a) (a 3, τ3 a) (b, τ b ) (b 0, τ0 b) (b 1, τ1 b) (b 2, τ2 b) (b 3, τ3 b) pre # (a, τ a ) (nil, ) (a 0, τ0 a) (a 1, τ1 a) (a 2, τ2 a) (a, τ a ) -> # (b, τ b ) (a 0, τ0 a) (b 1, τ1 b) (b 2, τ2 b) (b 3, τ3 b) 22 Scade tagged semantics c ANSYS, Inc.

25 Specific propagation rules and # (also exists for or # ): flow selection: a b a and # b false, τ a false, τ b false, τ a τ b false, τ a true, τ b false, τ a true, τ a false, τ b false, τ b true, τ a true, τ b true, τ a τ b if # (true, τ c ) then # (v 1, τ 1 ) else # (v 2, τ 2 ) =(v 1, τ c τ 1 ) if # ( false, τ c ) then # (v 1, τ 1 ) else # (v 2, τ 2 )=(v 2, τ c τ 2 ) 23 Scade tagged semantics c ANSYS, Inc.

26 Tags introduction sources are extended with an empty set of tags, memories are initialy extended with an empty set of tags, new primitives tag(e, t) and bool_tag(e, t 1, t 2 ) introduce tags: tag((v, τ), t) = (v, {t} τ) bool_tag((true, τ), t 1, t 2 ) = (true, {t 1 } τ) bool_tag(( false, τ), t 1, t 2 ) = ( false, {t 2 } τ) 24 Scade tagged semantics c ANSYS, Inc.

27 Tagged semantics for coverage purpose introduce a tag for each stream occurrence and register tags when reaching a monitor. 25 Scade tagged semantics c ANSYS, Inc.

28 A simple example of propagation model 26 Scade tagged semantics c ANSYS, Inc.

29 A simple example of propagation tagged model 26 Scade tagged semantics c ANSYS, Inc.

30 A simple example of propagation first cycle 26 Scade tagged semantics c ANSYS, Inc.

31 A simple example of propagation second cycle 26 Scade tagged semantics c ANSYS, Inc.

32 A simple example of propagation other cycles 26 Scade tagged semantics c ANSYS, Inc.

33 Intuition Ideal definition of coverage Scade tagged semantics Tag based definition of coverage Static tag reduction Conclusion 27 Tag based definition of coverage c ANSYS, Inc.

34 Influence tagged definition Coverage of stream x by T : Influence # (T, x, M) def = t x Otags(M # (T )) Coverage of model M by T S : x Streams (M). T T S. Influence # (T, x, M) 28 Tag based definition of coverage c ANSYS, Inc.

35 OMC/DC tagged definition Coverage of stream x by (T 1, T 2 ): def = Omcdc # (T 1, T 2, b, M) tb Otags(M# Bool (T 1)) t b Otags(M# Bool (T 2)) Coverage of model M by T S : b Streams (M). (T 1, T 2 ) T S T S. ( (b : bool) Omcdc # (T 1, T 2, b, M) ) 29 Tag based definition of coverage c ANSYS, Inc.

36 Gap with ideal definition There are situations where tags are propagated while no contribution can be observed: absorption: x * 0 unobservable selection: if c then x else x 30 Tag based definition of coverage c ANSYS, Inc.

37 Gap with ideal definition There are situations where tags are propagated while no contribution can be observed: absorption: x * 0 unobservable selection: if c then x else x Gaps exist but it still be a good compromise. 30 Tag based definition of coverage c ANSYS, Inc.

38 Intuition Ideal definition of coverage Scade tagged semantics Tag based definition of coverage Static tag reduction Conclusion 31 Static tag reduction c ANSYS, Inc.

39 Reduction Criteria are based on tags on all the expressions and sub-expressions big number of tags. Many tags are related: each time t 1 is observed t 2 is also observed. Reduction concists in removing tags whose observation can be deduced from other tags observation. Reduction is used in the model instrumentation phase. 32 Static tag reduction c ANSYS, Inc.

40 Example node N(a, b : bool ; i : i n t 1 6 ) r e t u r n s (o : i n t 1 6 ) var m : i n t 1 6 ; l e t m = pre o; o = 0 -> ( i f a and b then 2 * i e l s e i) + ( i f a or b then m / 4 e l s e m); t e l 33 Static tag reduction c ANSYS, Inc.

41 Example: initial tagging 34 Static tag reduction c ANSYS, Inc.

42 Example: initial tagging 27 tags 34 Static tag reduction c ANSYS, Inc.

43 Example: simple tag reduction 35 Static tag reduction c ANSYS, Inc.

44 Example: simple tag reduction 15 tags 35 Static tag reduction c ANSYS, Inc.

45 Example: + Boolean reduction 36 Static tag reduction c ANSYS, Inc.

46 Example: + Boolean reduction 11 tags 36 Static tag reduction c ANSYS, Inc.

47 Intuition Ideal definition of coverage Scade tagged semantics Tag based definition of coverage Static tag reduction Conclusion 37 Conclusion c ANSYS, Inc.

48 Conclusion extends to all Scade 6 language, including automata; implementation: instrumentation of the model (addition of tag(...)) and code generation for the tagged semantics; static reduction is important, divides by 2 to 3 the number of tags; good scale up (tested on big industrial models). 38 Conclusion c ANSYS, Inc.