Testing of software and of communication systems. Richard Castanet, Bordeaux

Transcription

1 Testing of software and of communication systems Richard Castanet, LaBRI Bordeaux 40ième anniversaire du LAAS 1

2 Overview Test positioning, definitions and norms Automatization andformal methods Test of reactive systems Optimizations (test on the fly, symbolic testing) Test of critical systems with temporal constraints Robustness testing Perspectives 40ième anniversaire du LAAS 2

3 Definition et positioning Test definitions : dynamic validation method, non exhaustive Need of the test : cost of systems and human life Live cycle : Position of the test t: test tof an implementation ti Types of test Conformance testing, interoperability testing, robustness testing, performance testing Test of unities, integration testing, test of the global system, test of acceptance Structural testing (white box) and functional testing (black box) Norms : ISO9646, DO178B, CEI 60880, CENELEC EN Need of automatization: test conception and test campaign Main questions about test : selection, coverage, testability, controllability 40ième anniversaire du LAAS 3

4 Use of formal methods Formal description language (well defined semantic) Scheme : Verification Informal specification Formal specification Simulation Proof Automatic test generation Automatic code generation 40ième anniversaire du LAAS 4

5 Test of reactive systems Formal languages: SDL, LOTOS, LUSTRE, ESTEREL Based model: transition systems: Labeled Transition Systems (LTS), IOLTS (semantic for non deterministic reactive systems) (Tretmans, Jéron), automata Formal approach for the test: conformance relation : conf, ioco (traces and suspension inclusion) 40ième anniversaire du LAAS 5

6 Automatic test sequence generation Types of test: Conformance/Interoperability/Performance/Robustness/ Conformance testing : Implantation conform to a specification Interoperability: Capacity several communicating system to interoperate Test Process: Property/ behavior Abstract test suite Executable test suite Specification Verdict: Impl. Under Test Pass Inconclusive Fail 40ième anniversaire du LAAS 6

7 Conformance relation : example After a visible behavior of the specification, the implementation is only authorized for the production of specified outputs or locking?a E1?a E2!b i!c!c?d S!e!f!e?g!x!f Non specified output Partial specification 40ième anniversaire du LAAS 7

8 Test case and tester Principles of a tester for communication with the implementation: inversion of inputs outputs Imp?a!a!b!c?e?b?c!e otherwise fail?d inconc!f?f!x Pass 40ième anniversaire du LAAS 8

9 Test Architectures : interoperability testing Observation Point (OP) and Control (COP) 2 types of test : black box/ grey box Architectures t for interoperability testing: ti Testeur TesteurB TesteurA PCO PCO PCO PCO B A B A PO PO PO PO Médium Médium 40ième anniversaire du LAAS 9

10 Automatic Test Generation Methods based on automata Paths in graphs Eulerian circuits Fault model, mutant method TT, W, UIO methods Optimization by chinese postman algorithm Executability problem 40ième anniversaire du LAAS 10

11 Generation Methods based on verification and simulation Construction of the reachability graph (all behaviors) Test sequence : pathin the reachability ab graph gap combinatorial explosion enumerative methods, interlacing. need of reduction methods (test number, test sequences length, ) 40ième anniversaire du LAAS 11

12 Test generation on the fly A method to reduce the combinatorial explosion Synchronous product between a test purpose and the specification Notion of observer Production of a part of all behaviors Optimizations with determinization of the graph 40ième anniversaire du LAAS 12

13 Suspension automaton The absence of visible behaviors is modeled by an output event!δ Δ(M) = M + loops of!δ on each quiescent states Suspended traces of M: STraces(M) = Traces(Δ(M)) det(δ(m)) characterizes the visible behaviors of M.!z τ 1?a!δ 2 τ!z 1?a!δ 3 4 Determinization 234 2,3,4!x!y τ τ 8!x!y!x 5 6 7!δ Δ(M)!δ!δ 5,8,7 6!δ!δ det(δ(m))!δ 40ième anniversaire du LAAS 13

14 An example of a specific tester related to a test purpose Specification: S Reachability of Det(Δ(S)): Conformance Relation: ioco suspended trace of S accepted by TP Property: Test purpose p TP 1!z!x!δ?a!y *!x *!z Accept 4?x?δ?otherwise 2 Fail!a 3?otherwise?y?δ Inc Fail Fail!δ!δ?δ?z?otherwise Det(Δ(S)) (S)) TP Inc Pass Fail TC 40ième anniversaire du LAAS 14

15 Symbolic testing Method used with test purpose Each state is linked a descriptor giving all the constraints on the variables ibl From a state to the next one, the constraints are modified by the actions of the state Use of a constraint solver 40ième anniversaire du LAAS 15

16 Symbolic Reachability Back observation (event or delay) current estimate runs matching observation next estimate 40ième anniversaire du LAAS 16

17 Test of critical temporal systems Need of test for critical system Model for critical temporal system : temporized automata 40ième anniversaire du LAAS 17

18 Temporized automaton Temporized automaton: (S,L, C,S0,T) with : S 0 S : initial iti states, tt T S x S x L x 2 C x Φ(C ).?c, x:=0 y:=0 S0?b, x:=0 y:=0?a y:=0 x>1!d y<1,?a, y:=0 y=1,!b S3 S1 S2 x<1,!c 40ième anniversaire du LAAS 18

19 Execution of a temporized automaton?c, x:=0y:=0 0?b, x:=0y:=0 0 S0?a y:=0 x>1!d y<1,?a, y:=0 y=1,!b S3 S1 S2 x<1,!c (S0, x=y =0)? a! c? a (S1, x=0.5 y =0) (S3, x=0.75 y =0.25) t=0.5 t=0.75 t=0.85 (S1, x=0.85 y =0)! b t=1.85 (S2, x=1.85 y =1)? b t=6.85 (S0, 0=x=y )? a t=7.85 (S1, x=1 y =0)! b t=8.85 (S2, x=2 y =1) 40ième anniversaire du LAAS 19

20 Automatic test generation Discrete time or continous time Infinite states Method based on the region graph (finite graph and exponential complexity) Test on the fly Timed constraints resolution 40ième anniversaire du LAAS 20

21 Clock regions and region graph Let C ={x,y} and Φ(C C )a set of constraints on C with C x =3 and C y =1. An example for a region graph: y r1 [ (2<x<3), (0<y<x-1)] r2 [ (x=3), (0<y<1)] 1 r3 [ (x>3), (0<y<1)] r4 [ (x>3), (y=1)] r5 [ (x>3), (y>1)] x r6 [2<x<3, y=0] Temporal successor of a region: The temporal successors of r1: r2, r6, r7. 40ième anniversaire du LAAS 21

22 Region automaton Lt(S Let (S,L, C,S0,T) a temporized automaton) t State representation : <state, region>. Transition representation : <<e1,r1>, a,<e2,r2>>with: <e1, a, c, r, e2> T, r2 is a temporal successor of r1, r2 satisfies c. Ex Example with C ={x,y}, C x =1 et C y =1. y s0 1 0 a y:=0 a s0, x=y=0 s1 s1, y=0<x<1 s1, y=0 et x=1 s1, y=0 et 1<x 40ième anniversaire du LAAS 22 a a

23 1 S0 x=y=0? c? b? a? a!b 2 S1! b 3 S1! b 4 S1 5 0=y <x<1 y=0,x=1 y=0,x>1 S2 1=y <x! c? a? a 6 S3! d 7 S3! d 8 S3! d 9 S3! d 0<y <x<1 0<y <1<x 1=y <x x>1, y>1! d (S0, x=y =0)? a! c? a (S1, 0=y <x <1) (S3, 0<y <x<1) δ=0.5 δ=0.25 δ=0.1 (S1, 0=y <x<1)! b δ=1 (S2, 1=y <x)? b δ=5 (S0, 0=x=y )? a δ=1 (S1, x=1 0=y )! b δ=1 (S2, 1=y <x) 40ième anniversaire du LAAS 23

24 ?c, x:=0 y:=0 S0?b, x:=0 y:=0?a y:=0 x>1!d y <1,?a, y:=0 y=1,!b S3 S1 S2 x<1,!c (S0, x=y=0) =0)? a t=0.5 (S1, x=0.5 y=0)! c t=0.75 (S3, x=0.75 y=0.25)? a t=0.85 (S1, x=0.85 y=0)! b t=1.85 (S2, x=1.85 y=1)? b t=6.85 (S0, 0=x=y )? a t=7.85 (S1, x=1 y=0)! b t=8.85 (S2, x=2 y=1) 40ième anniversaire du LAAS 24

25 Test on the fly 40ième anniversaire du LAAS 25

26 Approaches with test purpose Main idea for test with test purpose Let : S specification of the system to be tested, O test purpose Problem: Find an execution of S drived by the test purpose. Model : temporized automaton Several approaches (conformance testing) Region graph Test purpose Proof assistant 40ième anniversaire du LAAS 26

27 Region graph approach Extraction of a test sequence Producing a trace drived by a test purpose on the region automaton 1. Sequence of transitions of the region automaton (non temporized) 2. Choice of δ : fire instants disadvantage: combinatory explosion Abstraction of the specification (temporized automaton) Equivalence of states Minimization of the region graph (partition of the state space) 40ième anniversaire du LAAS 27

28 Method with test purpose Definition: A Test purpose is an deterministic atomaton without cycle automaton and with an non empty set of special states : Accept(TP). Goal: Find a sequence of transitions of the specfication according to the test purpose. p Verdicts (Pass) : the event satisfies the Spec and the TP Pass : The event satisfies the Spec and the TP and TP is in an acceptance state Fil Fail : the event does not verify the Spec Inc : the event satisfies the Spec, but not the TP. Example of a coffee machine:? token? token!end ^ t[30,60]!end ^ t[45,60] R(t) R(t) Specification (Spec) R(t) Test Purpose(TP) R(t) 40ième anniversaire du LAAS 28

29 Synchronized product Synchronized product on the events: Rule 1: Sync : product automaton : (s1,s3) S(Sync) ^ (s1,μ,ct1,cv1,ρ1,β1,s2) T(spec) ^ (s3,μ,ct2,cv2,ρ2,β2,s4) T(Ot) (s2,s3) S(Sync)^((s1,s3),μ,Ct1,Cv1,ρ1,β1,(s2,s3)) T(Sync) Spec: A B C^t[3,6] R(t) t[2,5] R(t1) A B 1,5 2,5 3,5 TP: R(t) t[2,5] C 5 6 Product automaton R(t2) 40ième anniversaire du LAAS 29

30 Synchronized product Synchronized product: Rule 2: (s1,s3) S(Sync) ^ (s1,μ,ct1,cv1,ρ1,β1,s2) T(spec) ^ (s3,μ,ct2,cv2,ρ2,β2,s4) T(Ot) (s2,s4) S(Sync) s4) S(Sync)^((s1,s3),μ,Ct1UCt2,Cv1UCv2,ρ1Uρ2, s3) μ Cv1UCv2 β1uβ2,(s2,s4)) T(Sync) s4)) T(Sync) ^ (s2,s3) S(Sync)^((s1,s3),μ1,Ct1,Cv1,ρ1,β1,(s2,s3)) T(Sync) Spec: C^t[3,6] A B C^t[3,6] 4, R(t1) R(t) t[2,5] R(t1) A B C^t[3,6] TP: 1,5 2,5 3,5 C t[2,5] R(t1,t2) 4,6 R(t) 5 6 R(t2) Automate Produit 40ième anniversaire du LAAS 30

31 Synchronized product Temporal lsynchronisation i : (method similar than the computtion of the state classes of the temporized Petri nets ) Main idea: inequations system keeping the temporal relationship between the different clocks. a transition is aware once all the clocks in his temporal constraint have been reset. 40ième anniversaire du LAAS 31

32 Synchronized product When a transition is fired, the system of inequalities is updated in 3 stages: 1) Calculating the time remaining to make transitions sensitized 2) Remove unnecessary relations. 3) Taking into account the new transitions sensitized by resetting clocks. 40ième anniversaire du LAAS 32

33 Robustness testing Robustness definitions : IEEE : degree from one system to function properly in the presence of invalid entries or stressful environment ability to exhibit acceptable behavior in the presence of hazards hazard correct or acceptable Classification of the hazards Fault extern/intern Accidental / intentional Change use profile and charge robustness stronger than conformity or orthogonal? sometimes, no specification i of expected behavior over hazard Internal, external, out of the system Representable or no representable 40ième anniversaire du LAAS 33

34 Methods based on behavior models P : S specification, M fault model l( (set of potential ti faults and unanticipated i t events planned), P robustness property : an implantation I is robust iff I met P even in the presence of faults S // M Robustness property p degraded d ds observer product Test purpose Test case 40ième anniversaire du LAAS 34

35 Approach «increasing the specification»and refusal graph specification S : LTS, extension with unknown or incorrect event (increase the specification) Construction of the refusal graph (set of refuse in each state) Adding fault traces (sequence of actions alternating with set of refusal) Construction of the robustness tester: refusal graph + fault traces, Adding of unobservable action to go back to the initial state Test selection and coverage computation S Refusal Graph Adding fault traces Selection of test purpose Robustness tester 40ième anniversaire du LAAS 35

36 Example Construction ofthe refusalgraph a 1 a 4 Ref : {{b, c}} a Ref : {{a, b};{a, c}} b c b c Ref = {{a, b, c}} Fault traces: alternating sequences of actions and set of refusal b Construction of the robustness tester a 4 ({b, c},, ) r 5 ({a, c}, {a, b},, ) r 40ième anniversaire du LAAS 36

37 Approach based on a model on the entries Operational profile Equivalence classes Model for the nominal behavior + model of the hazards Overlay nominal activity nominal x hazards? Problem of insignificant experiences Ex: injection of a fault that will not be activated in the system analysis : Online system activity, gray box analysis Selecting relevant case in an objective verification ( evaluation) Ex: heuristic optimization to guide the research of test the most "dangerous" case ( the mostrepresentative) 40ième anniversaire du LAAS 37

38 Perspectives Improvement of the use of formal methods in industries Treatment of real, complex systems Best selection of the tests Best coverage of a test suite Combining several methods : verification, proof, 40ième anniversaire du LAAS 38

39 An example for conformance testing OT 0 [True] True,!Begin, - True,?Menv,,- Init [True] Wait [True] Sender True, -, x =S Finish [True] True,!Begin, x =S x=s (S-x)=Lambda,!End, - True,?Busy, - Transmit [(S-x)<Lambda] Paramètres: Lambda=808 Sig=26 1 [True] S-x<=2*Sig,?Busy, x =S 2 [True] True,!End, - True,? Busy, x =S True,? CD, x =S Retry [(S-x)<=2*Sig] (S-x)<Sig,?CD, x =S x=s (S-x)<=2*Sig,!Begin, x =S S-x<=2*Sig,?CD, x =S (Wait)!Begin (Transmit) S=t1?Busy (Transmit) S= ]t1,t1+lambda[!end S=t1+Lambda (Transit) 40ième anniversaire du LAAS 39

40 Calcul des intervalles de l horloge h(2) Exemple: h(t 0 )= [0,0] 0] h(t 1 )= [0,2][0,2][0,1]=[0,1] h(t 2 )= [0+2,1+5][2,7]=[2,6] h(t 3 ) =[2+2,6+6][3,13]=[4,12] h(t 4 ) =[0+3,1+6][2+3,6+7] [5,13] =[5,7] 40ième anniversaire du LAAS 40

41 Exemple Pour atteindre t Pour atteindre t 2 1 h[01 Horloge C1: 0 τ 1 2 Transition t 2: [0,1] Horloge C2: 0 τ 1 1 Horloge C1: 0 τ 1 1 s1, C1[0,1] Horloge h : 0 τ τ 2 τ 1 5 C2[0,1] RC(t 1,h) =[0,1] 2 τ 2 6 Transition t 1 Horloge C1: 0 τ 1 2 Horloge C2: 0 τ 1 1 h[0,1 ] s1, C1[0,1] C2[0,1 ] Horloge h : 0 τ 1 1 τ 1 τ 2 RC(t 1,h) =[0,1] RC(t 2,h) =[2,6] h[2 [2,6] s2, C1[2,5] 40ième anniversaire du LAAS 41

42 Exemple Pour atteindre t 4 Transition t 4: Horloge h: 5 τ 4 7 Horloge C1: 3 τ 4 τ 1 6 Horloge C2: 3 τ 4 τ 2 7 Transition t 3: Horloge h: 4 τ 3 12 Horloge C2: 2 τ 3 τ 2 6 Transition t 2: Horloge h: 2 τ 2 6 Horloge C1: 2 τ 2 τ 1 5 Transition t 1: Horloge h : 0 τ 1 1 Horloge C1: 0 τ 1 2 Horloge C2: 0 τ 1 1 Domaine de tir potentiel h[0,1] s1, C1[0,1] C 2[0,1] h[2,4] s2, C1[2,4] h[4 [4,7] s3, C1[2,5] Domaine de tir h[0,1] s1, C1[0,1] C 2[0,1] h[2,4] s2, C1[2,3] h[4 [4,7] s3, C1[2,4] τ 1 τ 2 τ 3 τ 4 RC(t 1,h) =[0,1] RC(t 2,h) =[2,4] RC(t 3,h) =[4,7] RC(t 4,h) =[5,7] h[5,7] s4, C1[4,6] C 2[3,5] h[5,7] s4, C1[5,6] 4] C 2[3,4 40ième anniversaire du LAAS 42