Quasimodo. under uncertainty. Alexandre David & K.G Larsen & Aalborg University, DK.

Transcription

1 Quasimodo Testing real-time systems under uncertainty Alexandre David & K.G Larsen & & Shuhaoh Li & Bi Brian Nielsen Aalborg University, DK FMCO, Graz, December 1, 2010 Page 1

2 Automated Model Based x>=2 Model DBLclick! Test Test Gene- Generator rator tool tool click? x:=0 click? x<2 Conformance Testing Selection & optimization Correctness Relation Test suite Test execution tool Adaptor Does the behavior of the (blackbox) implementation ti comply to that t of the specification? pass fail

3 Challenges of Testing Embedded Real-time systems On Timing and value tolerances Must switch within 10 ms LightLevel (90 2 lumens) Non-determinism as abstraction Internal actions and scheduling Implementation freedom Limited precision sensors and actuation Observed / produced lumination Tester has limited observability and controllability How to generate tests? for different classes of timed automata - for different classes of timed automata SUT uncertainty on actual SUT state

4 Overview of Techniques Runtime work (Uppaal) Timed Automata Technique Testsstructure Restricted Controllable Timed Automata Reachability Analysis Preset sequence Part. Obs. TA Observable TA Game solving Strategy (DAG) State-set tracking More Liberal Guided randomized

5 Conformance Relation coin? give? tea! Specification Implementation give? coin? coin? give? coin? give? coffee! Timed Automata with Timed-LTS semantics Input I actions (?) are controlled by the environment Output actions (!) are controlled by the implementation Implementations are input enabled Testing hypothesis: IUT can be modeled by some (unknown) TA

6 Timed Conformance Derived from Tretman s IOCO Quiescense observation of bounded real-time delays Let I, S be timed I/O LTS, P a set of states TTr(P): the set of timed traces from P Out(P after ) = possible outputs and delays after I rt-ioco S =def TTr(S): Out(I after ) Out(S after ) TTr(I) TTr(s) if s and I are input enabled Intuition no illegal output is produced, and required output is produced (at right time) Soundness and completeness of test generation algs. See also [Krichen&Tripakis, Khoumsi]

7 Timed Coffee Machine coin? give? tea! Timed Automata S1 I3 I4 =coin.give.7.coffee TTr(I3), TTr(S1) coffee! =coin.give.1.coffee TTr(I4), TTr(S1) out(i3 after coin.give.7)={coffee,0} out(s1 after coin.give.7)={} out(i4 after coin.give.1)={coffee,0...4} out(s1 after coin.give.1)={0...4}

8 Environment Models IUT-model C r On! Env-model Off! Low? Med? High? E M E 1 E 2 E L E 2 E 1 E M

9 Conformance relation Relativized i real-time io-conformance E 0,i 0, 1,i 1 Environment assumptions 0,o 0, 1,o 1 S System Model I IUT E,S, ES I are input enabled Timed LTS Let P be a set of states TTr(P): the set of timed traces from states in P P after = the set of states reachable after timed trace Out(P) = possible outputs and delays from states in P I rt-ioco E S = def TTr(E): Out((E,I) after ) Out((E,S) after ) I rt-ioco E s iff TTr(I) () TTr(E) TTr(S) TTr(E) // input enabled Intuition, for all assumed environment behaviors, the IUT never produces illegal output, and always produces required output in time

10 DEMO: Touch-sensitive ii Light-Controller ologi tekno tionst ormat Info Patient user: Wait= Impatient: Wait=15

11 Test Generation using Verification lightcontrol.xml System model Uppaal Model- Trace Checker (witness) Test purpose p Some Property Random E<> light.bright Shortest Fastest testlight5.trc Use trace scenario as test case??!! (s.t. purpose revealed on a correct impl.)

12 Controllable Timed I/O Automata t Inputs (?) are controllable Outputs (!) are uncontrollable output-urgent urgent deterministic isolated outputs Test Purpose: E<> light.bright 10.touch!.dim?.2.touch.bright?.pass (else FAIL) Test case is a preset sequence of timed I/O actions

13 DOUTA test generation Test purposes as reachability property Coverage of Models Location/Edge coverage, def-use pair coverage, N-switch Annotate model with aux variables Commercializable, quite large systems Many results on optimization Minimize number of resets Limit length of test cases Shortest test case Timewise fastest test case Cost-optimal (e.g energy) priced timed automata

14 Automated Model Based Conformance Testing Model (Sys Env) y x>=2 DBLclick! Test Test Gene- Generator rator tool tool click? x:=0 click? x<2 Selection & optimization Correctness Relation Test suite Test execution tool Adaptor Does the behavior of the (blackbox) implementation i comply to that of the specification? i pass fail

15 Online Testing Model (Sys Env) x>=2 DBLclick! click? x:=0 click? x<2 Test Test Generator tool Generator tool Selection & optimization Correctness Relation input output Test generated and executed event-by-event (randomly) A.K.A on-the-fly testing Test execution tool Adaptor pass fail

16 Unrestricted Timed-Automata Idea: State-set t t tracking Dynamically compute all potential states that the model M can reach after the timed trace = 0,i 0, 1,o 1, 2,i 2,o 2, [Tripakis] Failure Diagnosis Z=M after ( 0,i 0, 1,o 1, 2,i 2,o 2 ) If Z= the IUT has made a computation not in model: FAIL i is a relevant input in Env iff i EnvOutput(Z) Model states s o Z i, o, i, o, FAIL

17 (Abstract) Online Algorithm Algorithm TestGenExe (S, E, IUT, T ) returns {pass, fail) Z := {(s0, e0)}. while Z iterations T do either randomly: 1. // offer an input if EnvOutput(Z) randomly choose i EnvOutput(Z) send ito IUT Z := Z After i 2. // wait d for an output randomly choose d Delays(Z) wait (for d time units or output o at d d) if o occurred then Z := Z After d Z := Z After o // may become ( fail) else Z := Z After d // no output within d delay 3. restart: Z := {(s0, e0)}, reset IUT //reset and restart if Z = then return fail else return pass

18 (Abstract) Online Algorithm Algorithm TestGenExe (S, E, IUT, T ) returns {pass, fail) Z := {(s0, e0)}. while Z iterations T do either randomly: 1. // offer an input if EnvOutput(Z) randomly choose i EnvOutput(Z) send ito IUT Z := Z After Sound i Complete (as T ) ) 2. // wait d for an output randomly choose (Under d Delays(Z) some technical wait (for d assumptions) time units or output o at d d) if o occurred then Z := Z After d Z := Z After o // may become ( fail) else Z := Z After d // no output within d delay 3. restart: Z := {(s0, e0)}, reset IUT //reset and restart if Z = then return fail else return pass

19 Real-time Online Specification TA-network State-set explorer: maintain and analyse a set of symbolic states (zones) in real time! Z 4 ZZ 115 Z i! Z Z Z Z 14 Z Z O? Z 9 Z 6 2 Z 18 Z 15 Z 12 System Under Test

20 On-line Testing Light Controller ologi tekno mousepress mouserelease Info ormat tionst UPPAAL TRON Testing Host TRON tcp/ip Real-time Simulated time Test Fixture setlevel grasp release LightControllerGUI setlevel LightController grasp release Release 1.5

21 Industrial Application Danfoss Electronic Cooling Controller Sensor Input air temperature t sensor defrost temperature sensor (door open sensor) Keypad Input 2 buttons (~40 user settable parameters) Output t Relays compressor relay defrost relay alarm relay (fan relay) Display Output alarm / error indication mode indication current calculated temperature Optional real-time clock or LON network module

22 Controllable Timed I/O Automata t Inputs (?) are controllable Outputs (!) are uncontrollable deterministic isolated outputs output-urgent T t i t f ti d I/O ti Test case is a preset sequence of timed I/O actions Time and resource optimal tests can be generated

23 TA with Uncertainty Inputs (?)are controllable Outputs (!) are uncontrollable Tidle=20 Tsw=4 timing uncertainty of outputs multiple enabled outputs

24 Timed Game Automata [Maler, Pnueli, Sifakis 95]. Uncontrollable Controllable The controller continuously observes all delays & moves Move: controllable edge: c delay: Winning strategy: a function that tells the controller how to move in any given state to win the game: Memoryless strategy: F : State E c Reachability Games: Reach Goal Safety Games: Avoid loose

25 Timed Games a winning strategy: L0: L1: L2: L3:

26 Testing as Playing Games (the tester) Controlled only by the tester the game player stimuli reactions the game opponent Controlled only by the System Under Test (the IUT)

27 Test generation using TGA System Model Control or Test Purpose Property Uppaal-Tiga Game Solver Strategy (Directed acyclic graph) Efficient zone-based on-the fly algorithms forward algorithm with a backward fix-point computation of the winning/losing sets. Production Cell Brick Sorting Stable climate Control

28 Observable Timed Automata Tidle=20 Tsw=4 timing uncertainty of outputs Inputs (?) are controllable Outputs (!) are uncontrollable uncontrollable outputs control: A<> Bright Off-line test-case generation = Compute winning strategy for reaching Bright Assign verdicts st. lost game means IUT not conforming

29 A trick light control Tidle=20 Tsw=4 How to test for Bright? E<> (control: A<> Bright) or <<c,u>> (<<c>> ( c Bright)

30 Cooperative Strategies FAIL initial Model Statespacet possibly winning winning i Goal (pass) INCONC loosing Uppaal-Tiga extended to compute this partitioning motivated by testing applications

31 Partially Observable Systems What if : -LocationsOff and Bright can be sensed; - Dim1 and Dim2 are indistinguishable - Other locations (L1, L2): don t care; -Clock y can only be checked if y [0, 1). Strategy: If in Dim1 then bla bla bla... - We can tell it is in Dim1 or Dim2 ; but not exactly which one

32 Specifying Observations Smart Light Controller test purpose p Using a set of observable predicates: (In some location?, clocks satisfy some constraints?) e.g., { ({Off}, true), ({Dim1,Dim2}, true), ({Bright}, true), ( any, 0=<y<1) }

33 Test Generation for Partially Observable bl Systems PO-TGA models observations { ({Off}, true), ({Dim1,Dim2}, true), ({Bright}, true), ( L, 0=<y<1) <1) } Uppaal-TIGA P.O game solving test purpose Control:A<> Bright winning game strategy

34 Test Execution for Partially Observable bl Timed Systems ({Off}, true) ({Dim1, Dim2}, true) ({Bright}, true) (L, 0=<y<1) Sketch of Test Execution Algorithm: 1. If goal observation is reached, then pass ; else continue; 2. Offer input or do a delay as instructed by the strategy, until the observation changes; 3. If the new observation is allowed, then continue on, otherwise fail.

35 Results wsn-leader election node 1 node 3 buffer... node n node i Promising (but may be costly) Surprisingly P.O test generation scales better Different algorithms for game solving Finer (fully observable) vs. Coarser (partially observable) state space partitioning

36 Ongoing Work Extensons for Probabilistic Models Real-time testing of the gmac (gossip Medium Access Control)

37 END Informationsteknologi