Model-Based Testing. with TorXakis. TestNet WerkGroep Model-Based testing. Jan Tretmans Piërre van de Laar. model. TorXakis.

Transcription

1 Model-Based Testing with TorXakis model TestNet WerkGroep Model-Based testing Jan Tretmans Piërre van de Laar TNO Embedded Systems Innovation TorXakis pass fail system under test

2 Model-Based Testing 2

3 Model-Based Testing MBT next step in test automation model-based test generation model + test generation + result analysis test execution system pass fail 3

4 Software & System Testing Checking or measuring some quality characteristics of an executing software object by performing experiments in a controlled way w.r.t. a specification tester specification-based, active, black-box testing of functionality specification SUT System Under Test 4

5 1 : Manual Testing 1. Manual testing SUT pass fail System Under Test 5

6 2 : Scripted Testing test cases TTCN 1. Manual testing 2. Scripted testing test execution SUT pass fail 6

7 3 : Keyword-Driven Testing high-level test notation test scripts 1. Manual testing 2. Scripted testing 3. Keyword-driven testing test execution SUT pass fail 7

8 4 : Model-Based Testing model-based TTCN Test test generation cases system model 1. Manual testing 2. Scripted testing 3. Keyword-driven testing 4. Model-based testing test execution pass fail SUT 8

9 MBT : Example Models!coffee?coin!alarm?button?button 9

10 MBT : Validation, Verification, Testing informal requirements validation informal world formal world model verification real world model-based testing SUT 10

11 MBT : Ingredients off-line MBT model-based test generation system model requirements ideas test cases TTCN test execution SUT SUT verdict test result analysis pass fail test harness 11

12 MBT : Ingredients requirements ideas on-the-fly MBT model based test generation + execution system model SUT verdict test result analysis pass fail test harness 12

13 MBT : Many Tools AETG Agatha Agedis Autolink Axini Test Manager Conformiq Cooper Cover DTM G st Gotcha Graphwalker JTorX MaTeLo MBT suite M-Frame MISTA NModel OSMO ParTeG Phact/The Kit QuickCheck Reactis Recover RT-Tester SaMsTaG Smartesting CertifyIt Spec Explorer Statemate STG Temppo TestGen (Stirling) TestGen (INT) TestComposer TestOptimal TGV Tigris TorX TorXakis T-Vec Uppaal-Cover Uppaal-Tron Tveda

14 14 Model-Based Testing with TorXakis

15 15 Yet Another MBT Tool : TorXakis JTorX TorXakis TorX

16 Testing : Trends & Challenges complexity size heterogeneous components model-driven development, continuous * Model Based Testing uncertainty quest for quality systems-of-systems 16

17 MBT : Next Generation Challenges state + complex data heterogeneous components model composition model-driven test development, criteria continuous * selection abstraction complexity size Model Based Testing concurrency parallelism quest for quality usage profiles for testing scalability link to MBSD systems-of-systems multiple paradigms integration uncertainty uncertainty nondeterminism 17

18 MBT : Next Generation Challenges abstraction concurrency parallelism state + complex data model composition State of the Art MBT Tools usage profiles for testing scalability test selection criteria link to MBSD multiple paradigms integration uncertainty nondeterminism 18

19 Next Generation MBT : TorXakis abstraction concurrency parallelism state + complex data usage profiles for testing model composition TorXakis model system under test scalability test selection criteria link to MBSD multiple paradigms integration uncertainty nondeterminism 19

20 20 TorXakis 1. My First TorXakis Model SUT Model Adapter 2. My First TorXakis Test Run 3. More TorXakis Models

21 TorXakis : An On-Line MBT Tool model TorXakis SUT pass fail 21

22 TorXakis : A Black-Box View on Systems model a?n black-box system view x!n+1 a y!`yes` a?n b?m y b SUT x a?k x!42 y!`no` modelled as state-transition system 22

23 23 TorXakis : Definition of SUT In SUT Out SUT real, black-box system communicating with its environment via messages on input- and output channels SUTDEF Sut ::= SUT IN In :: String SUT OUT Out :: String SOCK IN In HOST "localhost" PORT 7890 SOCK OUT Out HOST "localhost" PORT 7890 ENDDEF

24 24 TorXakis : Definition of Model MODEL Stimulus Response labelled transition system with actions on input- and output channels SPECDEF Spec ::= CHAN IN CHAN OUT BEHAVIOUR ENDDEF Stim Resp Stim >-> Resp

25 TorXakis : An On-Line MBT Tool model TorXakis SUT pass fail 25

26 Adapter: Connecting TorXakis and SUT model ADAPTER connecting real SUT with formal model TorXakis adapter SUT pass fail 26

27 TorXakis : Definition of Adapter ADAPTER model connecting real SUT Stim TorXakis with formal model Resp adapter SUT In SUT Out ADAPDEF Adap ::= CHAN IN Stim SUT IN In :: String CHAN OUT Resp SUT OUT Out :: String MAP IN Stim -> In! stim MAP OUT Out? s -> Resp ENDDEF 27

28 28 TorXakis 1. My First TorXakis Model SUT Model Adapter...\examps.testnet\StimulusResponse\model\SRfinite.txs

29 29 TorXakis 1. My First TorXakis Model SUT Model Adapter 2. My First TorXakis Test Run 3. More TorXakis Models

30 30 Running TorXakis and SUT SPECDEF Spec ::= CHAN IN Stim CHAN OUT Resp ENDDEF BEHAVIOUR Stimulus >-> Response ADAPDEF Adap ::= CHAN IN Stim SUT IN In :: String CHAN OUT Resp SUT OUT Out :: String ENDDEF model MAP IN Stimu -> In! " MAP OUT Out? s -> Response SUTDEF Sut ::= SUT IN In :: String SUT OUT Out :: String model.txs TorXakis ENDDEF SOCK IN In HOST "localhost" PORT 7890 SOCK OUT Out HOST "localhost" PORT 7890 client server adapter adapter socket host portnumber SUT

31 TorXakis : Running a Test (Windows) 1. Start two Command Prompt windows, one for TorXakis, one for the SUT 2. There are two versions of SUTs: pre-compiled executables and Java sources 3. For pre-compiled, go to...\examps.testnet\stimulusresponse\winexe and run the SUT: C:> SRfinite.exe <port nr> 4. For Java JDK required* - go to:...\examps.testnet\stimulusresponse\java and compile and run the SUT C:> javac SRfinite.java C:> java SRfinite 5. In the TorXakis window, go to...\examps.testnet\stimulusresponse\model 6. Start TorXakis with the model file: C:> torxakis.exe SRfinite.txs 7. Try some TorXakis commands: TXS >> help *

32 32 Running TorXakis and SUT $ torxakis.exe StimulusResponse.txs TXS >> TorXakis :: Model-Based Testing TXS >> TorXakis :: Input File parsed... TXS >> TorXakis :: SMT Solver (Z3) initialized... TXS >> TorXakis :: Internal Environment initialized... TXS >> help

33 33 Running TorXakis and SUT TXS >> spec Spec TXS >> adap Adap TXS >> sut Sut TXS >> test 4 1 >> Input: ActIn { { ( Stimulus, [] ) } } 2 >> Output: ActOut { { ( Response, [] ) } } 3 >> Output: Delta 4 >> Output: Delta PASS TXS >>

34 34 TXS >>> help quit, q exit, x help, h,? : stop TorXakis completely : exit command run : show help spec <spec-name> adap <adap-name> sut <sut-name> : (re)set the state : (re)set the adapter : (re)set the sut

35 35 TXS >>> help const <const-def> func <func-def> var <var-decl> val <value-def> eval <value-expr> : define a constant : define a function : declare variables : define values : evaluate value-expression satsolve <value-expr> : solve value expression (open, boolean) ransolve <value-expr> : solve randomly unisolve <value-expr> : solve uniquely

36 36 TXS >>> help state btree [<state>] goto [<state>] back [<n>] init path trace : show current state number : show internal state : goto <state> number : go back <n>/[one] steps : go to initial state : path from initial state : show action trace from the initial state

37 37 TXS >>> help menu [<state>] : give possible actions step [<n>] step <action> test <action> test test <n> : make <n>/[one] random steps : make a step <action> : make test step identified by (visible) input <action> : make a test step by observing output : make <n> random test steps

38 38 TXS >>> help sutin <action> sutout adapin <action> : send input <action> to sut : receive output from sut : send input <action> through adapter to sut adapout : receive output action from sut <command> '$<' <file> : read command arg from <file> <command> args '$>' <file> : write output of <command> to <file> run <file> : run script from <file>

39 39 TorXakis: Exercise Test SUT = SRfinite.java/.exe Use Client.exe and Server.exe in...\utils to test manually via socket (with Client.exe) or simulate a SUT (Server.exe) [or use telnet or nc] against MODEL = SRfinite.txs Test SUT = SRnone.java/.exe against MODEL = SRfinite.txs Adapt SRfinite.txs such that there are two Stimuli and two Responses; test with SUT = SRfinite.java/.exe Stim Resp Stim Resp

40 40 TorXakis 1. My First TorXakis Model SUT Model Adapter 2. My First TorXakis Test Run 3. More TorXakis Models

41 41 Input for TorXakis TorXakis input = model = list of definitions behaviour / labelled transition system PROCESS PROCDEF test architecture data SPECIFICATION SPECDEF TYPE TYPEDEF ADAPTER ADAPDEF FUNCTION FUNCDEF SUT SUTDEF CONSTANT CONSTDEF

42 42 TorXakis: Process Definition Stim Resp SPECDEF Spec ::= CHAN IN Stim CHAN OUT Resp BEHAVIOUR Stim >-> Resp ENDDEF

43 43 TorXakis: Process Definition Stim PROCDEF stimresp [ Stim, Resp ] ( ) ::= Stimulus >-> Response ENDDEF Resp SPECDEF Spec ::= CHAN IN CHAN OUT BEHAVIOUR Stim Resp stimresp [ Stim, Resp ] ( ) ENDDEF

44 44 TorXakis: Process Definition Stim Resp {- Cyclic Stimulus-Response -} PROCDEF stimurespo [ Stim, Resp ] ( ) ::= Stim >-> Resp >-> stimurespo [ Stim, Resp ] ( ) ENDDEF SPECDEF Spec ::= CHAN IN Stim CHAN OUT Resp BEHAVIOUR stimurespo [ Stim, Resp ] ( ) ENDDEF

45 45 TorXakis: Exercise Try to a model for the looping StimulusResponse system ( or look at SRloop.txs ) Test SUT = SRloop.java/.exe against your looping StimulusResponsemodel (SRloop.txs) Test the finite system SUT = SRfinite.java/.exe against your looping StimulusResponsemodel. Repeat for the looping SUT = SRloop.java/.exe against the old model = SRfinite.txs. Explain the results.

46 46 TorXakis: Exercise Result Stim Resp implements Stim Resp Stim Resp implements Stim Resp

47 47 Input for TorXakis TorXakis input = model = list of definitions behaviour / labelled transition system PROCESS PROCDEF test architecture data SPECIFICATION SPECDEF TYPE TYPEDEF ADAPTER ADAPDEF FUNCTION FUNCDEF SUT SUTDEF CONSTANT CONSTDEF

48 48 TorXakis : Defining Behaviour basic behaviour = transition system complex behaviour = combining transition systems a named behaviour definition named behaviour use y x a b sequence choice parallel communication a x y exception interrupt hiding

49 49 Basis : Labelled Transition Systems - LTS a a a a b b c b c a b a b a c a b b c a d c b d a a d b

50 50 Representation of LTS S S0 10c S1 Explicit : { S0, S1, S2, S3 }, {10c,coffee,tea}, { (S0,10c,S1), ( S1, coffee, S2 ), (S1,tea,S3) }, S0 coffee tea Transition tree / graph S2 S3 Language : S ::= 10c >-> ( coffee ## tea )

51 51 Representation of LTS a >-> b ## a >-> c a a a b c b c a c a >-> ( b ## c ) b c a d c b d a a >-> b c >-> d d b

52 52 Representation of LTS 0 a 1 b 2 a >-> b 0 c 1 d 2 c >-> d 0,0 a c b 1,0 0,1 c a d 2,0 1,1 0,2 c b d a 2,1 1,2 d b 2,2 a >-> b c >-> d

53 53 Representation of LTS Q where Q ::= a >-> ( b Q ) a a b b a P where P ::= a >-> P a a b b

54 54 TorXakis: Choice Resp Stim Err -- Stimulus-Response with Error PROCDEF errsr [ Stim, Resp, Err ] () ::= Stim >-> ( Resp >-> errsr [Stim,Resp,Err] () ## Err >-> errsr [Stim,Resp,Err] () ) ENDDEF

55 55 TorXakis: Parallel Interleaving Stim1 Resp1 Stim2 Resp2 Stim1 Stim2 Resp1 Resp2 -- Parallelism with interleaving: Stim2 Resp2 Resp1 Stim1 StimResp1 StimResp2

56 56 TorXakis: Parallel Communication [ M ] Stim1 M Resp M Resp2 Stim1 Resp2 M -- Parallelism with communication: StimResp1 [ M ] StimResp2 Resp2 Stim1

57 57 TorXakis: Communication + Hiding Stim1 [ M ] Stim1 M Resp M Resp2 Resp2 Stim1 Resp2 Resp2 Stim1 -- Communication + Hiding: HIDE [ M ] IN StimResp1 [ M ] StimResp2 NI

58 58 TorXakis: Exercise Experiment with, using testing or stepping, with SRfinite.txs, SRlnone.txs, SRloop,txs, SRnone, SRparallel.txs and corresponding SUTs. If you have JDK installed you can make mutants, i.e., small modifications/errors in the Java SUTs, and see whether you can detect the errors.

59 59 Input for TorXakis TorXakis input = model = list of definitions behaviour / labelled transition system PROCESS PROCDEF test architecture data SPECIFICATION SPECDEF TYPE TYPEDEF ADAPTER ADAPDEF FUNCTION FUNCDEF SUT SUTDEF CONSTANT CONSTDEF

60 60 TorXakis: Data Types Standard types: Int, Bool, String Algebraic data types TYPEDEF Colour ::= Red Yellow Blue TYPEDEF IntList ::= Nil Cons { hd :: Int, tl :: IntList }

61 61 TorXakis: Functions TYPEDEF Colour ::= Red Yellow Blue Functions: name, parameters, type, (recursive) value expression Overloading Standard functions for: Int, Bool, String TYPEDEF IntList ::= Nil Cons { hd :: Int, tl :: IntList } FUNCDEF add ( x, y :: Int ) :: Int ::= x + y FUNCDEF add ( x :: Int; lst :: IntList ) :: IntList ::= IF isnil ( lst ) THEN Cons ( x, Nil ) ELSE Cons ( hd ( lst ), add ( x, tl ( lst ) ) ) FI

62 62 TorXakis: Adder Plus? x? y Minus? x? y SUT Result! x + y Result! x - y Use of Data in processes

63 63 TorXakis: Exercise Test with the Adder model in SRadder.txs and corresponding SUTs. Make a mutant of SRadder.java. Investigate the possibilities with Strings and Regular Expressions using the model SRrr.txs and corresponding SUTs.

64 A More Elaborate Example: The Dispatcher-Processing System input jobs..\examps.testnet\dispatchprocs dispatcher processor processor processor processor processed jobs 64

65 65 Example: Dispatcher-Processing System Start job processor Idle processor Start Finish Finish job DisPro01-proc.txs state transition system Processing

66 66 Example: Two Parallel Processors Start Start Start Finish Start Finish Finish Finish

67 67 Example: Two Parallel Processors processor 1 processor 2 S1 0,0 S2 0 S1 1 F1 2 parallelism 0 S2 1 F2 2 1,0 0,1 F1 S2 S1 F2 2,0 1,1 0,2 S2 F1 F2 S1 2,1 1,2 F2 F1 2,2 processor 1 processor 2

68 68 Example: Four Parallel Processors Start Start Start Start Start Finish Start Finish Start Finish Start Finish Finish Finish Finish Finish DisPro03-procs.txs

69 69 Example: Four Parallel Processors parallelism

70 70 Example: Dispatcher-Processing System Start Start Start Start processor(i) Start Finish Start Finish Start Finish Start Finish Start Finish Finish Finish Finish Finish composition processors ::= processor(1) processor(2) processor(3) processor(4)

71 71 Example: Dispatcher-Processing System Job DisPro03-procs.txs dispatcher Start Finish Start Finish Start Finish Start Finish Job Start dispatcher Finish Finish Finish Finish dispatch_procs ::= processors [ Start ] dispatcher composition

72 Example: Dispatcher-Processing System Job DisPro04-hide.txs dispatcher Start Finish Start Finish Start Finish Start Finish Job Start dispatcher Finish Finish Finish Finish dispatch_procs ::= HIDE [ Start ] IN processors [ Start ] dispatcher NI abstraction 72

73 Example: Dispatcher Processing System 73

74 74 Example: Dispatcher-Processing System Job Inputs: Job1, Job2, Job3: dispatcher Job1 Start Finish Start Finish Start Finish Start Finish J2 J3 Finish Finish Finish Finish F1 F2 Finish3 F2 F3 F1 F3 F1 F2 uncertainty no unique expected result F3 F2 F3 F1 F2 F1

75 75 Example: Dispatcher-Processing System uncertainty non-determinism Inputs: Job1, Job2, Job3: Job1 F1 J2 J2 F1 J3 F2 F2 J3 F1 F2 Finish3 J3 F2 F3 F2 F3 F1 F3 F1 F2 F3 F3 F2 F3 F2 F3 F1 F2 F1

76 76 TorXakis: Exercise Use TXS >> step to step through model DisPro04-hide.txs and check possible execution sequences. Investigate, using stepping, the use of Data and Type Definitions in DisPro05-adder.txs : for modelling an Adder DisPro06-constr.txs : for modelling input constraints and a non-deterministic result DisPro-07-nprocs.txs : for modelling an arbitrary number of processors Test with model TXDisPro08-gcd.txs and SUT SutGcd.java. Repeat with model TXDisPro09-purp.txs. Explain the difference.

77 Example: Dispatcher-Processing System FUNCDEF gcd ( a, b :: Int ) :: Int ::= IF a == b THEN a ELSE IF a > b THEN gcd ( a - b, b ) ELSE gcd ( a, b - a ) FI FI Start job state + data Start Finish? job :: JobData! gcd ( job.x, job.y ) [[ isvalidjob(job) ]] TYPEDEF JobData ::= JobData { jobid :: Int ; jobdescr :: String ; x, y :: Int } processor Finish job FUNCDEF isvalidjob ( jobdata :: JobData ) :: Bool ::= jobdata.jobid > 0 /\ strinre ( jobdata.jobdescr, REGEX('[A-Z][0-9]{2}[a-z]+') ) /\

78 78 TorXakis: Exercise Test with model TXDisPro10-jobdata.txs and SUT SutJobData.java. Repeat with SUT model SutWithError.java. What is the Error?

79 79 More Complex Data Test data generation from XSD (XML) descriptions with constraints complex data

80 80 TorXakis: Exercise Go to...\examps.testnet\readwriteconflict. Consider the model ReadWrite.txs. This model has finite behaviour, i.e., it stops after 7 steps (deadlocks). From he model, argue what the last action is. Check your argumentation by stepping through the model.

81 81 A Goat, a Wolf, Cabbage, and a Cat..\examps.testnet\RiverCrossing Model-Based Testing of a Little Game Input via standard IO Inputs: g? w? c? n? Outputs: eaten! retry! done!

82 82 A Goat, a Wolf, Cabbage, and a Cat model TorXakis adapter SUT

83 A Goat, a Wolf, Cabbage, and a Cat model TorXakis channels In socket g? w? c? n? std IO g? w? c? n? TorXakis adapter external adapter Out eaten! retry! done! eaten! retry! done! 83

84 84 TorXakis: Exercise Go to...\examps.testnet\rivercrossing. Consider the model RiverState.txs (visualization in RiverState.pdf) The SUT is sut.bat or sut.sh. Does the behaviour of SUT correspond to the model? Explain...

85 85 Next Generation MBT : TorXakis Status abstraction model composition uncertainty nondeterminism state + complex data concurrency parallelism scalability link to MBSD usage profiles multiple paradigm integration test selection criteria

86 86 TorXakis Questions? model-based test generation model test execution system pass fail

87 87