Model-Based Testing. The Next Step in Test Automation!? Jan Tretmans SUT. model. TorXakis. SUMBAT: SUpersizing. Model.

Transcription

1 Model-Based Testing The Next Step in Test Automation!? model Jan Tretmans SUMBAT: SUpersizing Model TorXakis BAsed Testing SUT TNO Embedded Systems Innovation Radboud University, Nijmegen pass fail

2 2 Overview Model-Based Testing : Motivation What Model-Based Development Status Research A Tool Example

3 3 Embedded Systems or: What do Dutch Dykes have to do with Model-Based Testing?

4 4 Embedded Systems or: What do Dutch Dykes have to do with Model-Based Testing?

5 Testing High-Tech Systems 5

6 6 Quality of Embedded Systems Software is brain of system software controls, connects, monitors almost any aspect of ES system behaviour majority of innovation is in software Software determines quality and reliability of Embedded System often > 50 % of system defects are software bugs

7 Quality Software 7

8 8 Testing Challenges : Complexity Testing effort grows exponentially with system size Testing cannot keep pace with development x : [0..9] x : [0..9] y : [0..9] y : [0..9] x : [0..9] z : [0..9] 10 ways that it can go wrong 10 combinations of inputs to check 100 ways that it can go wrong 100 combinations of inputs to check 1000 ways that it can go wrong 1000 combinations of inputs to check combinatorial explosion of required testing effort

9 9 Testing Challenges : Complexity Testing effort grows exponentially with system size Testing cannot keep pace with development x : [0..9] x : [0..9] y : [0..9] y : [0..9] x : [0..9] z : [0..9] 10 ways that it can go wrong 10 combinations of inputs to check 100 ways that it can go wrong 100 combinations of inputs to check 1000 ways that it can go wrong 1000 combinations of inputs to check combinatorial explosion of required testing effort

10 10 Testing Challenges : Components Components come from anywhere legacy system

11 11 Testing Challenges : Uncertainty Sometimes you don t know.. testing a search engine, weather forecast, systems-of-systems, big data,... Sometimes you don t want to know.. no details abstraction particular view check: -20 o <= temp <= 40 o Uncertainty of test outcomes & oracles!coffee non-determinism, probabilities, constraints?coin!alarm?button?button

12 Trends & Challenges complexity connectivity change variability evolvability model based testing uncertainty multi disciplinarity heterogeneous components 12

13 Model-Based Testing 13

14 Model-Based Testing MBT next step in test automation: model-based test generation model + test generation + result analysis test execution SUT pass fail 14

15 1 : Manual Testing 1. Manual testing SUT pass fail System Under Test 15

16 2 : Scripted Testing test cases TTCN 1. Manual testing 2. Scripted testing test execution SUT pass fail 16

17 3 : Keyword-Driven Testing high-level test notation test scripts 1. Manual testing 2. Scripted testing 3. Keyword-driven testing test execution SUT pass fail 17

18 4 : Model-Based Testing model-based TTCN Test test generation cases system model 1. Manual testing 2. Scripted testing 3. Keyword-driven testing 4. Model-based testing test execution pass fail SUT 18

19 MBT : Example Models!coffee?coin!alarm?button?button 19

20 MBT : Benefits model-based test generation test execution pass fail model SUT detecting more bugs faster and cheaper MBT: next step in test automation Automatic test generation + test execution + result analysis More, longer, and diversified test cases more variation in test flow and in test data Model is precise and consistent test basis unambiguous analysis of test results Test maintenance by maintaining models improved regression testing Expressing test coverage model coverage customer profile coverage

21 MBT : Many Tools AETG Agatha Agedis Autolink Axini Test Manager Conformiq Cooper Cover DTM fmbt G st Gotcha Graphwalker JTorX MaTeLo MBTsuite M-Frame MISTA NModel OSMO ParTeG Phact/The Kit PyModel QuickCheck Reactis Recover RT-Tester SaMsTaG Smartesting CertifyIt Spec Explorer StateMate STG Temppo TestGen (Stirling) TestGen (INT) TestComposer TestOptimal TGV Tigris TorX TorXakis T-Vec Uppaal-Cover Uppaal-Tron Tveda

22 22 Model-Based Verification, Validation, Testing,.....

23 Validation, Verification, Testing informal requirements validation informal world formal world model verification real world model-based testing SUT 23

24 24 Doing Something with Models Modelling making a model reveals errors Simulation go step-by-step through the model Model checking go through all states of the model Theorem proving prove theorems about the model Code generation executable code from the model Testing test an implementation for compliance Model learning generate a model from observation

25 25 Code Generation from a Model A model is more (less) than code generation: views abstraction testing of aspects verification and validation of aspects

26 26 Spectrum of Models abstract (test) models design models virtualization realization

27 27 Code Generation from a Model! y y x y = x model of x? x (x >= 0) specification of properties rather than construction under-specification non-determinism

28 28 Model-Based Testing Status

29 MBT : Benefits? But... If doing MBT is so smart, why ain t you rich? MBT : State of the Art promising, emerging a number of successful applications many companies are experimenting MBT : State for the Future (for High-tech Embedded Systems)? MBT : State of Practice lagging behind Reasons technical tools organizational maturity of testing educational

30 Testing High-Tech Systems 30

31 Trends & Challenges complexity connectivity change variability evolvability model based testing uncertainty multi disciplinarity heterogeneous components 31

32 MBT : Next Generation Challenges abstraction concurrency parallelism state + complex data usage profiles for testing model composition test selection criteria Model Based Testing scalability link to MBSD multiple paradigms integration uncertainty nondeterminism 32

33 33 Model-Based Testing Research SUMBAT

34 Informationsteknologi SUMBAT SUpersizing Model BAsed Testing

35 Informationsteknologi SUMBAT goal SUpersizing Model-BAsed Testing: making MBT applicable to large, complex, high-tech embedded software systems, with millions of lines of code, distribution, concurrency, complex data- and state-based behaviour, uncertainty, non-determinism, and complex and heterogeneous interfaces

36 Informationsteknologi SUMBAT method theory LTS ioco automata learning test selection..... Cases: ASML Océ PANalytical..... applications tools TorXakis TorXakis LTSmin LearnLib.....

37 Next Generation MBT : TorXakis abstraction concurrency parallelism state + complex data model composition test selection criteria TorXakis model SUT usage profiles for testing scalability link to MBSD multiple paradigms integration uncertainty nondeterminism 37

38 38 TorXakis A bit more detail

39 TorXakis : A Black-Box View on Systems model a?n black-box system view x!n+1 a y!`yes` a?n b?m y b SUT x a?k x!42 y!`no` modelled as state-transition system 39

40 40 TorXakis : Testing Theory SUT conforms to model sound exhaustive SUT passes tests model-based Test TTCN test cases generation test execution system model SUT SUT conforms to model pass fail

41 41 TorXakis : On-the Fly MBT on-the-fly MBT Test TTCN cases TorXakis system model SUT pass fail

42 42 TorXakis : Testing Theory ioco i ioco s = def Straces (s) : out (i after ) out (s after ) s is a Labelled Transition System i is (assumed to be) an input-enabled LTS p p =!x L U { }. p!x Straces ( s ) = { ( L { } )* s } p after = { p p p } out ( P ) = {!x L U p!x, p P } { p p, p P }

43 43 TorXakis : Overview Models state-based control flow and complex data support for parallel, concurrent systems composing complex models from simple models non-determinism, uncertainty abstraction, under-specification But... research prototype poor usability Tool on-line MBT tool Current Research scalability test selection Under the hood powerful constraint/smt solvers (Z3, CVC4) well-defined semantics and algorithms ioco testing theory for symbolic transition systems algebraic data-type definitions

44 44 Model-Based Testing Research Test Selection

45 45 Test Selection Exhaustiveness never achieved in practice Test selection = select subset of exhaustive test suite, to achieve confidence in quality of tested product select best test cases capable of detecting failures measure to what extent testing was exhaustive : coverage Optimization problem best possible testing within cost/time constraints

46 Testing and Quality Quality-assurance costs Cost Remaining-defects costs # Test cases

47 47 Test Selection Extra information / domain information required : which test cases have high value? which errors are likely? which errors have high impact? what is the user / customer doing? risk usage profiling statistical testing

48 48 Model-Based Testing Research How to Get these D Models

49 Testing : Model-Based Testing model-based test generation model test execution system pass fail

50 Test-Based Modeling model-based Model test Learner generation model test execution system

51 51 TorXakis Demo

52 Tic-Tac-Toe CHANDEF MyChannels ::= In ; Out ENDDEF Tic-Tac-Toe Rules MODELDEF TicTacToe ::= CHAN IN In CHAN OUT Out BEHAVIOUR In X >-> Out O ENDDEF model.txs CNECTDEF Sut ::= CLIENTSOCK CHAN OUT In HOST "localhost" PORT 7890 ENCODE In X ->! X CHAN IN Out HOST "localhost" PORT 7890 DECODE Out <-? s ENDDEF TorXakis socket pass fail selenium 52

53 Model-Based Testing The Next Step in Test Automation!? MBT: Is it the promising future of software testing? Can we do without MBT? If not MBT, what then?? TorXakis model SUT pass fail