L04. Model-based Testing: Principles Henry Muccini. Dipartimento di Informatica, Universityof L Aquila

Transcription

1 Università degli Studi dell Aquila L04. Model-based Testing: Principles Henry Muccini Dipartimento di Informatica, Universityof L Aquila henry.muccini@univaq.it

2 Copyright Notice The material in these slides may be freely reproduced and distributed, partially or totally, as far as an explicit reference or acknowledge to the material author is preserved. Henry Muccini

3

4 Agenda Modeling MBT: Why? Models for MBT UML-based Testing Testing activities

5 AGENDA Modeling Model-based testing Why? Models for MBT UML-based Testing

6 Modeling: there is not just programming What is a model: A model is a simplification of the reality The reality is abstracted, since it is too complex Why modeling Models are built for a better understanding of the system under development Everything is a model?!?

7 Non-software Models

8 Specification: What

9 Software Models

10 OtherSoftware Models(Web models)

11 othersoftware models

12 and manyothers Pilot Multifunction Display1 Pilot Multifunction Display2 CoPilot Multifunction Display1 CoPilot Multifunction Display2 Display Processor Display Processor Display Processor Display Processor High speed network High speed network Mission Processor Mission Processor Mission Processor bus bus Auto-Pilot GPS Nav Radio

13 Model Driven Development (MDD) Models are becoming first core assets in Software Engineering/Architecting: models are primary artifactsretained as first class entities that can be analyzed and manipulated by means of automated tools the focus moves from writing and sharing documentsto writing and sharing models.

14 From Document Driven to Model Driven

15 Models in MDD Models in MDD can be used for: Documenting design decisions Guiding the coding phase For analyzingand validatingdesign choices by complementing traditional code-level analysis techniques

16 Why modeling? For Documenting and Analysis

17 What is MBT? It consists in: Extracting Test Cases from the Model Run the test case over the system implementation Purpose: To validate the implementation conformance to the model The model itself is the oracle, i.e., it represents the expected behavior

18 Model-based Testing A model based testing approach accepts two main inputs a model of the software under test, a set of test generation directiveswhich guide the test cases selection and outputs a test specification which includes a set of stimulithe tester should introduce in the system together with expected responses

19 Modeling for testing Essentially, we need (at least) a behavioral modelof the software system: Labelled Transition System StateCharts UML State of Sequence Diagrams IOLTS Finite State Automata

20 The base-line technique for designing test cases Timely Often useful in refining specifications and assessing testability before code is written Effective finds some classes of fault (e.g., missing logic) that can elude other approaches Widely applicable to any description of program behavior serving as spec at any level of granularity from module to system testing. Economical typically less expensive to design and execute than structural (code-based) test cases (C) 2007 MAURO PEZZÈ & MICHAL YOUNG

21 Program code is not necessary Only a description of intended behavior is needed Even incomplete and informal specifications can be used Although precise, complete specifications lead to better test suites Early functional test design has side benefits Often reveals ambiguities and inconsistency in spec Useful for assessing testability And improving test schedule and budget by improving spec Useful explanation of specification or in the extreme case (as in XP), test cases are the spec (C) 2007 MAURO PEZZÈ & MICHAL YOUNG

22 Different testing strategies (functional, structural, fault-based, model-based) are most effective for different classes of faults Functional testing is best for missing logic faults A common problem: Some program logic was simply forgotten Structural (code-based) testing will never focus on code that isn t there! (C) 2007 MAURO PEZZÈ & MICHAL YOUNG

23 Modeling for testing Such a model can be produced: 1. Through a formal specification language Typically referred as formal testing 2. Through a diagrammatic(visual) notation Typically referred as UML-based Testing

24 1. MBT basedon FormalSpecifications Since the 80 s, many specification-based testing approaches have been proposed, based on formal languages such as Z, VDM, CSP, CCS, LOTOS, SDL, and Petri Nets. More recently: Based on dynamic behavior: FSM-based (Bochmann&Petrenko, Lee&Yannakis, ) LTS-based (Brinksma&Tretmans, Jard&Jeron ) Focussing on static aspects, ADT theory (Bernot&Gaudel ) Z-based (Hierons )

25 Formal Specification-based Testing TorX(Côte de Resyste) on-the-fly test generation and execution random LOTOS and Promela TGV(IRISA - Rennes) derives tests in TTCN from LOTOS or SDL uses test purposes TVEDA(CNET - France Telecom) derives TTCN tests from SDL specification

26 2. MBT basedon the UML UML State-machine based testing: which might require the translation of the UML diagrams into an intermediate formal description Scenario-basedtesting: based on the use of opportunely annotated UML Sequence Diagram State-based and Scenario-based: combine the useof interactiondiagramsand state diagram, possibly annotated with further(formal) information (e.g. OCL)

27 Model-based Testing: the overall idea From Leila Naslavsky Advancement report

28 AsmL TESTOR UMLAUT Model-based Testing Approaches UMLTest AGEDIS TOTEM SOOTF COW-SUITE SCENTOR SeDiTeC UCSC-System

29 Main Testing Activities Test Selection/Generation: it consists in selecting a suitable and finite set of test cases from the possibly infinite set; Test Execution and Evaluation: it consists in executing the code accordingly to the selected test cases and comparing real and expected results;

30 Test Selection A test case, in MBT, is typically a possible scenario In scenario-based testing: A scenario itself is an abstract test case In state-based testing Execution flows are extraced from the state machine and represent a test case MB coverage criteriaare applied on this model

31 Test Case Selection Abstract Test case= path over the LTS/FSM, corresponding to sequences of events They are not concrete test cases, but abstract (depending on the specification) Concrete Test cases= mappingasbtract test cases into execution statements to be launched

32 Test Case Selection 1 insd C1 C2 Test directives 1 Model of the system m9 m1 2 outsd1 C1 C2 C3 m9 m3 Test specification m8 m1

33 Test Case Selection

34 Test Execution It consists in running the test cases on the system implementation So to compare the real execution with the expected behavior

35 MBT practical concerns Traceability, i.e. relating the abstract valuesof the specification to the concrete values of the implementation. Execution, i.e. forcing the ImplementationUnder Test (IUT) to execute the specific sequence of events that has been selected.

36 Testing and Industry In ordertobesuitableforindustrial needsa testing approach has to emphasize the following qualities: Usability Not ad-hoc models Timeliness Soonand evenincomplete Tool support Opposed to accuracy Goal of a suitable testing approach should be to improve the test results accuracy without sensibly raising the testing effort

37 MBT Tools There are some automated tools for MBT: Telelogic Tau and Tau Tester Leirios Test Designer Conformiq Qtronic Rhapsody Test Conductor and Automatic Test Generation AGEDIS

38 MBT in industry: challenges and issues As remarked by Alan Hartman in his 2006 speech at ISSTA: Only 1% of industry uses it Why?

39 Why? The main reasons I foresee: Model2Code (and viceversa) traceability problems modeling is notyet so widely recognizedas part of the development process. Legacy systems because of tool/service problems cost/effectiveness is still under exploration

40 Conformiq: Twomodelsare requiredin Input SutModel.Java (in QML) SutModel.xmi(GraphicalNotation)

41 QML Just like Java, QML consists of variables and expressions, all strictly typed and known before the compilation. The types are divided into the same three groups that Java has: primitives, references and values

42 QML QML extends Java with: Records Ports System block State machines

43 SutModelGraphical

44 SutModelGraphical

45 Coverage Figure 5 The Coverage Editor window

46 Test Case Execution Fig 8 Execution of Test Case 1 on the SUT: the figure illustrates the detection of a bug.