Certified Model-Based Tester

Transcription

1 Certified Model-Based Tester 1 Introduction to this Syllabus 1.1 Purpose of this Document This syllabus defines the content of the international qualification scheme for the "Certified Model-Based Tester" (CMBT). It is established by the Special Interest Group SIG MBT of the International Software Quality Institute (isqi). CMBT is an introduction to Model Based Testing. It describes the basics of model based testing and it provides a foundation for further development as a professional in model based testing. The isqi SIG MBT has created: - The syllabus - The Business Outcomes - The Course Material The Course Material can be licensed to training Providers. In order to license the Training provider must have at least two trainers that hold the CMBT and that have attended the Train the Trainer for CMBT. The CMBT qualification is aimed at testers, test analysts, test designers, test managers (in relation to test processes and in contact with stakeholders) and managers. Basic knowledge of concepts in software testing is required. It is recommended that the candidate holds a foundation level certificate as "ISTQB Certified Tester" (CTFL) or has equivalent knowledge. 1.2 Cognitive Level of Knowledge Detailed learning objectives are indicated for each section in this syllabus. These objectives identify what the trainee will be able to do following the completion of each module. They are classified as follows: Level 1: Remember (K1) Level 2: Understand (K2) Level 3: Apply (K3) Level 4: Analyze (K4) The top-level heading for each chapter shows the highest level of learning objectives that is covered within the chapter. The definition of these cognitive levels matches the definition given in the ISTQB Certified Tester scheme to guarantee compliance with and thus integrability to this scheme. Please refer to [CTFL2011] for more details. Note to training providers: The cognitive level K2 and above requires at least one example in the training material. The levels K3 and above require at least one practical exercise. isqi CMBT Syllabus V2.3 May 14, 2013 Page 1 of 26

2 1.3 The Examination The CMBT certificate examination will be based on this syllabus. Answers to examination questions may require the use of material based on more than one section of this syllabus. All sections of the syllabus are examinable. The exam is a 40 question, multiple choice exam. Examinations may be taken as part of a training course or taken independently (e.g. at an examination center or in a public examination). 1.4 Business Outcome This section lists the Business Outcomes expected of a candidate who has achieved the CMBT certification. A Certified Model Based Tester shall be able to CMBT-1 Apply general approaches of MBT techniques for testing CMBT-2 Valuate the benefits and risks of MBT and its introduction CMBT-3 Introduce and improve MBT techniques with a basic level of complexity on a project scope CMBT-4 Provide support for assessment and evaluation of constraints, objectives, benefits and risks in the context of MBT CMBT-5 Provide support for evaluation, introduction, implementation and improvement of MBT CMBT-6 Evaluate and analyze MBT artifacts CMBT-7 Generate test cases by applying modeling strategies and techniques CMBT-8, Manage test models and tool support with a medium level of complexity on a project scope CMBT-9 Generate and manage test artifacts from MBT artifacts CMBT-10 Provide support to develop and specify MBT concepts on a project scope In general, a Model Based Tester is expected to have acquired the necessary skills to enable further development at Advanced Level in the areas of application, introduction and improvement of MBT to the test process. The CMBT extends the learning objectives for Test Design Techniques of CT-FL section 4. In particular, CT-FL LO Write test cases from given software models using equivalence partitioning, boundary value analysis, decision tables and state transition diagrams/tables is extended to the creation of test models. Other CT-FL learning objectives extended in this syllabus are: CT-FL LO Evaluate the quality of test cases in terms of clear traceability to the requirements and expected results CT-FL LO Explain the concept of use case testing and its benefits CT-FL LO Explain the concepts of statement and decision coverage, and give reasons why these concepts can also be used at test level other than component testing CT-FL LO Write test cases from given control flows using statement and decision test design techniques isqi CMBT Syllabus V2.3 May 14, 2013 Page 2 of 26

3 1.5 How this Syllabus is organized This document describes the content of CMBT. For trainees that look for deeper insight into the subject, the following advanced modules are under consideration: Additional examples and exercises (test management with tools, writing test models, deactivate the bomb ), Specific topics (TTCN-3, The UML Testing profile, Markov chain usage models), Tooling (tool classification and evaluation, MBT tools in practice). The syllabus is split in sub-sections (modules). For the basics each sub-section starts with a list of terms the trainees should remember (cognitive level K1), followed by the learning objectives of the module. For the addons, learning objectives, duration, cognitive level and a description of the mandatory content is given. 2 CMBT (735 min, K4) 2.1 Motivation for MBT (110 min, K2) Terms model, model-based testing, system model, test model Learning objectives: Challenges of SW testing LO Recall the challenges of traditional software testing (K1) Definitions and basic concepts LO Recall definitions of MBT and their related context (K1) LO Describe what a model is and its applicability to software (K2) LO Describe the basic concepts of MBT (K2) LO Compare system model and test model (K2) Objectives of MBT LO Recall the objectives of MBT (K1) Benefits and challenges of MBT LO Recall typical MBT business scenarios (K1) LO Describe the global benefits of MBT (K2) LO Describe the challenges of and potential obstacles introducing MBT (K2) Integrating MBT in the Fundamental Test Process LO Recall the integration of MBT to the fundamental test process (K1) Challenges of SW testing (15 min, K1) Software testing today faces various challenges. In particular, there is a need for higher efficiency and quality including objectivity, repeatability, transparency, etc. The effort related to testing activities (and thus the costs isqi CMBT Syllabus V2.3 May 14, 2013 Page 3 of 26

4 of testing) are usually higher than estimated. In addition, these costs are not transparent to developers and, what is worse, to managers. As a result, testing is perceived as cost factor (known as cost of conformance ) that is not counterbalanced by a measurable gain (the cost of non-conformance usually not being quantified). Since it is not transparent, what testers do, they do not receive appropriate appraisal. The job image is rather poor, associating testing with minor qualification. In order to increase the efficiency of tests, many companies concentrate on test automation. In this context model-based testing came in. The main idea was to automate as many activities related to test case specification as possible. Test cases are generated from a formal description of the system, from its usage scenarios or in general of its requirements using a tool and by applying predefined test criteria. Often, the obstacle of MBT is delivering much more test cases and thus increasing test effort. However, more tests do not necessarily imply better tests. In fact, it is quite difficult to determine test quality, i.e. its effectiveness. The expectation on MBT regarding increased efficiency and effectiveness are high and in some cases even exorbitant. One objective of this course is to give an idea on what MBT can achieve, how to do it and where its limitations are Definitions and basic concept (20 min, K2) Following Herbert Stachowiak s [Stachowiak73] general definition, a model is a pragmatic and abbreviated view on a real world entity (might also be another model). A model serves a certain purpose for both the model designer and the model reader. Thus models are judged whether they are appropriately expressed for the required purpose or not. As long as models are not ill-formed/ill-constructed they are considered to be valid. According to the definition given in the IREB (R) CPRE- FL, "a model is an abstraction of existing reality or a plan for reality to be created". (More details will be given in section 2.2). For model-based testing several definitions exist. In ETSI ES V1.1.1 ( ) the European Telecommunications Standards Institute defines model-based testing as an umbrella of approaches that generate tests from models. However, this generation is not necessarily done in an automated way. Tests may also be derived manually from models. In that case, the model is used for visualization and documentation purposes (e.g. to describe the test idea or test scenario or test objectives respectively). This leads to a more general definition of MBT: An umbrella of engineering approaches where (semi-)formal models are used within the test process in order to specify and/or generate test artifacts. Examples for generated artifacts are test cases, test data or test scripts (on one of several possible abstraction layers of the model). Note to training providers: Examples for a system model, a test model and the resulting test artifacts are required. In the following A system model is a model describing the system (functionality, intended behavior ). In the development process a system model is usually specified on / an artifact of the left side of the V-model. A test model is a model describing the aspects pertinent for testing a system (test architecture, procedures, test cases, test steps, etc. In the development process a test model is specified on / an artifact of the right side of the V-model. isqi CMBT Syllabus V2.3 May 14, 2013 Page 4 of 26

5 Several variants of MBT exist based on the use of one model (either system model or test model) or with two (system and test model) that may either be generated one from the other (and refined afterwards) or written complete independently [Ros2010]. Depending on the way the models are designed, models may be used for both static and dynamic testing and on any test level. The knowledge of CT-FL testing techniques is a prerequisite for MBT Objectives of MBT (30 min, K1) MBT pursues various objectives: improved management of complexity (hierarchical models helps to get grip of large and complex systems) automation (generation of test artifacts) increase of efficiency via automated processes improved communication make imperfect or implicit knowledge explicit increase of re-use (development models, parts of test models in other context) improvements regarding test design systematic (transparency and documentation) improvements in test completeness and test quality (making it transparent, verifiable and measurable) improvements regarding the effectiveness of the test process (early start of test activities, improved communication) additional support of test management (e.g. by selecting the best test sample ) improved traceability to requirements and early validation of requirements early testing by use of model simulation early integration of end users Benefits and Challenges of MBT (15 min, K2) Models are a comprehensible foundation for communication and documentation: with models one or multiple different abstraction levels ( levels of understanding ) are established, that can be understood and will be discussed by all stakeholders involved in this abstraction level e.g. also by product managers or customers models are also in particular making things explicit and formalized (explicit design) It becomes more transparent what the test is doing. The test coverage becomes controllable and measurable based on mathematical concepts e.g. from graph theory. Developing the test model is an early validation of the requirements. CPRE mentions this switch between different styles of documentation as one out of six principles for checking requirements. In addition, the syntactical correctness as well as the semantic correctness of models may be checked by a tool. Using specific models for the test design independent from existing development models extends the benefits of MBT: pure verification of development artifacts gets validation of the system under test. The automated process of test generation helps avoiding human errors. Also, maintainability is improved since changes are only required in the model (and not in the generated artifacts). This is particularly interesting for agile development. isqi CMBT Syllabus V2.3 May 14, 2013 Page 5 of 26

6 However, models also have some draw-backs. Unlike natural language, additional effort is typically needed for training and tool licenses. Also, MBT requires some re-thinking. The problem of test case explosion (e.g. for unit testing or white box testing) must be controlled (usually, not all test cases may be executed) and thus test management (e.g. the definition and application of selection strategies) becomes more important. Last, but not least, human resistance against change should be taken into account. Of course the introduction of MBT also includes challenging requirements: integration into development and quality assurance processes and their tool chains, linking of automatically generated tests to requirements, creation of abstract domain-specific languages to remove the need for testers to learn new (formal) modeling languages, etc. To avoid mistakes, it is important to meet with potential obstacles adequately. To build up related capabilities is one of the main objectives of this training course Integrating MBT in the Fundamental Test Process (30 min, K1) From an MBT perspective, models may be used for each step of the fundamental test process as defined in CTFL. For example, the master test plan may be based on a system model. Test models may be used to describe test conditions and test cases. The latter can be generated automatically. Test coverage can be controlled by mirroring the executed test cases back into the model. A specific coverage may be required as exit criteria for a given test phase. Last, but not least, test models may be collected for further reuse. MBT concerns all roles and has implications for all activities that exist in the fundamental test process. 2.2 Modeling in Engineering (170 min, K4) Terms graphical model, textual model, structural model, behavioral model, data model, model element, diagram, environmental model, usage model, model and meta-model Learning objectives: Models, Approaches and Notations LO Recall the different model types and other terms listed above (K1) LO Select the appropriate model type in a given context and for a given purpose (K3) LO Recall relevant modeling standards and notations (K1) Models in SW Processes LO Describe, how and where models can be applied within the SW development process (K2) LO Recall the pros and cons related to model development, use and application (K1) LO Describe the difference between the top-down and bottom-up modeling approaches (K2) Working with Models LO Be able to design and develop models for simple situations using one modeling notation (K3) LO Analyze given models with given context and evaluate whether models fulfil the needs/requirements to model the relevant fact described in the context (consider the three quality criteria (form/syntax, semantics/content, adequacy) (K4) isqi CMBT Syllabus V2.3 May 14, 2013 Page 6 of 26

7 LO Recall the quality criteria for models and appropriate verification methods (K1) LO Describe, why modeling guidelines are required (K2) Models, Approaches and Notations (60 min, K3) Models always represent an abstraction of reality. They either concentrate on specific aspects neglecting the rest or they combine aspects to a condensed overview. Models are always written for a specific purpose. Depending on context and purpose, different approaches (having different focus) may be chosen. The resulting model may be categorized as follows: 1. structural / data-oriented / behavioral 2. computation-independent / platform-independent / platform-dependent 3. functional / non-functional 4. requirements / design / system / environment / usage / test models For example, a model may be data-oriented and platform-independent, describing functional requirements. Structural models describe the structural aspects of a system (classes, use cases, data models ). Behavioral models describe the dynamic behavior of a system (control flow, data flow, state machines, scenarios, Markov chains ). Both structural or/and behavioral models may also be used to describe test artifacts (e.g. test data or test scenarios). Models are usually represented in graphical form, but textual and even hybrid notations also exist. Note to training providers: Examples for the comparison between graphical and textual representations of models and (textual) system/test documentation/specifications are required. Syntax and semantics of the model are defined by the modeling language. Some examples for modeling standards are: Unified Modeling Language (UML) standardized modeling language ((OMG UML = ISO )), graphical notation for both structural and behavioral models Object Constraint Language (OCL) textual extension of UML (OMG OCL = ISO 19507), describing additional constraints (as natural text or using predefined keywords) Business Process Modeling Notation (BPMN) graphical notation for business processes (OMG Business Modeling Specification BPMN 2.0), only for behavioral models Message Sequence Charts (MSC) graphical notation (ITU-T Z.120) to describe interactions between components or systems, only for behavioral models, very similar to UML sequence diagrams Testing and Test Control Notation 3 (TTCN-3) international standard (ES to 7), textual notation for black-box testing and certification Specific UML profiles have been defined to support particular application domains. Examples are: System Modeling Language (SysML) defines additional stereotypes (OMG Domain Specification SyML 1.3) for the model-based description of larger system and embedded systems isqi CMBT Syllabus V2.3 May 14, 2013 Page 7 of 26

8 UML Testing Profile (UTP) provides extensions to UML (OMG UML Profile specification UTP 1.1) to support the design, visualization, specification, analysis, construction, and documentation of the artifacts involved in testing UML Profile for Modeling and Analysis of Real-time and Embedded Systems (OMG UML Profile specification MARTE 1.1) adds capabilities to UML for model-driven development of Real Time and Embedded Systems, provides support for specification, design, verification and validation of these systems Models proved to be helpful to describe complex systems in a structured way. Due to the strict syntax of the modeling notation, the specification is less ambiguous and, thus, less error-prone. However, usually models do not cover all aspects of the system under test. Especially non-functional requirements and constraints are often described in natural language Models in SW Processes (20 min, K2) MBT relies on methods that have been established in SW engineering over the last 30 years. Therefore, the trainee should understand the basic concepts and principles of common standard frameworks: Object-oriented analysis / Object-oriented design (OOA/OOD) In OOA modeling techniques are used to analyze a task (e.g. business process), a system and/or requirements. In contrast to that, OOD elaborates the analysis models to produce implementation specifications. Model-driven architecture / Model-driven development (MDA/MDD) The two terms are often used as synonyms. Both approaches focus on code generation from models. These models may be either graphical or textual using a domain-specific language. These approaches use models on the left hand side of the V-model. It depends on the design phase (and thus on the level of abstraction) which model type is optimal. MBT introduces models on the right hand side of the V-models. Test models may either be obtained by re-using design models (and augment them with testrelevant information) or by writing dedicated test models. As for design models, the test phase (and thus the level of abstraction) influences the model type that fits best. In the W-model, test models are written on the left hand side. This possibility to start early with the test specification is one of the major advantages of MBT. MBT may also be introduced into other process models. The top-down approach, the high degree of automation obtained by test case generation and the excellent maintainability of models are important arguments in favor of MBT in iterative and agile processes. In agile teams, the test model should be written by dedicated testers that may be part of the team or not Working with Models (90 min, K4) The following model engineering techniques exist: writing from scratch re-use of existing models reverse engineering (obtaining models from source code) model transformation (obtaining models from models using transformation languages, e.g. QVT) isqi CMBT Syllabus V2.3 May 14, 2013 Page 8 of 26

9 It is even possible to perform round-trip engineering with models (e.g. original model => generated source code => generated model). Obviously, model quality influences the quality of the resulting output (test cases, source code, other models ). Three aspects must be checked: syntactical quality (form), semantical quality (content) and pragmatical quality (adequacy). Quality criteria for models are ([Pfa2008], [Zei2007]): 1. Syntactical quality (form) a. correctness b. completeness c. consistency d. understandability / readability 2. Semantical quality (content) a. correctness b. completeness c. consistency (within the model and to other models) 3. Pragmatical quality (adequacy) a. appropriateness b. simplicity c. maturity d. maintainability e. verifiability and f. aptitude for tool support. The semantical and pragmatical quality of models may be checked manually during reviews (pair review, walkthrough, technical review and inspection). The syntactical quality is best checked with a tool (using features of the modeling tool and/or a model checker). These techniques of static analysis may be completed by simulations. Models may either be written top-down or bottom-up, depending on which tests are required first and what information is already available. Models should be organized in a hierarchical way. This also allows the re-use of nested diagrams in other parts of the model (or other models). Modeling guidelines are required to ensure consistency, common understanding of all stakeholders and aptitude for tool support. Note to training providers: A practical exercise of 60 minutes duration is required! 2.3 Test Models (280 min, K4) Terms modeling language, [test step], [verification point], [test oracle], [test data], [test configuration], test management information, environment model, system model, test model, usage model, [abstract test case], [concrete test case], notation, notation paradigm, generation strategies, selection strategies []: comes from CTFL isqi CMBT Syllabus V2.3 May 14, 2013 Page 9 of 26

10 Learning objectives: Test models definition and taxonomy LO Describe the specific aspects (taxonomy) of test models (K2) LO Describe reasonable mappings between the test process and its elements/artifacts and models that are used and/or created by the tester (K2) LO Recall test modeling standards (K1) LO Describe, how test models support a higher level of abstraction and thus improve readability, usability, completeness regarding the test objectives (K2) Approaches to test modeling LO Describe the concept and focus of different MBT approaches (system / test / environment models, system model-driven / test model-driven / separate models) (K2) LO Describe the advantages and disadvantages of independent system and test models (K2) LO Recall application focus of structural and behavioral test models (K1) LO Recall what models may be used for and what may be generated (K1) LO Recall model notation paradigms (K1) LO Recall the test process characteristics that influence the way test models are written (K2) Test model design and development LO Decide, which notation paradigm should be used for which purpose (K3) LO Analyze and understand test models of medium complexity (K4) LO Write behavioral test models for simple situations using e.g. a graph-based notation (K3) Managing models and tool support LO Recall what tools are required for which task (K1) Test case generation and selection LO Describe, why test case generation and selection strategies have to be applied and learn basic principles of generation and selection strategies (K2) Test case generation and selection LO Apply modeling best practices (K3) LO Recall the application, benefits and challenges of MBT in comparison to traditional testing (K1) Note to training providers: This section is based on and worked out along examples. isqi CMBT Syllabus V2.3 May 14, 2013 Page 10 of 26

11 2.3.1 Test model definition and taxonomy (40 min, K2) In MBT models are either used for automation of test activities and/or to describe artifacts within the test process. Structural models like class diagrams or classification trees contain a description of the structure that is to be tested (e.g. possible object constellations or combinations of classes). A behavioral test model like, e.g., a state machine, a sequence diagram, or an activity diagram may contain test steps, verification points, pre- and post-conditions, test data, test oracles, test objectives, test configuration and, possibly, test management information. Additionally, test components, types, instances, interfaces and ports can be modeled. Typical graph based notation elements are nodes, edges (transitions) and constraints. They are used to visualize pre- and post-conditions, data entries (test step and related test data) and expected results (verification points and test oracles). Specific standards exist for test models (UTP, ETSI, TTCN-3). It is also possible to define tool-specific or project-specific notations that must be documented in the modeling guidelines. According to the definition MBT is an umbrella of approaches where models are used as an engineering approach within the test process in order to describe and/or generate test artifacts ([Roß2010]). How these models look like, depend on various aspects. In order to categorize a model or a certain MBT approach engineers use a so called taxonomy. Based on publications like [Pre2005], [Roß2010], [Sch2007] and [Utt2011] one of the two following taxonomies may be used to categorize models. MBT taxonomy [Utt2011] Model o Subject (Environment, SUT) o Redundancy (Shared test & system model, Separate test model) o Characteristics (Deterministic / Non-Det., Timed / Untimed, Discrete / Hybrid / Continuous) o Paradigm (Pre-Post, Transition-based, History-based, Functional, Operational) Test Generation o Test Selection Criteria ( Structural Model Coverage, Data Coverage, Requirements Coverage, Test Case Specifications, Random&Stochastic, Fault-based) o Technology (Manual, Random generation, Graph search algorithms, Model checking, Symbolic execution, Theorem proving) Test Execution (Online, Offline) MBT taxonomy [Roß2010] and [Sch2007] Model o System model o Test model (Test basis model, Test case specification model) o Environmental model Test level o Unit level testing o Integration level testing o System level testing o Acceptance level testing Maturity level o Model-oriented testing o Model-driven testing isqi CMBT Syllabus V2.3 May 14, 2013 Page 11 of 26

12 o Model-centric testing Modeling approaches flows o System model-driven (System model only, System model to test model) o Test model-driven (Test model only, Test model to system model) o Independent models Test generation o Test cases o Test data o Both In the world of MBT the term MBT Scenario (see [Pre2005]) is also widely used. Especially the aspects Redundancy and Modeling approaches flow listed in the taxonomies above discern the scenarios. The term redundancy is related to the independency of the models used by testers from those created by other roles (e.g. system developer). Higher redundancy (separate models for system development and test) is directly related to higher defect-detection rate. Note to trainers: Show 2-3 examples of different MBT approaches and models in order to recognize the usage of the taxonomy by the attendees. Models support a higher level of abstraction and thus improve readability, usability and completeness of the test specification regarding the test objectives Approaches to test modeling (30 min, K2) As the taxonomies suggest, a common distinction is made between the environment model, system model, and test model (see [Roß2010]): The environment model describes aspects of the environment of the system The system model describes the system itself, both static and dynamic aspects The test model describes the system from the point of view of the tester. It contains the aspects relevant for describing and/or generating test cases. The test model can be quite close to the system model (e.g. for communication protocol tests) or completely different (e.g. describing the behavior of a manual tester and the system s reaction as stimulus and expected result). A very common approach of MBT is the use of a system model (either exclusively or to derive a test model). The system model is commonly derived from the system s requirement specification. The test model and/or the test system (or parts of both) may be generated from the system model. Also, it is possible to generate the system itself (or parts of it). An advantage of the system model-driven approaches is that one model only is used as a single source. Thus, modeling expenditures are reduced and inconsistencies between the system model and the test model are limited. However, it bears also a big disadvantage: since the system model and the test basically coincide, the independence of development and testing is missing: the test can only check that the system model is correctly, but cannot identify errors that have been introduced while developing the system model from the requirements (validation). In a test model-driven approach the test model is manually developed on basis of the system s specification. Test generation is used to generate the test system (e.g. simulators). The test model may even be used to generate (parts of) the system model or the system itself, which can be seen as an implementation of a testdriven development approach. The test model-only approaches offer the advantage of a dedicated model for isqi CMBT Syllabus V2.3 May 14, 2013 Page 12 of 26

13 testing, which is able to represent the tester s needs. It is however not easy to validate such a stand-alone test model with respect to correctness, test coverage, etc. Instead, test model guidelines and rules are checked. The test model to system model approach has the same critical single source problem as the system model to test model approach. If however the two principles test first and model based are to be applied, the test model-driven approaches would be the solutions. In general, the main advantage of all single source approaches are the reduced development costs to obtain the other model. On the other side, this is the most significant disadvantage, since errors which are introduced into the source model will be propagated into the other model. Using two separate and independent models, i.e., a system model for the development activities, and a test model for the test activities, is considered the best approach. In this case, system and test model really have different content (as implied by the definition given above). Both models can be checked for correctness. Also the consistency between the two can be verified. Detected discrepancies between the two models are a clear indicator that the underlying requirements may be interpreted in several ways. Furthermore coverage criteria can now be established with respect to both models. The apparent advantages come with increased efforts for model development. As mentioned in section test models may describe the structural or the dynamic aspects (workflow, scenarios, behavior) of a system. Structural and behavioral models have a different application focus. Test models may be used to plan the test workflows, to generate the test cases without detailed instructions or to generate the complete test specification The second variant is sometimes called test design model (Testbasis-Modell), the third is called test specification model (or test case model, Test-Spezifikationsmodell). Test case models may be used to generate either abstract or concrete test. Abstract test cases do not contain exact values for test data, but for instance equivalence classes (i.e. the allowed data ranges for the test data), whereas concrete test cases require the modeling of concrete test data within the model. The choice and concepts for an MBT approach highly depend on the characteristics of the test process: Application focus (structural / data-oriented / behavioral / usage-based) Platform dependency (platform-independent / platform-dependent) Requirements type (functional / non-functional) Determinism (deterministic / non-deterministic) Timing behavior (timed / untimed) Continuity (discrete / hybrid / continuous) Test case abstraction (abstract / concrete) Based on the system under test together with domain specific best practices and well established techniques and together with application or company specific processes, a concept for the model notation paradigm has to be specified. The notation paradigm can be: state-based notation transition-base notation isqi CMBT Syllabus V2.3 May 14, 2013 Page 13 of 26

14 domain-specific language history-based notation functional-based notation operational notation stochastic notation data-flow notation business process notation scenario-based notation (specification by example) One or multiple notation types can be used for a certain testing scope. For the latter option, concepts for model transformations or tracing within the model have to be specified Test model design and development (120 min, K4) Some preliminary thoughts are necessary prior to test model design: What do we want to achieve? Are models appropriate to achieve these goals? What is the underlying testing approach (risk-based, requirements-driven, system design-based )? What exactly do we want to test (static structure/behavior, requirements/risks/workflows/, verification/validation )? What information shall be contained in the model (test steps, verification points, test data, test oracles, test objectives, (varying) test configurations, requirements/risks, test management information)? What information shall be kept outside the model (e.g. test oracle or test data)? What tools shall be used and what modeling constraints result from the tool usage? Input artifacts for test models are: documents (textual requirements, risk analysis, user guides ), existing system models for behavior and/or structure (that may be re-used or not), intended system usage, tester s mindset, test management decisions (e.g. priorities), change requests, process requirements (modeling guidelines, traceability concept, templates ). Links to requirements and risks can be set visually in the model for better traceability. Test models development is an iterative process. The model acts as communication basis with the stakeholders. Feedback to requirements can be expected. Note to training providers: A practical exercise of at least 60 minutes duration is required! Managing models and tool support (10 min, K1) Test models must be included to the project documentation e.g. by including figures into existing documentation (e.g. test plan). The model itself must be held under configuration management. The minimum tool required for MBT is an editor. If artifacts shall be generated, a generator is required. Basically, three tools categories can be distinguished: model-based test data generators, model-based test case editors and model-based test case generators. If a tool is also executing the generated test cases we can isqi CMBT Syllabus V2.3 May 14, 2013 Page 14 of 26

15 speak of a model-based test case executor. If it also gives a verdict to the observation which is made at the system under test (like pass/fail), we can speak of a model-based test case executor and validator. When doing on-the-fly testing, the test case generation, execution, and validation is one aggregated process, so here we cannot make a clear distinction between the three activities. Other tool support may be: model checker, configuration management tool, test management tool, test automation tool, requirements management tool, defect tracking tool, and other process supporting software Test case generation and selection (20 min, K3) In practice full combinatorial generation of test cases from a model rapidly leads to test case explosion. Therefore, strategies must be developed to generate fewer test cases without loss of quality. Thus test case generation and selection strategies have to be applied. The main drivers for test case selection are risks (What should be / must be tested?) and value (What may be left out without loss?). Risks may be either the formal risks identified in the risk analysis of the system or risks of the test process that include several perspectives (data integrity, safety, security, business, time, etc.). The test case selection is then expressed by coverage criteria (against the test basis). Possible test case generation strategies are requirements coverage and test management based, structural model coverage (control flow / transitions, data flow), data coverage, statistic test case generation or explicit test case generation (=exactly this test case). Control-flow-based coverage criteria are well known from white box testing and described in CTFL. In a brief example model, the coverage criteria from the CTFL syllabus are demonstrated including their impact and meaning for MBT. Other advanced test case generation strategies are: on-the-fly testing, evolutionary / genetic algorithms and fuzzy testing Known difficulties and common best practices (60 min, K3) In the following models means test models. Depending on the used modeling notation, not all paths through the model (for behavioral models) or instances of the model (for structural models) are meaningful test cases. Unwanted paths or instances can (and should) be excluded, e.g. using guard conditions, i.e. Boolean expressions that have to be true for the corresponding path to be valid. In behavioral models, loops usually lead to test cases with limited added value (test case explosion).the following best practices may help to avoid common pitfalls: Remember that models are always pragmatic. Test models should be designed to meet the test objectives. Leave out everything that is irrelevant for the test (even if it is an important functionality of the system). On the other hand, the test model may describe situations that should never occur in reality (which is exactly what should be tested). isqi CMBT Syllabus V2.3 May 14, 2013 Page 15 of 26

16 There is not the one-and-only test model. Write different test models that focus on specific aspects and condense the rest. Test models are an important part of the test documentation. Review them thoroughly (especially, if used for test case generation). Always establish modeling guidelines first. This includes the definition, what information shall be contained in the model and where it may be found. Use hierarchical models, but do not push this too hard. If there are too many levels, it becomes difficult to keep the overview. A single diagram should not exceed one page. Also, do not use too many different element types. This makes it easier to understand and to review. Be cautious when using loops. They rapidly lead to test case explosion. Use alternative notations instead (e.g. linearized flow). Named transitions help to describe (and select) specific paths in the model. Major benefits of MBT with respect to traditional testing are the structured approach, the higher abstraction level, better documentation of the test idea, improved communication, early feedback to requirements and increased efficiency and effectiveness due to automation of time-consuming tasks. Major drawbacks are the necessity of training and tool support, limitations in model application (e.g. usability tests), limitations in tool support (e.g. interpretation of parallelisms in the model) and the initial threshold of process changes in general. 2.4 Model-Based Test Processes (175 min, K3) Terms model-oriented testing, model-driven testing, model-centric testing, [fundamental test process], model-based test data generator, model-based test case editor, model-based test case generator Learning objectives: Test models in the fundamental test process LO Describe, how MBT influences/changes the fundamental test process (K2) LO Explain the necessity to really integrate MBT into the fundamental test process (K2) LO Describe, how models fit into the development lifecycle management (K2) Implementing MBT to an organization LO Recall the differences in maturity of model-based test processes (model-oriented / - based / -centric testing) according to the role of models within the process (K1) LO Know and address key success factors to implement a model-based test process (K3) ROI considerations LO Describe the success factors, the possibilities and limits and the economic aspects of MBT (K2) MBT tool support LO Describe the MBT tool landscape (K2) isqi CMBT Syllabus V2.3 May 14, 2013 Page 16 of 26

17 LO Describe the different tool categories (K2) LO Describe the impact of the selection of wrongs tools (K2) LO Describe the tool evaluation process (K2) MBT tools in practice LO Have a feeling of how the daily work in model-based test processes looks like (K1) Test models in the fundamental test process (35 min, K2) MBT may be introduced on all test levels. Models may be used for static and dynamic testing (with the exception of component and deployment diagrams that are only meaningful for static tests). Nevertheless, there is no stringent rule that maps model taxonomy to the test level or functional application domain for the test model. The impact of model usage on the test process increases with increasing artifact generation and organizational sponsoring. As described in section 2.1.5, the fundamental test process consists of five main activities: Test planning and control o The introduction of MBT must be taken into account in the risk analysis (both as potential source for new risks and as possible mitigation). o The test strategy must define where MBT shall be applied and provide rational for its use. o Modeling activities must be taken into account in the effort estimation. o The test concept must contain information on required tools. o Possibly, trainings must be planned. o Model-based metrics should be defined to measure effectiveness, efficiency and quality of MBT. o Model development must be controlled. Version management for models is required. Change and defect management must be planned. Test analysis and design o Obviously, this activity is highly affected by the use of models. Here, the dimensions Engineering approach and Redundancy of the MBT process taxonomy play an important role. Test implementation and execution o It strongly depends on the degree of artifact generation how far the test implementation activity is changed. o Test cases (test scripts / test data) may be generated automatically. o The introduction of MBT facilitates the introduction of other automation and/or process supporting tools. Also, MBT can very well be combined with keyword-driven testing. o The test model is a helpful tool for test management (prioritization, test case generation strategies). o With deeper insight the test model must possibly be modified. Evaluating exit criteria and reporting o Failures found can originate from the system under test, but can also originate from failures in the test model (= the test design) o The impact of faults and changes may be traced back into the model. isqi CMBT Syllabus V2.3 May 14, 2013 Page 17 of 26

18 o Model coverage serves as metric for the test progress and as (one) exit criterion for the test. o Faults may be clustered. o Traceability information from the model is an important part of the documentation (e.g. verdict for a given requirement). Test closure activities o The effectiveness, efficiency and quality of MBT should be assessed. o Model libraries may be established for re-use in future projects. Model-based and traditional tests will always co-exist. Thus, additional effort must be planned to synchronize both processes. Common templates must be created. Configuration (at least version) management must be set up for both. If artifacts are generated we face an additional challenge: the generated artifacts may be very different in format (e.g. paper-based test specifications), but still belong together. This implies that either both the model and the generated artifacts or the model and the generator must be versioned and archived together. Depending on the tool support configuration management may also be set up for sub-models or model elements. In short, the lifecycle management of test models is very similar to the one for documents or code. Models should be reviewed and released. Changes should follow a well-defined change management process. Defects must be traceable to the related model element. The tool chain must be controlled. Test models may be used to plan the test workflows, to generate the test cases without detailed instructions or to generate the complete test specification Implementing MBT to an organization (60 min, K3) The major obstacles for the successful introduction of MBT are unrealistic expectations, unclear goals, selection of inappropriate tools, missing knowledge of mechanisms to manage test case explosion and, finally, overhastiness. If the existing test process is immature, MBT will not solve the problem. Key success factors for the introduction of MBT are well defined, measurable goals and acceptance by all stakeholders. Introducing MBT is a typical process improvement activity that should be planned and not be rushed. Maturity models provide guidance for stepwise improvement. Based on the concept of TPI a maturity model may be defined for MBT, identifying three maturity levels (with increasing maturity from left to right): Artifact generation Sponsor model-oriented testing model-driven testing model-centric testing none none partly partly complete complete individuals project management project management organization project management *) organization *) Individual projects pass to complete artifact generation, while the rest of the organization remains at partly. isqi CMBT Syllabus V2.3 May 14, 2013 Page 18 of 26

19 In model-oriented testing, models are used for analysis and documentation. Depending on the sponsor the method may be anchored in the project s test process or not. MBT is not part of the organizational best practices. The maximum detailing of the test model is blueprint. If test cases or test data is derived from the model this is done manually. In model-driven testing, a generator is used to (at least partly) derive artifacts from the model. Models are more and more integrated in the organizational test process, passing from optional use to mandatory (e.g. for safety-relevant components). They are integrated into the documentation. Training programs are set up, since modeling competencies become an essential tester s skill. The test model s detailing ranges from blueprint to program. In model-centric testing MBT becomes more and more strategic for the organization. The test model acts as single repository for all test relevant information. It is the master document and must always be up-to-date. Model libraries are established to foster re-use of models. The test model s detailing is program. Information (e.g. from test execution) is mirrored back into the model. On the highest maturity level, the tool chain becomes bi-directional. Even the lowest maturity level (model-oriented testing) represents already a considerable added value thanks to the visualization of complex problems. It has also the advantage that the initial threshold is quite low. The MBT introduction is done in five steps: 1. The existing process (use cases, actors / roles, artifacts, existing tools) is analyzed. Also, the stakeholders must be identified. At the end of this analysis it must be clear what shall be achieved with MBT (goals and scope) and how success shall be measured (metrics). 2. The new method is specified, resulting in a modified test process description. The impact of changes and the necessity for training is analyzed. A tool evaluation is conducted against previously defined evaluation criteria. 3. The new test process is implemented. This usually requires some tool customization, either to support the process or to integrate the new tool into the existing tool landscape (e.g. for automated test execution). Also, roles and user access rights must be set and managed. Training material is provided. 4. A pilot project is conducted, thus validating the processes and roles. The first users are trained. At the end of the pilot the efficiency and cost effectiveness of MBT is assessed. In safety-critical domains, process validation must be planned and performed. 5. Finally, the new test process is rolled out ROI considerations (30 min, K2) Of course, MBT is not for free. Costs occur both during the introduction and the operating phase. They can be split into cost of manpower and tool costs (licenses) or in initial and forthcoming costs. These costs are counterbalanced by the following benefits: Early defect or failure identification The earlier a fault is detected, the cheaper it is to fix. In particular, MBT helps to identify missing or immature requirements (which are still the most important cause for project failure). Automated test case generation Test design and implementation is becomes more efficient. Changes are performed at one central location and will be automatically transferred to all test cases that are affected by the change. The isqi CMBT Syllabus V2.3 May 14, 2013 Page 19 of 26