2016 / 2017 Model-based Testing User Survey: Results

Size: px

Start display at page:

Download "2016 / 2017 Model-based Testing User Survey: Results"

Jodie Wood
5 years ago
Views:

2 Contents 1 Overview About the 2016 MBT User Survey About this document Result overview Detailed Results... 3 Question 1: Degree of experience... 3 Question 2: Expectations... 4 Question 3: Does MBT fulfil those expectations?... 5 Question 4: Test level(s)... 6 Question 5: Type(s) of testing... 7 Question 6: Stage of MBT adoption... 8 Question 7: Use of models in other development phases... 9 Question 8: Type of models in those other development phases Question 9: Degree of reuse Question 10: Testing effort Question 11: MBT model focus Question 12: Degree of detail in MBT model Question 13: Notation type Question 14: Traceability Question 15: Generated artifacts Question 16: Test case generation Question 17: Selection criteria Question 18: Test execution Question 19: Tools integration Question 20: Role the model plays in MBT approach Question 21: How effective do you think MBT has been? Question 22: How likely are you to continue to use MBT? Question 23: Biggest limitation of MBT Question 24: Biggest advantage of MBT Question 25: Best estimate of the effects of MBT Question 26: Hours needed to become a proficient MBT user Question 27: ISTQB Certified Model-Based Tester Question 28: General application domain of the system under test Question 29: MBT project sponsor Question 30: Size of organization Question 31: Overall process approach Question 32: Role of respondents Page 1 of 34

3 1 Overview 1.1 About the 2016 / 2017 MBT User Survey This survey is a follow up to the 2014 MBT User Survey (cf. and the 2011/2012 MBT User Survey (cf. It was conducted from May to December model-based testing practitioners answered, providing a relevant picture of the current state of the practice in the MBT area. The purpose of the survey was to collect data and experience on the usage of Model-based Testing on a regular basis. We want to learn how MBT users view the efficiency and effectiveness of the approach, what works, what does not work and how MBT evolves. 1.2 About this document This document summarizes the results of the 32 questions. For each question, we first provide a graphical representation of the 2016 result together with some raw data for both 2016 and For numerical answers, we provided both the average and the median. The median indicates the midle of a distribution, that is, the point where there are as many answers below as above. It is less sensitive to extreme answers and usualy more informative than the average. MBT tools are out of scope of the survey and, thus, not mentioned in this document. Please, let us know if you have any comments, question or remarks regarding this MBT User Survey Anne Kramer, sepp.med gmbh - anne.kramer@seppmed.de Bruno Legeard, Smartesting & University of Franche-Comté bruno.legeard@smartesting.com Robert V. Binder - rvbinder@gmail.com 1.3 Result overview In 2016, you have been less to answer the survey, but those who did proved to be more persevering, meaning that fewer questions were skipped. When comparing the results to those from 2014, we observe a shift from towards a more intense use of MBT in system testing instead of integration testing (question 4). Moreover, the MBT models seem to contain fewer details than stated in 2014 (question 9). Finally, but not least, more respondents replied to employ on-the-fly execution of model-based test cases (question 18). The clearest trend, however, concerns the overall process approach. In 2014, more than 55% of the respondents still relied on phased approaches. Since then, agile approaches have gathered even more adepts. This time, 62,5% of the respondents used MBT in an agile overall process approach (question 31). A part from those highlights, it is difficult to identify trends, especially if we take the statistical error taken into account. One answer was particularly surprising to us: one fifth of the respondents do not establish traceability between model elements and requirements/ use cases or features/ user stories (question 14) Page 2 of 34

4 2 Detailed Results Question 1: Degree of experience Page 3 of 34

5 Question 2: Expectations Page 4 of 34

6 Question 3: Does MBT fulfil those expectations? Page 5 of 34

7 Question 4: Test level(s) Page 6 of 34

8 Question 5: Type(s) of testing Page 7 of 34

9 Question 6: Stage of MBT adoption Page 8 of 34

10 Question 7: Use of models in other development phases Page 9 of 34

11 Question 8: Type of models in those other development phases Page 10 of 34

12 Question 9: Degree of reuse Page 11 of 34

13 Question 10: Testing effort Page 12 of 34

14 Question 11: MBT model focus Page 13 of 34

15 Question 12: Degree of detail in MBT model Page 14 of 34

16 Question 13: Notation type Page 15 of 34

17 Question 14: Traceability Page 16 of 34

18 Question 15: Generated artifacts Page 17 of 34

19 Question 16: Test case generation This question aimed at the test artifact generation process. For tool integration, see question Page 18 of 34

20 Question 17: Selection criteria i) Structural model coverage is a generic term for a variety of coverage criteria that are based on the internal structure of the model (e.g. transition coverage). ii) Data coverage focuses on the equivalence partitions of input and output data, possibly combined with a boundary value analysis. iii) Scenarios or patterns are specific paths through the model that correspond to frequently used functionality or fault-based scenarios. iv) Project-driven test case selection criteria uses specific additional information added to the model (e.g. risk, priorities ) to drive test generation to achieve specific test objectives for the project Page 19 of 34

21 Question 18: Test execution Page 20 of 34

22 Question 19: Tools integration Page 21 of 34

23 Question 20: Role the model plays in MBT approach Page 22 of 34

24 Question 21: How effective do you think MBT has been? Page 23 of 34

25 Question 22: How likely are you to continue to use MBT? Page 24 of 34

26 Question 23: Biggest limitation of MBT Page 25 of 34

27 Question 24: Biggest advantage of MBT Page 26 of 34

28 Question 25: Best estimate of the effects of MBT Page 27 of 34

29 Question 26: Hours needed to become a proficient MBT user Page 28 of 34

30 Question 27: ISTQB Certified Model-Based Tester Page 29 of 34

31 Question 28: General application domain of the system under test Page 30 of 34

32 Question 29: MBT project sponsor Page 31 of 34

33 Question 30: Size of organization Page 32 of 34

34 Question 31: Overall process approach Page 33 of 34

35 Question 32: Role of respondents Page 34 of 34

From Test Legacy to Model- Based Testing How to refactor an existing test repository into an MBT model?

From Test Legacy to Model- Based Testing How to refactor an existing test repository into an MBT model? UCAAT 2014 September 16/18 - Munich Arnaud Bouzy Bruno Legeard {arnaud.bouzy, bruno.legeard}@smartesting.com