Testing DSLs How to test DSLs, their IDEs and Programs

Transcription

1 Testing DSLs How to test DSLs, their IDEs and Programs Markus Voelter independent/itemis +Markus Voelter

2 A DSL is a focussed, processable language for describing a specific concern when building a system in a specific domain. The abstractions and notations used are natural/suitable for the stakeholders who specify that particular concern.

3 Limited Expressiveness. Reduced Need For Tests.

4 Constraint Checks. A Form of Test.

5 Test Semantics, not Structure! Example Models Generator Code Based On Test Cases Tests Binary

6 This tests only the generators! Generator Code Model Tests Generator Test Code

7 Separate test models and generated test code Model Generator Code based on tests Test Model (Test Language) Generator Test Code

8 Separate test models and generated test code Model Generator Code Mocks based on tests Test Model (Test Language) Generator Test Code

9 Testing the Language Testing Programs

10 Does the DSL cover cover? the domain it is intended to Build Examples and discuss with Stakeholders.

11 Interview Experts Get Feedback Structure their Knowledge Create Examples Build the Language

12 ? definition process Can the language all of the relevant notation? (Parser Test) Create and maintain example programs and keep trying to parse them

13 /p/xtext-utils/wiki/unit_testing

14 /p/xtext-utils/wiki/unit_testing

15 Do the scopes work?

16 work? Do the constraints and typesystem

17 ? with certain program Are there any error messages associated elements? Create positive and negative examples and check for correct error annotations.

18

19 tests

20 The method fortype( t ) get( index ) inline( line ) withstringfeaturevalue ( n, v ) errorsonly() named( n ) forelement( t, n ) under( t ) under( t, n ) r e t u r n s a n e w IssueCollection that contains only those issues that are attached to an instance of t are at position index in the IssueCollection are in line line in the model file whose feature named n has the value (tostring()) v contains no warnings contains only those issues that are attached to an element with name property value n contains only those issues that are attached to elements of The type method t that have the name sizeis( n s ) contains only those issues whose element has an ancestor of type t oneofthemcontains( t ) contains only those issues whose element has an ancestor of type t named n allofthemcontain( t ) theoneandonlycontain s(t) asserts that t h e s i z e o f t h e c u r r e n t collection is s the collection has any size, and one of the error messages contains the substring t the collection has any size, and all the error messages contains the substring t the collection is of size 1 and the message of the singe error contains the substring t

21

22

23

24

25 ? Do the generators, transformations or interpreters work? Run them; execute test cases against the running code.

26 You can write test cases in the target language manually. Or you can express test cases on the model level

27

28 Interpreting the tests directly in the IDE reduced turn around time.

29 Behavior

30

31 Behavior

32 Behavior

33 Multiple Mappings at the same time L D L x L y L z Similar Semantics?

34 Multiple Mappings at the same time L D T Similar Semantics? L x L y L z T T T all green!

35 Multiple Mappings alternatively, selectably L D Extend L D to include explicit data that determines transformation L x L y L z

36 Multiple Mappings alternatively, selectably T L D TESTING! L x L y L z T T T

37 ? cannot execute the Generators: What do I do if I programs?

38 ? cannot execute the Generators: What do I do if I programs? Then (and only then!) use structure analysis

39 ? editor perform its Does the actual services correctly? This is highly tool specific

40

41 Moritz Eysholdt s Xpect

42 ? Can I do proofs, and not just trial and error testing? Yes. Formal Methods.

43 Model Checking

44 Challenges

45 State Based

46 mbeddr Approach

47 mbeddr Approach easier to use hopefully used more full power: write CTL/ LTL if you want to

48 Model Checking

49 Model Checking

50 Model Checking

51 Model Checking Finds problems in state machines. even ones you didn t think of! Much more complete than manual testing.

52 Model Checking

53 mbeddr Future

54 mbeddr Future SAT Solving for PLE Variability Decision Table Completeness

55 Abstract Execution Klaus Birken Executing a Program using all possible values and execution paths at the same time

56 Abstract Execution Klaus Birken { [1..3] } + { [10..15], 30 } = { [11..18], [31..33] }

57 Abstract Execution Klaus Birken b ==?

58 Abstract Execution Klaus Birken a == { [7..12] } a == { [13..20] } b == { [7..12], 2011 }

59 Abstract Execution Klaus Birken customer requirements validated against / based on validated against / based on software model online validation with abstract execution abstract testcases model

60 Abstract Execution Klaus Birken customer requirements validated against / based on validated against / based on software model online validation with abstract execution abstract testcases model transformation transformation (generated) system automated black-box regression tests testcases

61 Abstract Execution test for implementation of Audio_Routing_Contr ol service Klaus Birken abstract parameter specification specify reaction of underlying subsystem (with timings!) expected test result: positive response with 0 <= $Result < 100 Warning: implementation will respond with concrete value

62 Abstract Execution Klaus Birken code coverage for abstract execution of testcase against this implementation

63 Abstract Execution Klaus Birken fully explore the state space with abstract test cases maximise coverage, increase confidence, reduce risk tightly integrate development and test iteratively develop implementation and test cases, each uncovering incompleteness in the other

64 The End. This material is part of my upcoming (early 2013) book DSL Engineering with Language Workbenches Stay in touch; it will be cheap or maybe even free :-) +Markus Voelter