DSL Implementation. ... with language Workbenches. v1.1 Jan 16, Markus Voelter independent/itemis

Transcription

1 DSL Implementation... with language Workbenches. v1.1 Jan 16, 2013 Markus Voelter independent/itemis +Markus Voelter

2

3 Last Year s Talk This Year s Talk Design No Tools The why Implementation 3 Tools The how

4 Last Year s Talk This Year s Talk Design No Tools The why Implementation 3 Tools The how Both taken from my new book: available Feb 2013

5

6 Intro- duction

7 Language Workbench (Mar1n Fowler)

8 Language Workbench (Mar1n Fowler) Freely define languages and integrate them

9 Language Workbench (Mar1n Fowler)? use persistent abstract representa1on

10 Language Workbench (Mar1n Fowler) language ::= + schema + editors generators

11 Language Workbench (Mar1n Fowler)? projec1onal edi1ng

12 Language Workbench (Mar1n Fowler) persist incomplete or contradictory informa1on

13 Language Workbench (Mar1n Fowler) + powerful edi1ng tes1ng refactoring debugging groupware language defini1on implies IDE defini1on

14 Language Workbench (Mar1n Fowler) support for classical programming and + classical modeling

15

16 more in GPLs more in DSL Domain Size large and complex smaller and well-defined Designed by guru or committee a few engineers and domain experts Language Size large small Turing-completeness almost always often not User Community In-language abstraction large, anonymous and widespread sophisticated small, accessible and local limited Lifespan years to decades months to years (driven by context) Evolution Incompatible Changes slow, often standardized almost impossible fast-paced feasible

17 General Purpose C Components State Machines Domain Specific Sensor Access LEGO Robot Control

18

19 Model or Code?

20 Model or Code? next [alt > 0] beforeflight reset crashed next [alt == 0 && speed == 0] airborne flying next [alt == 0 && speed > 0] landing next [alt == 0 && speed == 0] landed

21 Model or Code?

22 Model or Code?

23 Model or Code? Does it really matter? What is the difference? Who cares?

24

25 We don t want to model, we want to program!

26 We don t want to model, we want to program! at different levels of abstraction from different viewpoints integrated!

27 We don t want to model, we want to program! with different degrees of domain-specificity with suitable notations with suitable expressiveness

28 We don t want to model, we want to program! and always: precise and tool processable

29

30 Big Language o a b n c m k L d e j f i h g with many first class concepts!

31 Small Language α δ L β ω λ with a few, orthogonal and poweful concepts

32 Modular Language α my L d e f g h i j k l β with many optional, composable modules

33

34 Example Tools

35 Example Tools Parser-based LL(k)

36 Example Tools Projectional MPS

37 Example Tools Parser-based GLR Spoofax

38

39 Syntax

40 Concrete and Abstract

41 Parsing Parsing

42 Parsing vs. Projection Parsing Projection

43

44 Parsing BNF, EBNF Grammars

45 Parsing Grammar Classes LL }Limitations LL(k) wrt. all LL(*) Context Free LR Grammars

46 Parsing Grammar Class Limitations Left Recursion! Not supported by LL Parsers

47 Parsing Grammar Class Limitations Left Recursion! Not supported by LL Parsers -> Left Factoring

48 Parsing Grammar Class Limitations Alternative: GLR or Earley parsers -> parses all CFGs -> O(n), O(n 3 ) worst case -> LL(1)

49 Parsing Ambiguities & Grammar Classes

50 Parsing Ambiguities & Grammar Classes -> LR Problem: Precedence encoded in grammar structure! -> LL

51 Parsing Spoofax Ambiguities & GLR Parsers Better: Declarative, Extensible

52

53 Projection MPS Very Flexible Syntax

54 Projection MPS Very Flexible Syntax

55 Projection MPS Very Flexible Syntax Challenges: Editor Usabililty, Infrastructure Integration

56 Projection MPS Editor Usability Traditionally a challenge. Mostly solved in MPS.

57 Projection MPS Editor Usability Traditionally a challenge. Mostly solved in MPS. Aliases Side Transformations Smart References Smart Delimiters Delete Actions Wrappers Create Ref Target

58 Projection MPS Infrastructure Integration

59

60 AST Formalisms The Essence Things with properties child Things references to other Things extending other Things

61 AST Formalisms Xtext Ecore

62 AST Formalisms MPS MPS Structure

63 AST Formalisms Spoofax Spoofax Aterms

64

65 Scoping

66 Typical Approach Today s Approach (Xtext or MPS) Scope = Function that returns the possible target elements for a referenece

67 Typical Approach Today s Approach (Xtext or MPS) Scope = Function that returns the possible target elements for a referenece Implemented as procedural code.

68 Typical Approach Today s Approach (Xtext or MPS) Scope = Function that returns the possible target elements for a referenece Implemented as procedural code. Associated with context directly, via naming convention or procedurally.

69 Typical Approach Xtext Today s Approach (Xtext or MPS) Xtext

70 Typical Approach MPS Today s Approach (Xtext or MPS) MPS

71

72 New Idea Using DSLs for defining DSLs Use dedicated DSLs to define aspects of DSLs

73 New Idea Using DSLs for defining DSLs Use dedicated DSLs to define aspects of DSLs Old idea for grammar/syntax.

74 New Idea Using DSLs for defining DSLs Use dedicated DSLs to define aspects of DSLs Old idea for grammar/syntax. Now increasingly used for other aspects such as scoping (we ll see type system, debugging later)

75 Spoofax NBL Spoofax Defining Namespaces Define Namespaces: Qualified Names

76 Spoofax NBL Spoofax Defining Namespaces Name Qualification implies Uniqueness Limitations on Visibility

77 Spoofax NBL Spoofax Defining References

78 Spoofax NBL Spoofax Defining References Not clear which Entity!

79 Spoofax NBL Spoofax Defining References Integration with type system required.

80

81 Constraints

82 Standard Constraints Xtext Boolean Expressions Xtext CheckType.NORMAL CheckType.FAST CheckType.EXPENSIVE

83 Standard Constraints MPS Boolean Expressions MPS Evaluated only on demand via ReadListeners

84 Standard Constraints Spoofax Boolean Expressions Spoofax Based on Pattern Matching -> very concise

85

86 Dataflow Analysis The problem Is code unreachable? Is a variable read before it is written? Can a value be null when it is read? Can we know statically that a variable always has a constant value (and hence can be optimized)? -> Dataflow Analysis

87 Dataflow Analysis MPS The Dataflow Graph

88 Dataflow Analysis MPS The Dataflow Graph

89 Dataflow Analysis MPS The Dataflow Graph

90 Dataflow Analysis MPS for the IfStatement

91 Dataflow Analysis MPS for the IfStatement

92 Dataflow Analysis MPS for the IfStatement

93 Dataflow Analysis MPS The actual Anylses predefined you can also build your own ones

94

95 Type System

96 Type Systems? Assign fixed types

97 Type Systems? Derive Types Assign fixed types

98 Type Systems? Derive Types Assign fixed types Calculate Common Types

99 Type Systems? Assign fixed types Derive Types Calculate Common Types Check Type Consistency

100

101 Classic Approach Xtext based on Functions/Recursion Example Code:

102 Classic Approach Xtext based on Functions/Recursion Example Code: Type System Implementation:

103 Classic Approach Xtext based on Functions/Recursion Better Type System Implementation:

104

105 Declarative Type Sys. MPS Unification/Solver (MPS) Example Code: Type System Implementation:

106 Declarative Type Sys. MPS Unification/Solver (MPS) Example Code: Type System Implementation: Example Scenario 1

107 Declarative Type Sys. MPS Unification/Solver (MPS) Example Code: Type System Implementation: Example Scenario 2

108 Declarative Type Sys. MPS Unification/Solver (MPS) Example Code: Type System Implementation: Example Scenario 3

109 Declarative Type Sys. MPS Unification/Solver (MPS) Example Code: Type System Implementation: Example Scenario 4

110 Declarative Type Sys. MPS Automatic Type Inference Example Code:

111 Declarative Type Sys. MPS Automatic Type Inference Example Code: Type System Implementation: That is all!

112

113 Declarative Type Sys. Spoofax Pattern Matching

114 Declarative Type Sys. Spoofax Pattern Matching

115 Declarative Type Sys. Spoofax Pattern Matching

116 Declarative Type Sys. Spoofax Pattern Matching

117

118 Transfor- mation & Generation

119 Text Template Engine Xtext Xtext s Xtend (M2T, M2M) Not just a template language like StringTemplate or Xpand. Actually a very nice and fullblown GPL with special support for text templating and model navigation, query and construction.

120 Text Template Engine Xtext Xtext s Xtend (M2T, M2M) Java Like Many improvemnts over Java Modularization via DI Template Expressions Smart Whitespace Handling Functional Abstractions Dispatch on Arguments Extension Methods Builders

121 Text Template Engine Xtext Xtext s Xtend

122 Text Template Engine Xtext Dispatch Methods

123 Text Template Engine Xtext Builder Syntax (1)

124 Text Template Engine Xtext Builder Syntax (2)

125 Model Inferrence Xtext Xtext, Xbase and Java Instead of generating Java source, M2M to Java AST (for structurally relevant parts)

126 Model Inferrence Xtext Xtext, Xbase and Java Instead of generating Java source, M2M to Java AST (for structurally relevant parts) You get code gen implicitly.

127 Model Inferrence Xtext Xtext, Xbase and Java Instead of generating Java source, M2M to Java AST (for structurally relevant parts) You get code gen implicitly, but also: scoping type system debugger

128 Model Inferrence Xtext Xtext, Xbase and Java

129 Model Inferrence Xtext Xtext, Xbase and Java

130 Model Inferrence Xtext Xtext, Xbase and Java

131 Model Inferrence Xtext Xtext, Xbase and Java

132 Model Inferrence Xtext Xtext, Xbase and Java Java text for the body/ implementation. (no IDE support here)

133

134 Multi-Stage Trafo MPS MPS Transformations

135 Multi-Stage Trafo MPS MPS Transformations

136 Transformations MPS with concrete Syntax & IDE IDE support for target language Macros contain template code M2M with M2T feel

137 Transformations MPS with concrete Syntax & IDE

138 Transformations MPS with concrete Syntax & IDE

139 Transformations MPS with concrete Syntax & IDE

140 Transformations MPS Builders are supported as well Language Extension: No builder functions necessary IDE support But is specific for Models

141

142 Pattern Matching Spoofax For text generation

143 Pattern Matching Spoofax For transformation

144 Pattern Matching Spoofax M2M with concrete Syntax

145 Pattern Matching Spoofax M2M with concrete Syntax Needs escaped composition grammar!

146

147 Inter- pretation

148 Interpreters Xtext Xtend is very useful! Java Like General Purpose Language Many improvemnts over Java Functional Abstractions Dispatch on Arguments Extension Methods Builders Template Expressions Smart Whitespace Handling Trafo Modularization via DI

149 Interpreters MPS MPS interpreter extension makes writing dispatchers much simpler modular extension!

150 Live IDEs MPS Embedding Interpreters real-time feedback in the IDE!

151

152 IDE Services

153 Syntax Highlighting MPS Declarative Approach

154 Syntax Highlighting MPS Declarative Approach

155 Syntax Highlighting Spoofax Declarative Approach

156 Find References MPS Custom Finders

157 Find References MPS Custom Finders

158 Formatting Spoofax Declarative Box Language

159 Visualization Xtext Jan Koehnlein s GraphView

160

161 Testing

162 Parser/Syntax Testing Spoofax Declarative Approach abstract syntax concrete syntax

163 Constraints/Type Sys. MPS Declarative Approach Test annotations on the test subject very concise.

164 Testing Semantics MPS Expressing tests in language Generate/Interpret the tests along with the application code

165 Testing Editors Xtext Using Moritz Eysholdt s Xpect Test expectations in comments. Can test various language aspects such as scopes, code completion, type system; also extensible.

166

167 Inspecting/ Debugging

168 Inspecting Programs MPS Parsing doesn t happen!

169 Debugging DSLs MPS Behavior

170 Debugging DSLs MPS Type System Declarative Type Systems use Equations and a Solver. Equations + Solver State can be visualized (not always intuitive)

171 Debugging DSLs Xtext Transformations & Code Gens

172 Debugging DSLs MPS Transformations Multi-Level, Declarative: show intermediate models and the templates involved in trafo.

173 Debugging DSL Programs Xtext Xbase Automatic Debugging Make your DSL extend Xbase Use a JVMModelInferrer -> Java you then get Debugging for free Mostly. Sometimes you have to manually establish traces.

174 Debugging DSL Programs MPS Extensible Debuggers

175 Debugging DSL Programs MPS Extensible Debuggers Language Extension Generated C Code

176 Debugging DSL Programs MPS Extensible Debuggers Stepping

177 Debugging DSL Programs MPS Extensible Debuggers Watches

178 Debugging DSL Programs MPS Debugger Architecture

179

180 Modularization, Extension and Composition

181 Language Extension MPS C extensibility in mbeddr Plain C

182 Language Extension MPS C extensibility in mbeddr C + Units Plain C

183 Language Extension MPS C extensibility in mbeddr C + State Machines

184 Language Extension MPS C extensibility in mbeddr C + State Machines and Units! -> combination of independent extensions!

185 Language Extension Xtext Embedding Xbase into DSL Entities using Java types and Xtend code in the body of operations.

186 Language Extension Xtext Embedding Xbase into DSL

187 Language Extension Xtext Embedding Xbase into DSL See before results in generator, scoping, typing, etc. with little additional work.

188 Language Extension Xtext Extending Xbase with Dates Grammar Repeat complete Xliteral rule, adding the XDateLiteral

189 Language Extension Xtext Extending Xbase with Dates Type system

190 Language Extension Xtext Extending Xbase with Dates Interpreter Compiler

191

192 Embedding MPS Useful for Meta Data No Dependencies on anything.

193

194 Meta

195 Extending the Tool MPS since it is bootstrapped!

196 Extending the Tool MPS since it is bootstrapped! Extending MPS is (almost) like extending any other language!

197 Extending the Tool Xtext using Platform APIs Jnario custom folding editor.

198

199 The Web

200 Web-based DSL Editing Challenges (Re-)Implementing LWB in the Browser Browser-Based Parsing Tier-Splitting Multi-User (real-time?) A Research Agenda:

201 Web-based DSL Editing Xtext Xtext Prototype Based on Orion An Eclipse-Instance On server per client Developed in a research project

202 Web-based DSL Editing MPS* JetBrains/MPS Generic Projectional Editor being developed; driven by Youtrack Workflows Java-Scipt based (completely in Browser) Will be made compatible with MPS language definitions

203

204 Other Tools

205 Intentional

206 Intentional

207 SugarJ (Uni Marburg)

208 Language Workbenches Language Workbench Comp {etition arison} Dozens of LWBs compared based on the same example languages

209

210 The End. voelter.de +Markus Voelter