Declarative Language Definition

Transcription

1 Declarative Language Definition (state-of-the-art & research agenda) Eelco Visser IPA Spring Days 2012 Garderen, Netherlands

2 Software Engineering Problem Domain Machine bridging the gap between problem domain and solution domain

3 High-Level Languages Problem Domain HLL Machine "A programming language is low level when its programs require attention to the irrelevant." Alan J. Perlis (1982) HLLs reduce gap

4 Domain-Specific Languages Problem Domain DSL HLL Machine domain-specific languages support more specialization

5 Software Language Design & Engineering enable software engineers to effectively design, implement, and apply domain-specific software languages

6 Research: Software Language Design systematically design domain-specific software languages with optimal tradeoff between expressivity, completeness, portability, coverage, and maintainability

7 Software Language Design Case Studies Mobl: client-side stateful web applications WebDSL: server-side restful web applications

8 Research: Software Language Engineering automatically derive efficient, scalable, incremental compiler + usable IDE from high-level, declarative language definition

9 The Spoofax Language Workbench creating full featured IDEs for domain-specific languages

10 Language Design

11 Language Design using Web; namespace Web { partial class URL { string GetLocation() { return location; namespace Blogger { class User { string name; URL homepage; class BlogPosting { User poster; string body; URL Homepage() { return poster.homepage; string Location() { return Homepage().GetLocation(); namespace Web { partial class URL { string location;

12 Test-Driven Language Design

13 Language Examples

14 Language Tests test compilation unit [[ using Web; namespace Blogger { class User { string name; class BlogPosting { User poster; string body; URL Homepage() { return poster.homepage; string Location() { return Homepage().GetLocation(); ]] parse

15 Declarative Syntax Definition

16 module Csharp imports Common exports context-free start-symbols CompUnit context-free syntax -> CompUnit {"CompUnit" %%...

17 Grammar Rules Using* Definition* -> CompUnit {"CompUnit" %% Namespaces "namespace" ID "{" Definition* "" -> Definition {"Namespace" "using" UsingPart ";" -> Using {"Using" ID -> UsingPart {"UsingPart" UsingPart "." ID -> UsingPart {"UsingPart"

18 Grammar Rules %% Classes "partial" "class" ID "{" ClassBodyDecl* "" -> Definition {"PartialClass" "class" ID "{" ClassBodyDecl* "" -> Definition {"Class" Type ID ";" -> ClassBodyDecl {"Field" Type ID "(" ")" "{" Stmt* "" -> ClassBodyDecl {"Method"

19 Grammar Rules %% Types ID -> Type {"Type" "string" -> Type {"StringType", prefer "int" -> Type {"IntType", prefer "void" -> Type {"VoidType", prefer %% Statements "return" Expr ";" -> Stmt {"Return" %% Expressions ID -> Expr {"FieldRef" Expr "." ID -> Expr {"FieldAccess" Expr "." ID "(" ")" -> Expr {"MethodCall" ID "(" ")" -> Expr {"ThisMethodCall"

20 Abstract Syntax Trees

21 Syntactic Editor Services module CSharp-Outliner outliner Definition.Namespace Definition.PartialClass Definition.Class ClassBodyDecl.Method module CSharp-Folding folding CompUnit.CompUnit Definition._ ClassBodyDecl.Method

22 Error Recovery

23 Robust Editors editor services based on parse/abstract syntax tree

24

25

26 Permissive Grammars exports sorts WATER WATERTOKEN WATERTOKENSTART WATERTOKENSEPARATOR WATERTOKENSTAR lexical syntax ~[A-Za-z0-9\_] -> WATERTOKENSTART {recover, avoid WATERTOKENSTART [A-Za-z0-9\_]* -> WATERTOKEN ~[A-Za-z0-9\_\ \t\12\r\n\*] -> WATERTOKENSEPARATOR {recover, avoid "*" -> WATERTOKENSTAR {recover, avoid WATERTOKEN -> WATER WATERTOKENSEPARATOR -> WATER WATERTOKENSTAR -> WATER -> WATEREOF lexical restrictions WATERTOKEN -/- [A-Za-z0-9\_] WATERTOKENSTAR -/- [\/] context-free syntax WATER -> LAYOUT {cons("water") Island Grammars [OOPSLA 09]

27 Permissive Grammars lexical syntax %% Lexical insertion recovery rules "\"" -> OQUOTE {recover OQUOTE StringChar* "\n" -> STRING {cons("insertend") OQUOTE StringChar* WATEREOF -> STRING {cons("insertend") "/*" -> ISTART {recover ISTART ( ~[\*] CommentChar )* WATEREOF -> LAYOUT {cons("insertend") -> "" {recover, cons("insertion") -> "using" {recover, cons("insertion") -> ";" {recover, cons("insertion") -> "return" {recover, cons("insertion")

28 Name Resolution

29 Reference Resolution using Web; namespace Web { partial class URL { string GetLocation() { return location; namespace Blogger { class User { string name; URL homepage; class BlogPosting { User poster; string body; URL Homepage() { return poster.homepage; string Location() { return Homepage().GetLocation(); namespace Web { partial class URL { string location;

30 Reference Resolution using Web; namespace Web { partial class URL { string GetLocation() { return location; namespace Blogger { class User { string name; URL homepage; class BlogPosting { User poster; string body; URL Homepage() { return poster.homepage; string Location() { return Homepage().GetLocation(); namespace Web { partial class URL { string location;

31 Testing Name Resolution test Resolve class reference [[ namespace TestsResolving { class [[B]] { [[B]] b; ]] resolve #2 to #1

32 Name Resolution: Classes & Types "class" "{" ClassBodyDecl* "" -> Definition {"Class" -> Type {"Type" test Resolve class reference [[ namespace TestsResolving { class [[B]] { [[B]] b; ]] resolve #2 to #1

33 Name Resolution: Namespace "namespace" "{" Definition* "" -> Definition {"Namespace" "using" ";" -> Using {"Using" UsingPart "." -> UsingPart {"UsingPart" -> UsingPart {"UsingPart"

34 test Resolve field [[ namespace TestsResolving { class B { B [[x]]; B f() { return [[x]]; ]] resolve #2 to #1 Resolving Field References

35 Name Resolution: Fields & Methods %% Declarations Type ";" -> ClassBodyDecl {"Field" Type "(" ")" "{" Stmt* "" -> ClassBodyDecl {"Method" %% Uses "return" Expr ";" -> Stmt {"Return" -> Expr {"FieldRef" Expr "." -> Expr {"FieldAccess" Expr "." "(" ")" -> Expr {"MethodCall" "(" ")" -> Expr {"ThisMethodCall"

36 Scope of Declarations "namespace" "{" Definition* "" -> Definition {"Namespace", scope(type, PartialType, Namespace) "class" "{" ClassBodyDecl* "" -> Definition {"Class", scope(field, Method)

37 "partial" "class" "{" ClassBodyDecl* "" -> Definition {"PartialClass", scope(field,method)

38 Type Analysis

39 Resolving Field Access test Resolve property access [[ namespace TestsResolving { class B { C v; C f() { return v.[[c]]; class C { C [[c]]; ]] resolve #1 to #2 interaction between type analysis and name resolution

40 Type Projection type-of: Field(x, _) -> <type-of> x type-of: PartialClass(x, _) -> TYPE(x) type-of: Class(x, _) -> TYPE(x) type-of: Method(x, _, _) -> <type-of> x type-of: Type(x) -> TYPE(x) type-of: Return(e) -> <type-of> e type-of: FieldRef(f) -> <type-of> <index-lookup> f type-of: FieldAccess(e, f) -> <type-of> <index-lookup> f type-of: MethodCall(e, m) -> <type-of> <index-lookup> m type-of: ThisMethodCall(m) -> <type-of> <index-lookup> m type-of: Def(uri) -> <index-get-value> Type(uri, ()) Stratego Rewrite Rules

41 Types of Definitions adjust-index-def-data(store-results namespace, path): def -> <store-results> Type([namespace path], type) where type := <type-of> def // Return a list of fields in the class on the left hand side. adjust-index-lookup(target namespace, path, prefix): FieldAccess(e, <target>) -> f* with if TYPE(e-type{_) := <type-of> e then f* := <index-lookup-children( Field(), prefix)> e-type else f* := StopLookup() end Adapt Generic Resolution Algorithm

42 Type Constraints

43 Reporting Errors

44 Testing Error Constraints test Resolve function call fails [[ namespace Tests { class B { P p; string f() { return p.geta(); ]] 1 error

45 Constraint Error Rules constraint-error: rhs) -> (e, $[Expected [<print-type> type1] but found [<print-type> type2].]) where type1 := <type-of> lhs; type2 := <type-of> rhs; not(<is-assignable-to> (type1, type2))

46 Unresolved References constraint-error: x{[unresolved(t) _] -> (x, $[Unable to resolve.])

47 Duplicate Definitions constraint-error: def -> (def, $[Duplicate definition]) where key{[ns _] := <nam-get-definition-key> def; <is-unique-namespace> ns; defs := <index-lookup-all> key; <gt> (<length> defs, 1) is-unique-namespace =?Type() is-unique-namespace =?Field() is-unique-namespace =?Method()

48 Code Generation

49 Code Generation to-java: Property(name, type) -> $[private [typename] [name]; public [typename] get_[name] { return [name]; public void set_[name]([typename] [name]) { this.[name] = [name]; ] where typename := <to-java> type

50 More Declarative Language Definition

51 Verifying Language Designs

52 Web IDE

53 Cloud9IDE

54

55

56 spoofax.org

57 Declarative Syntax Definition lexical syntax [a-za-z\_\$][a-za-z0-9\_]* -> ID context-free syntax STRING -> LimitedSetExp {cons("string") NUMBER -> LimitedSetExp {cons("num") LimitedSetExp -> Exp QId -> LimitedExp {cons("var") "(" Exp ")" -> LimitedExp {cons("brackets") Exp BoolMethodId Exp -> Exp {cons("binmethodcall"), left Exp CompareMethodId Exp -> Exp {cons("binmethodcall"), left Exp TermOperatorMethodId Exp -> Exp {cons("binmethodcall"), left Exp OperatorMethodId Exp -> Exp {cons("binmethodcall"), left character-level grammars: integration of lexical and context-free syntax

58 An Interactive Language Workbench develop and use language in same environment

59 Debugging Ambiguities

60 Declarative Disambiguation context-free priorities Exp "." ID -> LimitedExp > Exp "." ID "(" {NamedExp ","* ")" -> LimitedExp > Exp TermOperatorMethodId Exp -> Exp > Exp OperatorMethodId Exp -> Exp > Exp CompareMethodId Exp -> Exp > Exp BoolMethodId Exp -> Exp > "!" Exp -> Exp > Exp "?" Exp ":" Exp -> Exp > LimitedExp Filter+ -> Exp

61 Parsing after Disambiguation

62 Natural and Flexible Error Recovery* permissive grammar + backtracking + region discovery [*OOPSLA & SLE 2009]

63 Syntactic Normalization and Desugaring rules normalize : FunctionNoReturnType(manno*, name, farg*, stat*) -> Function(manno*, name, farg*, SimpleType(QId("mobl", "void")), stat*) normalize : IfNoElse(e, block) -> If(e, block, Block([])) desugar : ForInferred(lvalue, e, elem*) -> For(lvalue, t, e, elem*) where GenericType(_, [t]) := <type-of> e

64 CPS Transform cps-lift-exprs : [Return(e) stats] -> <concat>[stats2, [Return(e2) <cps-lift-expressions> stats]] where not(<is-sync> e) with { Exp : stats2 := <cps-lift-expression> e ; e2 := <Exp>

65 Type Analysis eq-type-of : String(_) -> SimpleType(QId("mobl", "String")) eq-type-of : FieldAccess(e, x) -> t where Property(_, _, t, _) := <lookup-property> (<type-of> e, x)

66 Type Checking constraint-error : t@simpletype(_) -> (t, $[Type is not defined: [<pp-mobl-type> t]]) where not(<lookup-type> t) constraint-error : t@fieldaccess(e, x) -> (t, $[Property [x] not defined]) where <type-of> e where not(type-of) origin tracking

67 Semantic Editor Services editor-analyze: (ast, path, project-path) -> (ast2, errors, warnings, notes) with //... editor-complete-proposal : SimpleType(COMPLETION(_)) -> proposals where all-types := <get-all-types>; proposals := <map(type-name-to-proposal); flatten-list> all-types editor-hover: (t@simpletype(_), position, ast, path, project-path) -> <get-doc> <lookup-type> t2 where t2 := <lookup-node> (position, ast) editor-resolve: (t@simpletype(qid), position, ast, path, project-path) -> target where target := <ensure-origin(lookup-type qid)> t

68 Editor Services Bindings module MoBL-Builders imports MoBL-Builders.generated builders provider : include/mobl.jar observer : editor-analyze on save : generate-artifacts builder : "Show ATerm" = generate-aterm (openeditor) (realtime) (meta) builder : "Format code" = format-code (openeditor) (realtime) builder : "Desugar" = editor-desugar (openeditor) (realtime) (meta) Connect transformations to IDE

69 behind the scenes

70 synchronous programming var results = Task.all().list(); for(var i = 0; i < results.length; i++) { alert(results[i].name);

71 render page query database and process results time...

72 render page browser freeze query database and process results time...

73 render page send query... time process query result...

74 asynchronous programming Task.all.list(function(results) { for(var i = 0; i < results.length; i++) { alert(results[i].name); );...

75 Task.all().list(function(results) { alert("hello, "); ); alert("world!"); breaks sequential execution assumption

76 Task.all().list(function(results) { // make changes... persistence.flush(function() { alert("changes saved!"); ); );

77 continuation-passing style transformation

78 function displaylocationandreturn() : Coordinates { var position = mobl::location::getposition(); log("lat: " + position.latitude); log("long: " + position.longitude); return position; function displaylocationandreturn(callback) { mobl.location.getposition(function(position) { console.log("lat: " + position.latitude); console.log("long: " + position.longitude); callback(position); ); ;

79 function displaylocationandreturn() : Coordinates { var position = mobl::location::getposition(); log("lat: " + position.latitude); log("long: " + position.longitude); return position; function displaylocationandreturn(callback) { mobl.location.getposition(function(position) { console.log("lat: " + position.latitude); console.log("long: " + position.longitude); callback(position); ); ;

80 function displaylocationandreturn() : Coordinates { var position = mobl::location::getposition(); log("lat: " + position.latitude); log("long: " + position.longitude); return position; function displaylocationandreturn(callback) { mobl.location.getposition(function(position) { console.log("lat: " + position.latitude); console.log("long: " + position.longitude); callback(position); ); ;