It s Turtles All the Way Up: Self-Extensible Programming Languages in PLT Scheme

Size: px

Start display at page:

Download "It s Turtles All the Way Up: Self-Extensible Programming Languages in PLT Scheme"

Arthur Grant
6 years ago
Views:

1 It s Turtles All the Way Up: Self-Extensible Programming Languages in PLT Scheme Matthew Flatt University of Utah 1

2 Different levels of extension... Syntactic abstraction (define-syntax-rule (define-thing id [cmd response...]...)...) New language constructs (class object% (define/public method...)...) New languages (: factorial (Number -> int f(int n) { return n+1; }... but all the same kind of turtle 2

3 Examples Lisp-Style Macros Scheme Lisp PLT Scheme Scheme Macros and Lexical Scope PLT Scheme Macros References and Related Work 3

4 Preprocessors vs. Macros import java.io.* house =... game.maze mazec javac 4

5 Preprocessors vs. Macros import java.io.* house.* javac game.java house =... house.maze mazec javac 15

6 Preprocessors vs. Macros import java.io.* house.* javac game.java house =... house.maze mazec javac maze.java java 26

7 Preprocessors vs. Macros (require "io.ss") (define-syntax define-place...) (define-place house...) game.ss schemec 27

8 Preprocessors vs. Macros (require "io.ss" "maze.ss") (define-place house...) game.ss schemec (define-syntax define-place...) maze.ss schemec 38

9 Preprocessors vs. Macros (require "io.ss" "house.ss") game.ss schemec (require "maze.ss") (define-place house...) house.ss schemec (define-syntax define-place...) maze.ss schemec 49

10 Parsing Source (... (...)...) S-expression function args body AST 50

11 Parsing #lang path Source (... (...)...) S-expression function args body AST 51

12 Parsing #lang path (module name path...) Source (... (...)...) S-expression function args body AST 52

13 Parsing #lang path (module name path...) Source (... (...)...) S-expression function args body AST (require path) 53

14 Parsing #lang path (module name path...) Source (... (...)...) S-expression function args body AST #reader path (require path) 54

15 Parsing #lang scheme (define (hi) "Hello") (module m scheme (define (hi) "Hello")) define hi function () "Hello" #lang (module m doclang (require scribble/manual) (bold "Hi")) import scribble/doc scribble/manual export doc define doc apply bold ("Hi") import apply #lang honu 1+2; (module m honu ; ) honu-procs print apply + (1 2) 55

16 Parsing Source (... (...)...) S-expression function args body AST 56

17 Parsing Source (... (...)...) S-expression function args body AST Read layer provides absolute control (

18 Parsing Source (... (...)...) S-expression function args body AST Expand layer can delay inside until after outside (define-place start... ([north house-front] [south desert])) (define-place house-front... ([in room] [south start])) int is_odd(int x) {... is_even(x-1); } int is_even(int x) {... is_odd(x-1); } 58

19 Expand-time Expressions (define-syntax three (lambda (stx) #'3)) (+ 1 (three)) 59

20 Expand-time Expressions (define-syntax three (lambda (stx) #'3)) (+ 1 (three)) 60

21 Expand-time Expressions (define-syntax three (lambda (stx) #'3)) (+ 1 (three)) 61

22 Lisp/Scheme: PL/I Template Haskell: Expand-time Expressions (define-syntax three (lambda (stx) #'3)) (+ 1 (three)) %Three : procedure ANS( 3 ); %end 1 + Three three s = [ 3 ] $(three) 62

23 Expand-time Expressions (require (for-syntax "roman-numerals.ss")) (define-syntax three (lambda (stx) #`(+ 1 #,(roman->number "II")))) (+ 1 (three)) 63

24 Expand-time Expressions (require (for-syntax "roman-numerals.ss")) (define-syntax three (lambda (stx) #`(+ 1 #,(roman->number "II")))) (+ 1 (three)) 64

25 Expand-time Expressions Demo: Check Syntax ( require (for-syntax "roman-numerals.ss")) (define-syntax three (lambda (stx) #`(+ 1 #,(roman->number "II")))) (+ 1 (three)) 65

26 Examples Lisp-Style Macros Scheme Macros and Lexical Scope Source (... (...)...) S-expression function args body AST PLT Scheme Macros References and Related Work 66

27 Pattern-Based Macros (define-syntax swap ) define-syntax indicates a macro definition 67

28 Pattern-Based Macros (define-syntax swap (syntax-rules () )) syntax-rules means a pattern-matching macro () means no keywords in the patterns 68

29 Pattern-Based Macros (define-syntax swap (syntax-rules () [pattern template]... [pattern template])) Any number of patterns to match Produce result from template of first match 69

30 Pattern-Based Macros (define-syntax swap (syntax-rules () ((swap a b) ))) Just one pattern for this macro: (swap a b) Each identifier matches anything in use (swap x y) a is x b is y (swap 9 (+ 1 7)) a is 9 b is (+ 1 7) 70

31 Pattern-Based Macros (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) Bindings substituted into template to generate the result (swap x y) (let ([tmp y]) (set! y x) (set! x tmp)) (swap 9 (+ 1 7)) (let ([tmp (+ 1 7)]) (set! (+ 1 7) 9) (set! 9 tmp)) 71

32 Lexical Scope (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) What if we swap a variable named tmp? (let ([tmp 5] [other 6]) (swap tmp other))? (let ([tmp 5] [other 6]) (let ([tmp other]) (set! other tmp) (set! tmp tmp))) 72

33 Lexical Scope (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) What if we swap a variable named tmp? (let ([tmp 5] [other 6]) (swap tmp other))? (let ([tmp 5] [other 6]) (let ([tmp other]) (set! other tmp) (set! tmp tmp))) This expansion would violate lexical scope 73

34 Lexical Scope (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) What if we swap a variable named tmp? (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 5] [other 6]) (let ([tmp 1 other]) (set! other tmp) (set! tmp tmp 1 ))) Rename the introduced binding 74

35 Lexical Scope: Local Bindings Lexical scope means that local macros work, too: (define (f x) (define-syntax swap-with-arg (syntax-rules () ((swap-with-arg y) (swap x y)))) (let ((z 12) (x 10)) ; Swaps z with original x: (swap-with-arg z)) ) 75

36 Transformer Definitions In general, define-syntax binds a transformer procedure: (define-syntax swap (syntax-rules...)) (define-syntax swap (lambda (stx) use syntax-object primitives to match stx and generate result )) 76

37 Matching Syntax and Having It, Too The syntax-case and #' forms combine patterns and arbitrary computation (syntax-case stx-expr () (pattern result-expr)... (pattern result-expr)) #'template 77

38 Matching Syntax and Having It, Too (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) (define-syntax swap (lambda (stx) (syntax-case stx () ((swap 1 a b) #'(let ((tmp b)) (set! b a) (set! a tmp)))))) 78

39 Matching Syntax and Having It, Too Check for identifiers before expanding: (define-syntax swap (lambda (stx) (syntax-case stx () ((swap a b) (if (and (identifier? #'a) (identifier? #'b)) #'(let ((tmp b)) (set! b a) (set! a tmp)) (raise-syntax-error 'swap "needs identifiers" stx)))))) 79

40 How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) Seems obvious that tmp can be renamed... 80

41 How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let-one (tmp b) (set! b a) (set! a tmp)) ))) 81

42 Can rename tmp: How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let-one (tmp b) (set! b a) (set! a tmp)) ))) (define-syntax let-one (syntax-rules () ((let-one (x v) body) (let ((x v)) body)))) 82

43 Cannot rename tmp: How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let-one (tmp b) (set! b a) (set! a tmp)) ))) (define-syntax let-one (syntax-rules () ((let-one (x v) body) (list 'x v body)))) 83

44 Cannot rename tmp: How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let-one (tmp b) (set! b a) (set! a tmp)) ))) (define-syntax let-one (syntax-rules () ((let-one (x v) body) (list 'x v body)))) Track identifier introductions, then rename only as binding forms are discovered 84

45 How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) Tracking avoids capture by introduced variables (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 5] [other 6]) (let 1 ([tmp 1 other]) (set! 1 other tmp) (set! 1 tmp tmp 1 ))) 1 means introduced by expansion tmp 1 does not capture tmp 85

46 How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) Tracking also avoids capture of introduced variables (let ([set! 5] [let 6]) (swap set! let)) (let ([set! 5] [let 6]) (let 1 ([tmp 1 let]) (set! 1 let set!) (set! 1 set! tmp 1 ))) set! does not capture set! 1 let does not capture let 1 86

47 How Lexical Scope Works (More Precisely) (let ([tmp 5] [other 6]) (swap tmp other)) 87

48 How Lexical Scope Works (More Precisely) (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 0 5] [other 0 6]) (swap tmp 0 other 0 )) To parse let, rename bindings by adding a subscript 88

49 How Lexical Scope Works (More Precisely) (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 0 5] [other 0 6]) (swap tmp 0 other 0 )) To parse let, rename bindings by adding a subscript To parse quote, drop the subscript (let ([x 1]) 'x) (let ([x 1 1]) 'x 1 ) (let ([x 1 1]) 'x) 89

50 How Lexical Scope Works (More Precisely) (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 0 5] [other 0 6]) (swap tmp 0 other 0 )) (let ([tmp 0 5] [other 0 6]) (let 1 ([tmp 1 other 0 ]) (set! 1 other 0 tmp 0 ) (set! 1 tmp 0 tmp 1 ))) Mark superscripts on introduced identifiers 90

51 How Lexical Scope Works (More Precisely) (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 0 5] [other 0 6]) (swap tmp 0 other 0 )) (let ([tmp 0 5] [other 0 6]) (let 1 ([tmp 1 other 0 ]) (set! 1 other 0 tmp 0 ) (set! 1 tmp 0 tmp 1 ))) (let ([tmp 0 5] [other 0 6]) (let 1 ([tmp 2 other 0 ]) (set! 1 other 0 tmp 0 ) (set! 1 tmp 0 tmp 2 ))) Rename for let but only where superscript marks match 91

52 How Lexical Scope Works (More Precisely) (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 0 5] [other 0 6]) (swap tmp 0 other 0 )) (let ([tmp 0 5] [other 0 6]) (let 1 ([tmp 1 other 0 ]) (set! 1 other 0 tmp 0 ) (set! 1 tmp 0 tmp 1 ))) (let ([tmp 0 5] [other 0 6]) (let 1 ([tmp 2 other 0 ]) (set! 1 other 0 tmp 0 ) (set! 1 tmp 0 tmp 2 ))) Introductions marked with fresh superscript Matching marks renamed with fresh subscript 92

53 How Lexical Scope Works (More Precisely) (let ([set! 5] [let 6]) (swap set! let)) 93

54 How Lexical Scope Works (More Precisely) (let ([set! 5] [let 6]) (swap set! let)) (let ([let 0 5] [set! 0 6]) (swap let 0 set! 0 )) 94

55 How Lexical Scope Works (More Precisely) (let ([set! 5] [let 6]) (swap set! let)) (let ([let 0 5] [set! 0 6]) (swap let 0 set! 0 )) (let ([let 0 5] [set 0 6]) (let 1 ([tmp 1 set! 0 ]) (set! 1 let 0 set! 0 ) (set! 1 let 0 tmp 1 ))) 95

56 How Lexical Scope Works (More Precisely) (let ([set! 5] [let 6]) (swap set! let)) (let ([let 0 5] [set! 0 6]) (swap let 0 set! 0 )) (let ([let 0 5] [set 0 6]) (let 1 ([tmp 1 set! 0 ]) (set! 1 let 0 set! 0 ) (set! 1 let 0 tmp 1 ))) (let ([set! 0 5] [let 0 6]) (let 1 ([tmp 2 let 0 ]) (set! 1 let 0 set! 0 ) (set! 1 set! 0 tmp 2 ))) Superscript mark is not a rename: let 1 refers to let 96

57 Representing Code #'lambda syntax-object datum srcloc context 'lambda file.ss:1:13 { mark 3, rename tmp 2 to tmp 1, lambda = kernel,... } 97

58 Representing Code #'(lambda (x) x) syntax-object datum srcloc context ( syntax-object datum srcloc context syntax-object datum srcloc context syntax-object datum srcloc context ) file.ss:1:12 { mark 3, rename tmp 2 to tmp 1, lambda = kernel,... } 98

59 Examples Lisp-Style Macros Scheme Macros and Lexical Scope PLT Scheme Macros Source (... (...)...) S-expression function args body AST References and Related Work 99

60 Representing an AST An AST is represented by a syntax object define doc apply bold ("Hi") = (define-values (doc) (#%app bold '"Hi")) 100

61 Representing an AST mod ::= (module id name-id (#%plain-module-begin mod-form...)) mod-form ::= expr (#%provide spec...) (define-values (id...) expr) (define-syntaxes (id...) expr) (define-values-for-syntax (id...) expr) (#%require raw-require-spec...) expr ::= id (#%plain-lambda formals expr...+) (case-lambda (formals expr...+)...) (if expr expr expr) (begin expr...+) (begin0 expr expr...) (let-values (((id...) expr)...) expr...+) (letrec-values (((id...) expr)...) expr...+) (set! id expr) (quote datum) (quote-syntax datum) (with-continuation-mark expr expr expr) (#%plain-app expr...+) (#%top. id) (#%variable-reference. global) formals ::= (id...) (id...+. id) id 101

62 (module m scheme (define x 3) (+ x 1)) Implicit Keywords Demo: Lazy Scheme, Beginner Scheme (+ 1 2) (#%app + 1 2) (#%app + (#%datum 1) (#%datum 2)) (#%app + '1 '2) (module m scheme (#%module-begin (define x 3) (+ x 1))) Demo: Scheme, Scribble 102

63 Explicit Expansion For tools that manipulate whole programs: expand : syntax -> syntax For macro transformers: local-expand : syntax... -> syntax... and extra arguments to say where to stop 103

64 Explicit Expansion (define-syntax-rule (thunk body) (lambda () body)) (class object% (define/public me (thunk this))...) Copy (define-syntax class (lambda (stx)... (local-expand #'method 'expression (list #'lambda...))...)) 104

65 Internal Definition Contexts Demo: define-package intdef #'defn-or-expr #'defn-or-expr #'defn-or-expr local-expand install definition #'(define id...) #'expr #'(define id...) #'expr #'(define id...) #'expr 105

66 Internal Definition Contexts (define-package p (y) (define x 10) (define y x)) syntax-object datum srcloc context 'x file.ss:1:8 { mark 3, { rename x to x 5,...}, rename tmp 2 to tmp 1, lambda = kernel,... } 106

67 Transformer Bindings (define-package Demo: provide a package p...) (open-package p)? (define-package p...) (define-syntax p (make-package...)) (open-package p)... #,(syntax-local-value #'p)

68 Transformer Bindings Even syntax pattern variables are implemented with macros (syntax-case stx () [(form id expr)... #'id...]) (let ([m (match-syntax stx...)]) (define-syntax form (make-pattern m 0)) (define-syntax id (make-pattern m 1)) (define-syntax expr (make-pattern m 2)) #,(syntax-local-value #'id)......) Primitives include quote-syntax, syntax-e, datum->syntax 108

69 Fixed Point New forms can have their own expanders (define-match-expander polar...) (match v [(polar mag theta)...]) New forms can have compile-time functions module and syntax-local-lift-expression set! and make-set!-transformer

70 Examples Lisp-Style Macros Scheme Macros and Lexical Scope PLT Scheme Macros References and Related Work 110

71 Lisp Macros origin Hart MACRO Definitions for LISP AI Project Memo 57, 1963 current Steele Common Lisp: The Language

72 Scheme Macros hygiene Kohlbecker, Friedman, Felleisen, and Duba Hygienic Macro Expansion LFP 1986 patterns Clinger and Rees Macros That Work POPL 1991 lexical scope Dybvig, Hieb, and Bruggeman Syntactic Abstraction in Scheme Lisp and Symbolic Computation 1993 splicing scope phases Waddell and Dybvig Extending the Scope of Syntactic Abstraction POPL 1999 Flatt Composable and Compilable Macros ICFP

73 Lisp/Scheme Macros without Parentheses lisp Shalit, Moon, and Starbuck The Dylan Reference Manual 1996 scheme Baker and Hsieh Maya: Multiple-dispatch Syntax Extension in Java PLDI 2002, Utah MS Thesis 113

74 Language Construction environments Reps and Teitelbaum The Synthesizer Generator 1988 more parsing Heering, Klint, et al. ASF+SDF Meta-Environment 1984 PEGs Ford Parsing Expression Grammars 2004 compiler Cox, Bergan, Clements, Kaashoek, and Kohler Xoc, An Extension-Oriented Compiler ASPLOS

75 Current Work at Utah Jon Rafkind: Honu Chongkai Zhu: ML and DrScheme Kevin Atkinson: C 115

76 Lisp A language should be extensible within itself, with the full language available for implementing extensions. Scheme Syntax objects for language extension should be lexically scoped, as opposed to mere source text. PLT Scheme A language should allow control over the starting point and access to the expander to enable entirely new languages, not just mere extensions. 116

Scheme-Style Macros: Patterns and Lexical Scope

Scheme-Style Macros: Patterns and Lexical Scope Matthew Flatt University of Utah 1 Why Macros? Language designers have to stop somewhere (544 pages) No language can provide every possible useful construct