It s Turtles All the Way Up: Self-Extensible Programming Languages in PLT Scheme

It s Turtles All the Way Up: Self-Extensible Programming Languages in PLT Scheme Matthew Flatt University of Utah 1

Different levels of extension... Syntactic abstraction (define-syntax-rule (define-thing id [cmd response...]...)...) New language constructs (class object% (define/public method...)...) New languages (: factorial (Number -> Number)) @section{hello} int f(int n) { return n+1; }... but all the same kind of turtle 2

Examples Lisp-Style Macros Scheme Lisp PLT Scheme Scheme Macros and Lexical Scope PLT Scheme Macros References and Related Work 3

Preprocessors vs. Macros import java.io.* house =... game.maze mazec javac 4

Preprocessors vs. Macros import java.io.* house.* javac game.java house =... house.maze mazec javac 15

Preprocessors vs. Macros import java.io.* house.* javac game.java house =... house.maze mazec javac maze.java java 26

Preprocessors vs. Macros (require "io.ss") (define-syntax define-place...) (define-place house...) game.ss schemec 27

Preprocessors vs. Macros (require "io.ss" "maze.ss") (define-place house...) game.ss schemec (define-syntax define-place...) maze.ss schemec 38

Preprocessors vs. Macros (require "io.ss" "house.ss") game.ss schemec (require "maze.ss") (define-place house...) house.ss schemec (define-syntax define-place...) maze.ss schemec 49

Parsing Source (... (...)...) S-expression function args body AST 50

Parsing #lang path Source (... (...)...) S-expression function args body AST 51

Parsing #lang path (module name path...) Source (... (...)...) S-expression function args body AST 52

Parsing #lang path (module name path...) Source (... (...)...) S-expression function args body AST (require path) 53

Parsing #lang path (module name path...) Source (... (...)...) S-expression function args body AST #reader path (require path) 54

Parsing #lang scheme (define (hi) "Hello") (module m scheme (define (hi) "Hello")) define hi function () "Hello" #lang scribble/doc @(require scribble/manual) @bold{hi} (module m doclang (require scribble/manual) (bold "Hi")) import scribble/doc scribble/manual export doc define doc apply bold ("Hi") import apply #lang honu 1+2; (module m honu 1 + 2 ; ) honu-procs print apply + (1 2) 55

Parsing Source (... (...)...) S-expression function args body AST 56

Parsing Source (... (...)...) S-expression function args body AST Read layer provides absolute control (+ 1 2) @bold{hi} 1+2 57

Parsing Source (... (...)...) S-expression function args body AST Expand layer can delay inside until after outside (define-place start... ([north house-front] [south desert])) (define-place house-front... ([in room] [south start])) int is_odd(int x) {... is_even(x-1); } int is_even(int x) {... is_odd(x-1); } 58

Expand-time Expressions (define-syntax three (lambda (stx) #'3)) (+ 1 (three)) 59

Expand-time Expressions (define-syntax three (lambda (stx) #'3)) (+ 1 (three)) 60

Expand-time Expressions (define-syntax three (lambda (stx) #'3)) (+ 1 (three)) 61

Lisp/Scheme: PL/I Template Haskell: Expand-time Expressions (define-syntax three (lambda (stx) #'3)) (+ 1 (three)) %Three : procedure ANS( 3 ); %end 1 + Three three s = [ 3 ] $(three) 62

Expand-time Expressions (require (for-syntax "roman-numerals.ss")) (define-syntax three (lambda (stx) #`(+ 1 #,(roman->number "II")))) (+ 1 (three)) 63

Expand-time Expressions (require (for-syntax "roman-numerals.ss")) (define-syntax three (lambda (stx) #`(+ 1 #,(roman->number "II")))) (+ 1 (three)) 64

Expand-time Expressions Demo: Check Syntax ( require (for-syntax "roman-numerals.ss")) (define-syntax three (lambda (stx) #`(+ 1 #,(roman->number "II")))) (+ 1 (three)) 65

Examples Lisp-Style Macros Scheme Macros and Lexical Scope Source (... (...)...) S-expression function args body AST PLT Scheme Macros References and Related Work 66

Pattern-Based Macros (define-syntax swap ) define-syntax indicates a macro definition 67

Pattern-Based Macros (define-syntax swap (syntax-rules () )) syntax-rules means a pattern-matching macro () means no keywords in the patterns 68

Pattern-Based Macros (define-syntax swap (syntax-rules () [pattern template]... [pattern template])) Any number of patterns to match Produce result from template of first match 69

Pattern-Based Macros (define-syntax swap (syntax-rules () ((swap a b) ))) Just one pattern for this macro: (swap a b) Each identifier matches anything in use (swap x y) a is x b is y (swap 9 (+ 1 7)) a is 9 b is (+ 1 7) 70

Pattern-Based Macros (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) Bindings substituted into template to generate the result (swap x y) (let ([tmp y]) (set! y x) (set! x tmp)) (swap 9 (+ 1 7)) (let ([tmp (+ 1 7)]) (set! (+ 1 7) 9) (set! 9 tmp)) 71

Lexical Scope (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) What if we swap a variable named tmp? (let ([tmp 5] [other 6]) (swap tmp other))? (let ([tmp 5] [other 6]) (let ([tmp other]) (set! other tmp) (set! tmp tmp))) 72

Lexical Scope (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) What if we swap a variable named tmp? (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 5] [other 6]) (let ([tmp 1 other]) (set! other tmp) (set! tmp tmp 1 ))) Rename the introduced binding 74

Lexical Scope: Local Bindings Lexical scope means that local macros work, too: (define (f x) (define-syntax swap-with-arg (syntax-rules () ((swap-with-arg y) (swap x y)))) (let ((z 12) (x 10)) ; Swaps z with original x: (swap-with-arg z)) ) 75

Transformer Definitions In general, define-syntax binds a transformer procedure: (define-syntax swap (syntax-rules...)) (define-syntax swap (lambda (stx) use syntax-object primitives to match stx and generate result )) 76

Matching Syntax and Having It, Too The syntax-case and #' forms combine patterns and arbitrary computation (syntax-case stx-expr () (pattern result-expr)... (pattern result-expr)) #'template 77

Matching Syntax and Having It, Too (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) (define-syntax swap (lambda (stx) (syntax-case stx () ((swap 1 a b) #'(let ((tmp b)) (set! b a) (set! a tmp)))))) 78

Matching Syntax and Having It, Too Check for identifiers before expanding: (define-syntax swap (lambda (stx) (syntax-case stx () ((swap a b) (if (and (identifier? #'a) (identifier? #'b)) #'(let ((tmp b)) (set! b a) (set! a tmp)) (raise-syntax-error 'swap "needs identifiers" stx)))))) 79

How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) Seems obvious that tmp can be renamed... 80

How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let-one (tmp b) (set! b a) (set! a tmp)) ))) 81

Can rename tmp: How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let-one (tmp b) (set! b a) (set! a tmp)) ))) (define-syntax let-one (syntax-rules () ((let-one (x v) body) (let ((x v)) body)))) 82

Cannot rename tmp: How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let-one (tmp b) (set! b a) (set! a tmp)) ))) (define-syntax let-one (syntax-rules () ((let-one (x v) body) (list 'x v body)))) 83

How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) Tracking avoids capture by introduced variables (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 5] [other 6]) (let 1 ([tmp 1 other]) (set! 1 other tmp) (set! 1 tmp tmp 1 ))) 1 means introduced by expansion tmp 1 does not capture tmp 85

How Lexical Scope Works (define-syntax swap (syntax-rules () ((swap a b) (let ((tmp b)) (set! b a) (set! a tmp))))) Tracking also avoids capture of introduced variables (let ([set! 5] [let 6]) (swap set! let)) (let ([set! 5] [let 6]) (let 1 ([tmp 1 let]) (set! 1 let set!) (set! 1 set! tmp 1 ))) set! does not capture set! 1 let does not capture let 1 86

How Lexical Scope Works (More Precisely) (let ([tmp 5] [other 6]) (swap tmp other)) 87

How Lexical Scope Works (More Precisely) (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 0 5] [other 0 6]) (swap tmp 0 other 0 )) To parse let, rename bindings by adding a subscript 88

How Lexical Scope Works (More Precisely) (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 0 5] [other 0 6]) (swap tmp 0 other 0 )) To parse let, rename bindings by adding a subscript To parse quote, drop the subscript (let ([x 1]) 'x) (let ([x 1 1]) 'x 1 ) (let ([x 1 1]) 'x) 89

How Lexical Scope Works (More Precisely) (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 0 5] [other 0 6]) (swap tmp 0 other 0 )) (let ([tmp 0 5] [other 0 6]) (let 1 ([tmp 1 other 0 ]) (set! 1 other 0 tmp 0 ) (set! 1 tmp 0 tmp 1 ))) (let ([tmp 0 5] [other 0 6]) (let 1 ([tmp 2 other 0 ]) (set! 1 other 0 tmp 0 ) (set! 1 tmp 0 tmp 2 ))) Rename for let but only where superscript marks match 91

How Lexical Scope Works (More Precisely) (let ([tmp 5] [other 6]) (swap tmp other)) (let ([tmp 0 5] [other 0 6]) (swap tmp 0 other 0 )) (let ([tmp 0 5] [other 0 6]) (let 1 ([tmp 1 other 0 ]) (set! 1 other 0 tmp 0 ) (set! 1 tmp 0 tmp 1 ))) (let ([tmp 0 5] [other 0 6]) (let 1 ([tmp 2 other 0 ]) (set! 1 other 0 tmp 0 ) (set! 1 tmp 0 tmp 2 ))) Introductions marked with fresh superscript Matching marks renamed with fresh subscript 92

How Lexical Scope Works (More Precisely) (let ([set! 5] [let 6]) (swap set! let)) 93

How Lexical Scope Works (More Precisely) (let ([set! 5] [let 6]) (swap set! let)) (let ([let 0 5] [set! 0 6]) (swap let 0 set! 0 )) 94

How Lexical Scope Works (More Precisely) (let ([set! 5] [let 6]) (swap set! let)) (let ([let 0 5] [set! 0 6]) (swap let 0 set! 0 )) (let ([let 0 5] [set 0 6]) (let 1 ([tmp 1 set! 0 ]) (set! 1 let 0 set! 0 ) (set! 1 let 0 tmp 1 ))) (let ([set! 0 5] [let 0 6]) (let 1 ([tmp 2 let 0 ]) (set! 1 let 0 set! 0 ) (set! 1 set! 0 tmp 2 ))) Superscript mark is not a rename: let 1 refers to let 96

Representing Code #'lambda syntax-object datum srcloc context 'lambda file.ss:1:13 { mark 3, rename tmp 2 to tmp 1, lambda = lambda @ kernel,... } 97

Representing Code #'(lambda (x) x) syntax-object datum srcloc context ( syntax-object datum srcloc context syntax-object datum srcloc context syntax-object datum srcloc context ) file.ss:1:12 { mark 3, rename tmp 2 to tmp 1, lambda = lambda @ kernel,... } 98

Examples Lisp-Style Macros Scheme Macros and Lexical Scope PLT Scheme Macros Source (... (...)...) S-expression function args body AST References and Related Work 99

Representing an AST An AST is represented by a syntax object define doc apply bold ("Hi") = (define-values (doc) (#%app bold '"Hi")) 100

Representing an AST mod ::= (module id name-id (#%plain-module-begin mod-form...)) mod-form ::= expr (#%provide spec...) (define-values (id...) expr) (define-syntaxes (id...) expr) (define-values-for-syntax (id...) expr) (#%require raw-require-spec...) expr ::= id (#%plain-lambda formals expr...+) (case-lambda (formals expr...+)...) (if expr expr expr) (begin expr...+) (begin0 expr expr...) (let-values (((id...) expr)...) expr...+) (letrec-values (((id...) expr)...) expr...+) (set! id expr) (quote datum) (quote-syntax datum) (with-continuation-mark expr expr expr) (#%plain-app expr...+) (#%top. id) (#%variable-reference. global) formals ::= (id...) (id...+. id) id 101

(module m scheme (define x 3) (+ x 1)) Implicit Keywords Demo: Lazy Scheme, Beginner Scheme (+ 1 2) (#%app + 1 2) (#%app + (#%datum 1) (#%datum 2)) (#%app + '1 '2) (module m scheme (#%module-begin (define x 3) (+ x 1))) Demo: Scheme, Scribble 102

Explicit Expansion For tools that manipulate whole programs: expand : syntax -> syntax For macro transformers: local-expand : syntax... -> syntax... and extra arguments to say where to stop 103

Explicit Expansion (define-syntax-rule (thunk body) (lambda () body)) (class object% (define/public me (thunk this))...) Copy (define-syntax class (lambda (stx)... (local-expand #'method 'expression (list #'lambda...))...)) 104

Internal Definition Contexts Demo: define-package intdef...... #'defn-or-expr #'defn-or-expr #'defn-or-expr local-expand install definition #'(define id...) #'expr #'(define id...) #'expr #'(define id...) #'expr 105

Internal Definition Contexts (define-package p (y) (define x 10) (define y x)) syntax-object datum srcloc context 'x file.ss:1:8 { mark 3, { rename x to x 5,...}, rename tmp 2 to tmp 1, lambda = lambda @ kernel,... } 106

Transformer Bindings (define-package Demo: provide a package p...) (open-package p)? (define-package p...) (define-syntax p (make-package...)) (open-package p)... #,(syntax-local-value #'p)... 107

Transformer Bindings Even syntax pattern variables are implemented with macros (syntax-case stx () [(form id expr)... #'id...]) (let ([m (match-syntax stx...)]) (define-syntax form (make-pattern m 0)) (define-syntax id (make-pattern m 1)) (define-syntax expr (make-pattern m 2))...... #,(syntax-local-value #'id)......) Primitives include quote-syntax, syntax-e, datum->syntax 108

Fixed Point New forms can have their own expanders (define-match-expander polar...) (match v [(polar mag theta)...]) New forms can have compile-time functions module and syntax-local-lift-expression set! and make-set!-transformer... 109

Examples Lisp-Style Macros Scheme Macros and Lexical Scope PLT Scheme Macros References and Related Work 110

Lisp Macros origin Hart MACRO Definitions for LISP AI Project Memo 57, 1963 current Steele Common Lisp: The Language 1990 111

Scheme Macros hygiene Kohlbecker, Friedman, Felleisen, and Duba Hygienic Macro Expansion LFP 1986 patterns Clinger and Rees Macros That Work POPL 1991 lexical scope Dybvig, Hieb, and Bruggeman Syntactic Abstraction in Scheme Lisp and Symbolic Computation 1993 splicing scope phases Waddell and Dybvig Extending the Scope of Syntactic Abstraction POPL 1999 Flatt Composable and Compilable Macros ICFP 2002 112

Lisp/Scheme Macros without Parentheses lisp Shalit, Moon, and Starbuck The Dylan Reference Manual 1996 scheme Baker and Hsieh Maya: Multiple-dispatch Syntax Extension in Java PLDI 2002, Utah MS Thesis 113

Language Construction environments Reps and Teitelbaum The Synthesizer Generator 1988 more parsing Heering, Klint, et al. ASF+SDF Meta-Environment 1984 PEGs Ford Parsing Expression Grammars 2004 compiler Cox, Bergan, Clements, Kaashoek, and Kohler Xoc, An Extension-Oriented Compiler ASPLOS 2008 114

Current Work at Utah Jon Rafkind: Honu Chongkai Zhu: ML and DrScheme Kevin Atkinson: C 115

Lisp A language should be extensible within itself, with the full language available for implementing extensions. Scheme Syntax objects for language extension should be lexically scoped, as opposed to mere source text. PLT Scheme A language should allow control over the starting point and access to the expander to enable entirely new languages, not just mere extensions. 116