INF4820. Common Lisp: Closures and Macros

INF4820 Common Lisp: Closures and Macros Erik Velldal University of Oslo Oct. 19, 2010 Erik Velldal INF4820 1 / 22

Topics for Today More Common Lisp A quick reminder: Scope, binding and shadowing Closures Anonymous functions Functions that return functions Memoization Code that generates code: Macros Erik Velldal INF4820 2 / 22

Some Terminology (from Seibel 2005) Binding form A form introducing a variable such as a function definition or a let expression. Scope The area of the program where the variable name can be used to refer to the variable s binding. Lexically scoped variables can be referred to only by code that is textually within their binding form. Shadowing When binding forms are nested and introduce variables of the same name, the innermost binding shadows the outer bindings. Erik Velldal INF4820 3 / 22

A Reminder: Bindings, Scope and Shadowing (setq foo 24) (let ((foo 42) (bar foo)) (print bar)) 24 Erik Velldal INF4820 4 / 22

A Reminder: Bindings, Scope and Shadowing (setq foo 24) (let ((foo 42) (bar foo)) (print bar)) 24 (let* ((foo 42) (bar foo)) (print bar)) 42 Erik Velldal INF4820 4 / 22

A Reminder: Bindings, Scope and Shadowing (setq foo 24) (let ((foo 42) (bar foo)) (print bar)) 24 (let* ((foo 42) (bar foo)) (print bar)) 42 (let ((foo 42)) (let ((bar foo)) (print bar))) 42 Erik Velldal INF4820 4 / 22

Closures Local variables in Common Lisp are based on lexical scoping. But, in CL the concept of closures still makes possible the use of variable references in functions that are called in code outside the scope of the binding form that introduced the variables. Provides many of the advantages of global variables and OO, without the disadvantages. Confused yet? Some examples will help... Erik Velldal INF4820 5 / 22

First, a rather boring example (no closures here) A function count() using the special variable *c*. (defvar *c* 0) (defun count (action &optional (n 1)) (case action (:add (incf *c* n)) (:sub (decf *c* n)) (:print (format t "~&Current count: ~d.~%" *c*)))) (count :add 5) 5 (count :print) Current count: 5. *c* 5 Erik Velldal INF4820 6 / 22

An Example Using Closures A version of count() based on a closure over a free variable. (let ((c 0)) (defun count (action &optional (n 1)) (case action (:add (incf c n)) (:sub (decf c n)) (:print (format t "~&Current count: ~d.~%" c))))) (count :sub 11) -11 (count :add) -10 (count :print) Current count: -10. Erik Velldal INF4820 7 / 22

Combining Closures with Anonymous Functions A function that returns an anonymous function with it s own closure: (defun make-count () (let ((c 0)) # (lambda (action &optional (n 1)) (case action (:add (incf c n)) (:sub (decf c n)) (:print (format t "~&Current count: ~d.~%" c)))))) (setq count1 (make-count)) (setq count2 (make-count)) (funcall count1 :add 5) 5 (funcall count2 :sub) -1 (funcall count1 :print) Current count: 5. (funcall count2 :print) Current count: -1. Erik Velldal INF4820 8 / 22

Closures + Anonymous Functions (Lambda expressions) When a function is defined in a non-null lexical environment, we say that it closes over and captures the bindings of its free variables. The power of closures is often best seen in combination with anonymous functions (as in make-count()). In fact, you ve probably already used closures in combination with anonymous functions in settings like: (let ((n 2)) (mapcar # (lambda(x) (expt x n)) (1 2 3))) where # (lambda(x) (expt x n)) closes over the free variable n. Erik Velldal INF4820 9 / 22

Memoization A general technique for speeding up (repeated) calls to a (computationally expensive) function. By caching the computed return values, subsequent calls can retrieve the value by look-up. memoize() below takes a function as an argument and returns a new anonymous memoized version of that function, with a closure that contains a hash-table for storing results. (defun memoize (fn) (let ((cache (make-hash-table :test # equal))) # (lambda (&rest args) (multiple-value-bind (val stored-p) (gethash args cache) (if stored-p val (setf (gethash args cache) (apply fn args))))))) Erik Velldal INF4820 10 / 22

Memoization (cont d) (defun fib (n) (if (or (zerop n) (= n 1)) 1 (+ (fib (- n 1)) (fib (- n 2))))) (setq mem-fib (memoize # fib)) Erik Velldal INF4820 11 / 22

Memoization (cont d) (defun fib (n) (if (or (zerop n) (= n 1)) 1 (+ (fib (- n 1)) (fib (- n 2))))) (setq mem-fib (memoize # fib)) (time (funcall mem-fib 30)) ; real time 27,248 msec ; space allocation: ; 63,594,037 cons cells, 6,721,623,680 other bytes 1346269 (time (funcall mem-fib 30)) ; real time 0 msec ; space allocation: ; 31 cons cells, 240 other bytes 1346269 Erik Velldal INF4820 11 / 22

Memoization (cont d) Our memoized Fibonacci function isn t taking full advantage of the memoization... Why? Erik Velldal INF4820 12 / 22

Memoization (cont d) Our memoized Fibonacci function isn t taking full advantage of the memoization... Why? The recursive calls within the function are still using the original non-memoized version! A quick and dirty fix: (setf (symbol-function fib) mem-fib) A good exercise: Write an improved version of memoize() that does this automatically! (PS: It should also save a copy of the original function object so we can unmemoize later.) Other useful improvements we could make: Specialized data structures and/or equality tests depending on argument type. Canonical sorting of arguments. Support for multiple return values, keyword arguments, etc. Erik Velldal INF4820 12 / 22

Macros Pitch: programs that generate programs. Macro expansion time vs runtime Allows us to control or prevent the evaluation of arguments. Erik Velldal INF4820 13 / 22

Macros Pitch: programs that generate programs. Macro expansion time vs runtime Allows us to control or prevent the evaluation of arguments. Some examples of important Common Lisp built-in macros come in the form of control structures: Conditioning: and, or, if, when, unless, cond, case,... Looping / iteration: do, do*, dolist, dotimes, loop,... Also definitions and assignment: defun, defvar, defparameter, setf,... Erik Velldal INF4820 13 / 22

Defining Our Own Macros With defmacro we can write Lisp code that generates Lisp code. Important operators when writing macros : Quote suppresses evaluation. : Backquote also suppresses evaluation, but..,: A comma inside a backquoted form means the following subform should be evaluated.,@: Comma-at splices lists. Erik Velldal INF4820 14 / 22

Example: A Rather Silly Macro CL-USER(53): (defmacro dolist-reverse ((e list) &rest body) (let ((r (reverse,list))) (dolist (,e r),@body))) DOLIST-REVERSE Erik Velldal INF4820 15 / 22

Example: A Rather Silly Macro CL-USER(53): (defmacro dolist-reverse ((e list) &rest body) (let ((r (reverse,list))) (dolist (,e r),@body))) DOLIST-REVERSE CL-USER(54): (dolist-reverse (x (list 1 2 3)) (print x)) 3 2 1 NIL Erik Velldal INF4820 15 / 22

Unintended variable capture can be a pitfall... CL-USER(55): (let ((r 42)) (dolist-reverse (x (list 1 2 3)) (print (list x r)))) (3 (3 2 1)) (2 (3 2 1)) (1 (3 2 1)) NIL Not quite what we wanted... The variable r of the enclosing let-form is shadowed by the let in the macro-expansion! Remedy: Use gensym() at macro expansion time to create unique variable names used in the expansion. Erik Velldal INF4820 16 / 22

gensym to the rescue! Below, r is a variable whose value is the (unique) name of another variable. CL-USER(56): (defmacro dolist-reverse ((e list) &rest body) (let ((r (gensym))) (let ((,r (reverse,list))) (dolist (,e,r),@body)))) DOLIST-REVERSE CL-USER(57): (let ((r 42)) (dolist-reverse (x (list 1 2 3)) (print (list x r)))) (3 42) (2 42) (1 42) Erik Velldal INF4820 17 / 22

Other common pitfalls when writing macros Unintentionally evaluating arguments multiple times. Evaluating arguments in an order unexpected by the caller (i.e. not left to right). A useful debugging tool: macroexpand-1(). Erik Velldal INF4820 18 / 22

Macro-expansion time vs runtime What happens when we use a macro? (1) First the expression specified by the macro definition is built, the so-called macro-expansion... (2)... and then that expression is evaluated in place of the original macro call. Step (1) happens when Lisp parses the program prior to compiling, and step (2) when the code is called at runtime. Two levels of code: Expander code (the code used in the macro to generate its expansion) Expansion code (the code in the expansion itself) Erik Velldal INF4820 19 / 22

A more useful example; dolist-permutations (defmacro dolist-permutations (lists &body body) (if (null (cdr lists)) (dolist,(first lists),@body) (dolist,(first lists) (dolist-permutations,(cdr lists),@body)))) The macro includes calls to other macros, including itself. Using self-reference / recursion in macros requires care to not send the compiler into an infinite loop! Recursion in expander code (as here) vs recursion in expansion code. While expanding dolist-permutations we recur over forms, not values (which are unavailable at expansion time). Erik Velldal INF4820 20 / 22

Using our new macro, dolist-permutations CL-USER(190): (dolist-permutations ((x (a b)) (y (1 2 3)) (z (foo))) (print (list x y z))) (A 1 FOO) (A 2 FOO) (A 3 FOO) (B 1 FOO) (B 2 FOO) (B 3 FOO) NIL Erik Velldal INF4820 21 / 22

For More on Macros... See Paul Graham s On Lisp: http://www.paulgraham.com/onlisp.html (Out of print but freely available online.) Includes several chapters on writing Lisp macros. Erik Velldal INF4820 22 / 22