Macros, and protocols, and metaprogramming - Oh My! (aka "What is Elixir and why should you care?") jim mccoy, Facebook Infosec Tools mccoy@fb.com (jim.mccoy@gmail.com)
Who? Currently build and maintain security tools for Facebook Build and maintain large-scale, distributed systems for myself and various companies Dabbler in Erlang for 10 or so years, Elixir enthusiast
What is Elixir? A functional, metaprogramming-aware language that runs on the Erlang VM Compiles to BEAM Supports transparent calling between code written in Elixir and in Erlang
What is Elixir? + Syntax is superficially similar to Ruby Lots of good ideas stolen from Clojure Erlang goodness underneath it all
Quick Intro: Data Types
Quick Intro: Data Types Mostly the same data types [lists,...] {tuples,...} :atoms <<binaries>> float and integer numbers pids, ports, refs
Quick Intro: Data Types A few new data types Range: 1..10 Regex: ~r/e[r]?l\w+/ A keyword list shortcut [name: "Bob", city: "London"] == [{:name, "Bob"}, {:city, "London"}]
Quick Intro: Data Types Strings 'foo' (single-quoted) are char lists "foo" (double-quoted) are strings (binaries) String module in stdlib very good unicode support #{varname} for string interpolation
Quick Intro: Syntax
Quick Intro: Syntax Immutable data, but not single assignment Single assignment within pattern match ^ before variable prevents rebinding Interactive Elixir (0.12.4) - press Ctrl+C to exit (type h() ENTER for help) iex(1)> a = 1 1 iex(2)> a = 2 2 iex(3)> ^a = 1 ** (MatchError) no match of right hand side value: 1 iex(3)> {a, b} = {1, 2} {1, 2} iex(4)> a 1 iex(5)> {a, b, a} = {1, 2, 3} ** (MatchError) no match of right hand side value: {1, 2, 3}
Quick Intro: Syntax Blocks do: (expressions...) combines expressions into a single block do... end is shorthand for do: () Parens are optional when calling functions foo(1, 3) == foo 1, 3
Quick Intro: Syntax Modules, Protocols & Records are BumpyCase but start with capital letter All others are lowercase All of erlang is available as :erlang.<somefunc> or :module.func
Quick Intro: Syntax Anonymous Functions Anonymous and named functions have distinct namespaces a_var = fn function expression end &(&1 + & 2) is anon func shorthand for adding two params &String.uppercase/1 is anon func shorthand for calling named function
Quick Intro: Syntax Anonymous functions Module.funcname() vs anon_func.() [the dot before the parens has meaning...] iex(1)> foo = &(&1 + & 2) &:erlang.+/2 iex(2)> foo.(4, 7) 11 iex(3)> bar = &(&1 <> "FizzBuzz") #Function<6.80484245/1 in :erl_eval.expr/5> iex(4)> bar.("15 is ") "15 is FizzBuzz"
Quick Intro: Syntax Modules All named functions live in modules def for public functions, defp for private
Quick Intro: Syntax Records Combine erlang "tuple with a name" and module to add additional functionality Decides at compile time if it will use standard Erlang tuple records or new Elixir records Automatic get, set & update functions
Elixir Benefits
Elixir Benefits Reduce boilerplate
Elixir Benefits Reduce boilerplate Improve productivity
Elixir Benefits Reduce boilerplate Improve productivity Macros!
Elixir Benefits Reduce boilerplate Improve productivity Macros! Protocols
Elixir Benefits Reduce boilerplate Improve productivity Macros! Protocols Sometimes faster that pure Erlang
Reduced Boilerplate
Reduced Boilerplate Eliminate needless text
Reduced Boilerplate Eliminate needless text -module(myserver). -behaviour(gen_server). -export([start_link/0]). -export([init/1, handle_call/3, handle_cast/2, handle_info/2,! terminate/2, code_change/3]). -define(server,?module). -record(state, {}). start_link() -> gen_server:start_link({local,?server},?module, [], []). init([]) - > {ok, #state{}}. handle_call(_request, _From, State) -> Reply = ok, {reply, Reply, State}. A gen_server template that does nothing handle_cast(_msg, State) -> {noreply, State}. handle_info(_info, State) -> {noreply, State}. terminate(_reason, _State) -> ok. code_change(_oldvsn, State, _Extra) -> {ok, State}.
Reduced Boilerplate Eliminate needless text defmodule MyServer do! use GenServer.Behavior! import GenX.GenServer! alias :gen_server, as: GenServer! def start_link do!! GenServer.start_link {:local, MODULE }, MODULE,!!! [], []! end! defrecord State, []! def init(_), do: {:ok, State.new} end The Elixir equivalent
Reduced Boilerplate Eliminate needless text defmodule MyServer do! use GenServer.Behavior! import GenX.GenServer! alias :gen_server, as: GenServer! def start_link do!! GenServer.start_link {:local, MODULE }, MODULE,!!! [], []! end! defrecord State, []! def init(_), do: {:ok, State.new} end defmodule Calculator do use ExActor defcast inc(x), state: value, do new_state(value + x) end defcast dec(x), state: value do new_state(value - x) end defcall get, state: value do value end end The Elixir equivalent Using a gen_server DSL to add handlers and api funcs automatically
Reduced Boilerplate Eliminate needless text Streamline data transforms with >
Reduced Boilerplate Eliminate needless text Streamline data transforms with > # Temp vars T1 = fold(mydata) T2 = spindle(t1) mutilate(t2) # Reversed nesting mutilate( spindle( fold(mydata) ) ) Erlang does not make data pipelines easy
Reduced Boilerplate Eliminate needless text Streamline data transforms with > mydata > fold > spindle > mutilate > takes output and sends it to next function as the first arg of that func
Reduced Boilerplate Eliminate needless text Streamline data transforms with > mydata > fold > spindle > mutilate {0, 1} > Stream.iterate(fn {a, b} -> {b, a + b} end) > Stream.map(elem &1, 0) > Enum.take(10) #=> [0, 1, 1, 2, 3, 5, 8, 13, 21, 34] > takes output and sends it to next function as the first arg of that func
Improve Productivity
Improve Productivity Mix EXPM and package management Elixir stdlib (Enum, Stream, HashDict, IO, File) Fixing things that are "broken"
Improve Productivity Mix Standard build and project tool Creates project templates and files Manages dependencies Runs unit tests
Improve Productivity Mix mix help mix # Run the default task (current: mix run) mix archive # Archive this project into a.ez file mix clean # Clean generated application files mix cmd # Executes the given command mix compile # Compile source files mix deps # List dependencies and their status mix deps.clean # Remove the given dependencies' files mix deps.compile # Compile dependencies mix deps.get # Get all out of date dependencies mix deps.unlock # Unlock the given dependencies mix deps.update # Update the given dependencies mix do # Executes the tasks separated by comma mix escriptize # Generates an escript for the project mix help # Print help information for tasks mix local # List local tasks mix local.install # Install a task or an archive locally mix local.rebar # Install rebar locally mix local.uninstall # Uninstall local tasks or archives mix new # Create a new Elixir project mix run # Run the given file or expression mix test # Run a project's tests iex -S mix # Start IEx and run the default task
Improve Productivity EXPM Centralized community package management Makes it easy to package libraries and use them in your projects Packages can be retrieved at runtime by modules Works for both Elixir and Erlang packages
Improve Productivity Elixir Standard Library Enum: makes working with collections very easy (and puts collection as first arg...) Stream: Lazy sequences and infinite streams IO, File: Elixir wrappers around standard Erlang i/o HashDict: very fast dictionaries
Improve Productivity Elixir Standard Library - Simplify APIs %% Erlang lists:map(fun A ->... end, List). dict:map(fun K, V ->... end, Dict). gb_trees:map(fun K, V ->... end, Tree). lists:map(fun A ->... end, sets:to_list(set)). ## Elixir Enum.map list, fn x ->... end Enum.map dict, fn {k, v} ->... end Enum.map set, fn x ->... end Enum.map 1..10, fn x ->... end Enum.map File.stream!("notes.txt"), fn x ->... end
Macros
Macros Compile-time code modification
Macros Compile-time code modification Homoiconic iex> contents = quote do...> defmodule HelloWorld do...> def hello_world do...> IO.puts "Hello world!"...> end...> end...> end {:defmodule,[context: Elixir],[{: aliases,[alias: false],[:helloworld]},[do: {:def, [context: Elixir],[{:hello_world,[],Elixir},[do: {\{:.,[],[{: aliases,[alias: false], [:IO]},:puts]},[],["Hello world!"]}]]}]]} iex> Code.eval_quoted contents {{:module,helloworld,<<70,79,82,49,0,0,7,104,66,69,65,77,65,116,111,109,0,0,0,132,0,0,0,13,17,69,108,105,120,105,114,46,72,101,108,108,111,87,111,114,108,100,8,95,95,105,110,102,11 1,95,...>>,{:hello_world,0}},[]} iex> HelloWorld.hello_world Hello world! :ok
Macros Compile-time code modification Homoiconic Elixir is written in Elixir macros defmacro unless(clause, options) do do_clause = Keyword.get(options, :do, nil)! else_clause = Keyword.get(options, :else, nil)! quote do!! if(unquote(clause), do: unquote(else_clause), else: unquote(do_clause))! end end
Macros Compile-time code modification Homoiconic Elixir is written in Elixir macros Enable you to extend Elixir with DSLs
Macros Compile-time code modification Homoiconic Elixir is written in Elixir macros Enable you to extend Elixir with DSLs A very sharp knife, with all of the good and band that this brings...
Macros quote - returns the internal representation of a block of code
Macros quote - returns the internal representation of a block of code iex(1)> quote do: 10 10 iex(2)> quote do: [:a, 45, 12.4] [:a, 45, 12.4] iex(3)> quote do: {1, 2, :z} {:{}, [], [1, 2, :z]} iex(4)> quote do: (2 + 3) {:+, [context: Elixir, import: Kernel], [2, 3]}
Macros unquote - injects code fragment iex(1)> foo = [1, 2, 3] [1, 2, 3] iex(2)> bar = quote do: [:a, :b, foo] [:a, :b, {:foo, [], Elixir}] iex(3)> baz = quote do: [:a, :b, unquote(foo)] [:a, :b, [1, 2, 3]]
Macros A simple example of the power of macros defmodule MimeTypes do HTTPotion.start HTTPotion.Response[body: body] = HTTPotion.get "http://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/conf/mime.types" Enum.each String.split(body, %r/\n/), fn (line) -> unless line == "" or line =~ %r/^#/ do [ mimetype _exts ] = String.split(line) def is_valid?(unquote(mimetype)), do: true end end def is_valid?(_mimetype), do: false end MimeTypes.is_valid?("application/vnd.exn") #=> false MimeTypes.is_valid?("application/json") #=> true
Macros quote - turn code into data unquote - turn data into code defmodule MimeTypes do HTTPotion.start HTTPotion.Response[body: body] = HTTPotion.get "http://svn.apache.org/repos/asf/httpd/httpd/trunk/docs/conf/mime.types" Enum.each String.split(body, %r/\n/), fn (line) -> unless line == "" or line =~ %r/^#/ do [ mimetype _exts ] = String.split(line) def is_valid?(unquote(mimetype)), do: true end end def is_valid?(_mimetype), do: false end MimeTypes.is_valid?("application/vnd.exn") #=> false MimeTypes.is_valid?("application/json") #=> true
Macros Example test framework defmodule Specs do defmacro describe(_, specs) do quote do unquote(specs) end end defmacro it(name, spec) do quote do def unquote(binary_to_atom("#{name} spec"))() do # Note: this is generating a function on the module unquote(spec) end end end defmacro should_eq(value1, value2) do quote do if unquote(value1)!= unquote(value2), do raise "#{unquote(value1)} did not equal #{unquote(value2)}!" end end end end
Macros Using the test framework defmodule SpecExample do import Specs describe "using macros" do it "passes as expected" do should_eq 1, 1 end it "fails as expected" do should_eq 1, 2 end end end apply(specexample, :"passes as expected spec", []) # => nil apply(specexample, :"fails as expected spec", []) # => (RuntimeError) 1 did not equal 2! This is essentially how ExUnit works...
Protocols
Protocols Adds polymorphic interfaces to Elixir data types
Protocols Adds polymorphic interfaces to Elixir data types Enable you to extend existing data types by adding behavior
Protocols Adds polymorphic interfaces to Elixir data types Enable you to extend existing data types by adding behavior Can define fallback implementation for data types that have not yet been defined or implemented (to extend someone else's data type or implement someone else's protocol)
Protocols Use defprotocol and defimpl keywords
Protocols Use defprotocol and defimpl keywords Apply a behavior to data, but outside of the module in which that data is defined
Sometimes Faster?
Sometimes Faster? HashDict is much faster than Erlang dicts
Sometimes Faster? HashDict is much faster than Erlang dicts...although possibly slower than maps
Sometimes Faster? HashDict is much faster than Erlang dicts...although possibly slower than maps Macros enable lazy evaluation when needed
Sometimes Faster? HashDict is much faster than Erlang dicts...although possibly slower than maps Macros enable lazy evaluation when needed Macros can provide compile time optimizations over standard Erlang
Why Else? Easier learning curve for Ruby/Python/Perl coders Easier to sneak into an org via DSLs It's fun!
More Info http://elixir-lang.org elixir-talk mailing list ElixirDose, Elixir Fountain, Elixir Sips. devintorr.es, theerlangist.com Books: Programming Elixir, Intro to Elixir, Elixir in Action
Thanks jim mccoy Facebook Infosec mccoy@fb.com/jim.mccoy@gmail.com