iex(1)> defmodule RepeatN do def repeat_n(function, count) do ...(1)> end repeat_n(function, count - 1) {:module, RepeatN,...}

Transcription

1 The other day

2 iex(1)>

3 iex(1)> defmodule RepeatN do...(1)> def repeat_n(_function, 0) do...(1)> # noop...(1)>...(1)> def repeat_n(function, 1) do...(1)> function.()...(1)>...(1)> def repeat_n(function, count) do...(1)> function.()...(1)> repeat_n(function, count - 1)...(1)>...(1)> {:module, RepeatN,...}

4 iex(1)> defmodule RepeatN do...(1)> def repeat_n(_function, 0) do...(1)> # noop...(1)>...(1)> def repeat_n(function, 1) do...(1)> function.()...(1)>...(1)> def repeat_n(function, count) do...(1)> function.()...(1)> repeat_n(function, count - 1)...(1)>...(1)> {:module, RepeatN,...} iex(2)> :timer.tc fn -> RepeatN.repeat_n(fn -> 0, 100) {210, 0}

5 iex(1)> defmodule RepeatN do...(1)> def repeat_n(_function, 0) do...(1)> # noop...(1)>...(1)> def repeat_n(function, 1) do...(1)> function.()...(1)>...(1)> def repeat_n(function, count) do...(1)> function.()...(1)> repeat_n(function, count - 1)...(1)>...(1)> {:module, RepeatN,...} iex(2)> :timer.tc fn -> RepeatN.repeat_n(fn -> 0, 100) {210, 0} iex(3)> list = Enum.to_list(1..100) [...] iex(4)> :timer.tc fn -> Enum.each(list, fn(_) -> 0 ) {165, :ok}

6 iex(1)> defmodule RepeatN do...(1)> def repeat_n(_function, 0) do...(1)> # noop...(1)>...(1)> def repeat_n(function, 1) do...(1)> function.()...(1)>...(1)> def repeat_n(function, count) do...(1)> function.()...(1)> repeat_n(function, count - 1)...(1)>...(1)> {:module, RepeatN,...} iex(2)> :timer.tc fn -> RepeatN.repeat_n(fn -> 0 {210, 0} iex(3)> list = Enum.to_list(1..100) [...] iex(4)> :timer.tc fn -> Enum.each(list, fn(_) -> {165, :ok} iex(5)> :timer.tc fn -> Enum.each(list, fn(_) -> {170, :ok} iex(6)> :timer.tc fn -> Enum.each(list, fn(_) -> {184, :ok}, 100) 0 ) 0 ) 0 )

7 Success!

8 The End?

9

10 Way too few samples Realistic data/multiple inputs? No warmup Non production environment Does creating the list matter? Is repeating really the bottle neck? Repeatability? Setup information Running on battery Lots of applications running Numerous Failures!

11 n range list fun = = = = 10_ n Enum.to_list range fn -> 0 Benchee.run %{ "Enum.each" => fn -> Enum.each(list, fn(_) -> fun.() ), "List comprehension" => fn -> for _ <- list, do: fun.(), "Recursion" => fn -> RepeatN.repeat_n(fun, n) }

12 Name Recursion Enum.each List comprehension ips 6.95 K 4.21 K 3.07 K average μs μs μs Comparison: Recursion Enum.each List comprehension 6.95 K 4.21 K x slower 3.07 K x slower deviation ±2.76% ±6.47% ±10.02% median 142 μs 230 μs 333 μs 99th % 156 μs 278 μs 373 μs

13 Iterations per Second Name Recursion Enum.each List comprehension ips 6.95 K 4.21 K 3.07 K average μs μs μs Comparison: Recursion Enum.each List comprehension 6.95 K 4.21 K x slower 3.07 K x slower deviation ±2.76% ±6.47% ±10.02% median 142 μs 230 μs 333 μs 99th % 156 μs 278 μs 373 μs

14 Stop guessing and Start Measuring Benchmarking in Practice Tobias pragtob.info github.com/pragtob/benchee

15

16

17

18 Concept vs Tool Usage

19 What s Benchmarking?

20

21 Profiling vs Benchmarking

22 Flame Graph El.. Elixir.Enum:-map/2-lc$^0/1-0-/2 El.. Elixir.Enum:-map/2-lc$^0/1-0-/2 El.. Elixir.Enum:-map/2-lc$^0/1-0-/2 El.. Elixir.Enum:-map/2-lc$^0/1-0-/2 El.. Elixir.Enum:-map/2-lc$^0/1-0-/2 El.. El.. Elixir.Life.Board:map/2 El.. Elixir.Life.Board:map/2 El.. Elixir.Life.Board:map/2 El.. Elixir.Life.Board:map/2 El.. Elixir.Life.Board:map/2 El.. Eli.. Elixir.Life:run_loop/3 eflame:apply1/3 (0.47.0)

23 What to benchmark?

24 What to measure? Runtime? Memory? Throughput? Custom?

25 The famous post

26 What to measure? Runtime! Memory? Throughput? Custom?

27 How long will this take?

28 Enum.sort/1 performance

29 Enum.sort/1 performance

30 Did we make it faster?

31 What s fastest?

32 What's the fastest way to sort a list of numbers largest to smallest?

33 list = 1..10_000 > Enum.to_list > Enum.shuffle Benchee.run %{ "sort(fun)" => fn -> Enum.sort(list, &(&1 > &2)), "sort > reverse" => fn -> list > Enum.sort > Enum.reverse, "sort_by(-value)" => fn -> Enum.sort_by(list, fn(val) -> -val ) }

34 list = 1..10_000 > Enum.to_list > Enum.shuffle Benchee.run %{ "sort(fun)" => fn -> Enum.sort(list, &(&1 > &2)), "sort > reverse" => fn -> list > Enum.sort > Enum.reverse, "sort_by(-value)" => fn -> Enum.sort_by(list, fn(val) -> -val ) }

35 list = 1..10_000 > Enum.to_list > Enum.shuffle Benchee.run %{ "sort(fun)" => fn -> Enum.sort(list, &(&1 > &2)), "sort > reverse" => fn -> list > Enum.sort > Enum.reverse, "sort_by(-value)" => fn -> Enum.sort_by(list, fn(val) -> -val ) }

36 list = 1..10_000 > Enum.to_list > Enum.shuffle Benchee.run %{ "sort(fun)" => fn -> Enum.sort(list, &(&1 > &2)), "sort > reverse" => fn -> list > Enum.sort > Enum.reverse, "sort_by(-value)" => fn -> Enum.sort_by(list, fn(val) -> -val ) }

37 Name sort > reverse sort(fun) sort_by(-value) ips average 1.74 ms 4.15 ms 6.67 ms Comparison: sort > reverse sort(fun) sort_by(-value) x slower x slower deviation ±7.11% ±3.04% ±6.06% median 1.70 ms 4.15 ms 6.50 ms 99th % 2.25 ms 4.60 ms 7.83 ms

38 Isn t that the root of all evil?

39 More likely, not reading the sources is the source of all evil. Me, just now

40 Yup it s there We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Donald Knuth, 1974 (Computing Surveys, Vol 6, No 4, December 1974)

41 The very next sentence Yet we should not pass up our opportunities in that critical 3%. Donald Knuth, 1974 (Computing Surveys, Vol 6, No 4, December 1974)

42 Prior Paragraph (...) a 12 % improvement, easily obtained, is never considered marginal Donald Knuth, 1974 (Computing Surveys, Vol 6, No 4, December 1974)

43 Different types of benchmarks

44 Testing Pyramid Feature Integration Unit

45 Benchmarking Pyramid Application Macro Micro

46 Micro Macro Application

47 Micro Macro Components involved Application

48 Micro Macro Components involved Setup Complexity Application

49 Micro Macro Components involved Setup Complexity Execution Time Application

50 Micro Macro Components involved Setup Complexity Execution Time Confidence of Real Impact Application

51 Micro Macro Components involved Setup Complexity Execution Time Confidence of Real Impact Chance of Interference Application

52 Micro Golden Middle Macro Components involved Setup Complexity Execution Time Confidence of Real Impact Chance of Interference Application

53 Good Benchmarking

54 What are you benchmarking for?

55 System Specification Elixir Erlang 20.2 i5-7200u 2 x 2.5GHz (Up to 3.10GHz) 8GB RAM Linux Mint bit (Ubuntu base)

56 System Specification elixir_playground $ mix run bench/something.exs Operating System: Linux CPU Information: Intel(R) Core(TM) i5-7200u 2.50GHz Number of Available Cores: 4 Available memory: 7.67 GB Elixir Erlang 20.2 Benchmark suite executing with the following configuration: warmup: 2 s time: 5 s parallel: 1 inputs: none specified Estimated total run time: 21 s

57

58

59 1.4!

60 1.3

61 Correct & Meaningful Setup

62 Interference free Environment

63 [info] GET / [debug] Processing by Rumbl.PageController.index/2 Parameters: %{} Pipelines: [:browser] [info] Sent 200 in 46ms [info] GET /sessions/new [debug] Processing by Rumbl.SessionController.new/2 Parameters: %{} Pipelines: [:browser] [info] Sent 200 in 5ms [info] GET /users/new [debug] Processing by Rumbl.UserController.new/2 Parameters: %{} Pipelines: [:browser] [info] Sent 200 in 7ms [info] POST /users [debug] Processing by Rumbl.UserController.create/2 Parameters: %{"_csrf_token" => "NUEUdRMNAiBfIHEeNwZkfA05PgAOJgAAf0ACXJqCjl7YojW+trdjdg==", "_utf8" => " ", "user" => %{"name" => "asdasd", "password" => "[FILTERED]", "username" => "Homer"}} Pipelines: [:browser] [debug] QUERY OK db=0.1ms begin [] [debug] QUERY OK db=0.9ms INSERT INTO "users" ("name","password_hash","username","inserted_at","updated_at") VALUES ($1,$2,$3,$4,$5) RETURNING "id" ["asdasd", "$2b$12$.qY/kpo0Dec7vMK1ClJoC.Lw77c3oGllX7uieZILMlFh2hFpJ3F.C", "Homer", {{2016, 12, 2}, {14, 10, 28, 0}}, {{2016, 12, 2}, {14, 10, 28, 0}}] Logging & Fris

64 Garbage Collection

65

66 Warmup

67 Probably not what you want $ time my_program

68 What s important for you?

69 What s important for you? Startup

70 What s important for you? Startup Warmup

71 What s important for you? Startup Warmup Runtime

72 28s 2s

73 Inputs matter!

74 Story Time

75 Boom

76 Boom Elixir.DBConnection.ConnectionError

77

78 Benchee.run %{ "DB View" => fn -> LatestCourierLocation > CourierLocation.with_courier_ids(courier_id) > Repo.one(timeout: :infinity), "with_courier_ids" => fn -> CourierLocation.with_courier_ids(courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity), "full custom" => fn -> CourierLocation > Ecto.Query.where(courier_id: ^courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity) }

79 Benchee.run %{ "DB View" => fn -> Database LatestCourierLocation > CourierLocation.with_courier_ids(courier_id) > Repo.one(timeout: :infinity), "with_courier_ids" => fn -> CourierLocation.with_courier_ids(courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity), "full custom" => fn -> CourierLocation > Ecto.Query.where(courier_id: ^courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity) } View

80 Benchee.run %{ "DB View" => fn -> Only difference LatestCourierLocation > CourierLocation.with_courier_ids(courier_id) > Repo.one(timeout: :infinity), "with_courier_ids" => fn -> CourierLocation.with_courier_ids(courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity), "full custom" => fn -> CourierLocation > Ecto.Query.where(courier_id: ^courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity) }

81 Benchee.run %{ "DB View" => fn -> LatestCourierLocation > CourierLocation.with_courier_ids(courier_id) > Repo.one(timeout: :infinity), "with_courier_ids" => fn -> CourierLocation.with_courier_ids(courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity), "full custom" => fn -> CourierLocation > Ecto.Query.where(courier_id: ^courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity) } Same

82 Another job well done? Name with_courier_ids full custom DB View ips average 1.02 ms 1.17 ms ms deviation ±43.22% ±53.86% ±4.89% Comparison: with_courier_ids full custom DB View x slower x slower median 0.91 ms 0.96 ms ms 99th % 3.19 ms 4.25 ms ms

83 Another job well done?

84 Boom

85 Boom Elixir.DBConnection.ConnectionError

86 Boom Elixir.DBConnection.ConnectionError Elixir.DBConnection.ConnectionError

87 Boom Elixir.DBConnection.ConnectionError Elixir.DBConnection.ConnectionError Elixir.DBConnection.ConnectionError Elixir.DBConnection.ConnectionError Elixir.DBConnection.ConnectionError Elixir.DBConnection.ConnectionError Elixir.DBConnection.ConnectionError Elixir.DBConnection.ConnectionError

88 Whaaaat?

89 inputs = %{ "Big 2.3 Million locations" "No locations" "~200k locations" "~20k locations" } => => => => Inputs to the rescue! 3799, 8901, 4238, 4201 Benchee.run %{... "full custom" => fn(courier_id) -> CourierLocation > Ecto.Query.where(courier_id: ^courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity) }, inputs: inputs, time: 25, warmup: 5

90 inputs = %{ "Big 2.3 Million locations" "No locations" "~200k locations" "~20k locations" } => => => => Inputs to the rescue! 3799, 8901, 4238, 4201 Benchee.run %{... "full custom" => fn(courier_id) -> CourierLocation > Ecto.Query.where(courier_id: ^courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity) }, inputs: inputs, time: 25, warmup: 5

91 inputs = %{ "Big 2.3 Million locations" "No locations" "~200k locations" "~20k locations" } => => => => Inputs to the rescue! 3799, 8901, 4238, 4201 Benchee.run %{... "full custom" => fn(courier_id) -> CourierLocation > Ecto.Query.where(courier_id: ^courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity) }, inputs: inputs, time: 25, warmup: 5

92 inputs = %{ "Big 2.3 Million locations" "No locations" "~200k locations" "~20k locations" } => => => => Inputs to the rescue! 3799, 8901, 4238, 4201 Benchee.run %{... "full custom" => fn(courier_id) -> CourierLocation > Ecto.Query.where(courier_id: ^courier_id) > Ecto.Query.order_by(desc: :time) > Ecto.Query.limit(1) > Repo.one(timeout: :infinity) }, inputs: inputs, time: 25, warmup: 5

93 ##### With input Big 2.5 million locations ##### with_courier_ids full custom x slower DB View x slower ##### With input ~200k locations ##### DB View 3.57 with_courier_ids x slower full custom x slower ##### With input ~20k locations ##### DB View with_courier_ids x slower full custom x slower ##### With input No locations ##### Comparison: DB View with_courier_ids x slower full custom x slower

94 ##### With input Big 2.5 million locations ##### with_courier_ids full custom x slower DB View x slower ##### With input ~200k locations ##### DB View 3.57 with_courier_ids x slower full custom x slower ##### With input ~20k locations ##### DB View with_courier_ids x slower full custom x slower ##### With input No Comparison: DB View with_courier_ids full custom locations ##### x slower x slower woops

95 EXPLAIN ANALYZE SELECT c0."id", c0."courier_id", c0."location", c0."time", c0."accuracy", c0."inserted_at", c0."updated_at" FROM "courier_locations" AS c0 WHERE (c0."courier_id" = 3799) ORDER BY c0."time" DESC LIMIT 1; QUERY PLAN Limit (cost= rows=1 width=72) (actual time= rows=1 loops=1) -> Index Scan Backward using courier_locations_time_index on courier_locations c0 (cost= rows= width=72) actual time= rows=1 loops=1) Filter: (courier_id = 3799) Rows Removed by Filter: 1371 Planning time: ms Execution time: ms (6 rows)

96 EXPLAIN ANALYZE SELECT c0."id", c0."courier_id", c0."location", c0."time", c0."accuracy", c0."inserted_at", c0."updated_at" FROM "courier_locations" AS c0 WHERE (c0."courier_id" = 3799) ORDER BY c0."time" DESC LIMIT 1; QUERY PLAN Limit (cost= rows=1 width=72) (actual time= rows=1 loops=1) -> Index Scan Backward using courier_locations_time_index on courier_locations c0 (cost= rows= width=72) actual time= rows=1 loops=1) Filter: (courier_id = 3799) Rows Removed by Filter: 1371 Planning time: ms Execution time: ms (6 rows)

97 EXPLAIN ANALYZE SELECT c0."id", c0."courier_id", c0."location", c0."time", c0."accuracy", c0."inserted_at", c0."updated_at" FROM "courier_locations" AS c0 WHERE (c0."courier_id" = 3799) ORDER BY c0."time" DESC LIMIT 1; QUERY PLAN Limit (cost= rows=1 width=72) (actual time= rows=1 loops=1) -> Index Scan Backward using courier_locations_time_index on courier_locations c0 (cost= rows= width=72) actual time= rows=1 loops=1) Filter: (courier_id = 3799) Rows Removed by Filter: 1371 Planning time: ms Execution time: ms (6 rows)

98 EXPLAIN ANALYZE SELECT c0."id", c0."courier_id", c0."location", c0."time", c0."accuracy", c0."inserted_at", c0."updated_at" FROM "courier_locations" AS c0 WHERE (c0."courier_id" = 3799) ORDER BY c0."time" DESC LIMIT 1; QUERY PLAN Limit (cost= rows=1 width=72) (actual time= rows=1 loops=1) -> Index Scan Backward using courier_locations_time_index on courier_locations c0 (cost= rows= width=72) actual time= rows=1 loops=1) Filter: (courier_id = 3799) Rows Removed by Filter: 1371 Planning time: ms Execution time: ms (6 rows)

99 Combined Indexes

100 No locations

101 20k locations

102 200k locations

103 Lots of locations

104 Let s talk about Tail Call Optimization

105 body defmodule MyMap do def map_body([], _func), do: [] def map_body([head tail], func) do [func.(head) map_body(tail, func)]

106 body defmodule MyMap do def map_body([], _func), do: [] def map_body([head tail], func) do [func.(head) map_body(tail, func)]

107 body defmodule MyMap do def map_body([], _func), do: [] def map_body([head tail], func) do [func.(head) map_body(tail, func)]

108 body defmodule MyMap do def map_body([], _func), do: [] def map_body([head tail], func) do [func.(head) map_body(tail, func)]

109 body defmodule MyMap do def map_body([], _func), do: [] def map_body([head tail], func) do [func.(head) map_body(tail, func)]

110 defmodule MyMap do TCO def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) defp do_map_tco(acc, [], _function) do acc defp do_map_tco(acc, [head tail], func) do do_map_tco([func.(head) acc], tail, func)

111 defmodule MyMap do TCO def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) defp do_map_tco(acc, [], _function) do acc defp do_map_tco(acc, [head tail], func) do do_map_tco([func.(head) acc], tail, func)

112 defmodule MyMap do TCO def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) defp do_map_tco(acc, [], _function) do acc defp do_map_tco(acc, [head tail], func) do do_map_tco([func.(head) acc], tail, func)

113 defmodule MyMap do TCO def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) defp do_map_tco(acc, [], _function) do acc defp do_map_tco(acc, [head tail], func) do do_map_tco([func.(head) acc], tail, func)

114 defmodule MyMap do TCO def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) defp do_map_tco(acc, [], _function) do acc defp do_map_tco(acc, [head tail], func) do do_map_tco([func.(head) acc], tail, func)

115 defmodule MyMap do TCO def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) defp do_map_tco(acc, [], _function) do acc defp do_map_tco(acc, [head tail], func) do do_map_tco([func.(head) acc], tail, func)

116 defmodule MyMap do TCO def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) defp do_map_tco(acc, [], _function) do acc defp do_map_tco(acc, [head tail], func) do do_map_tco([func.(head) acc], tail, func)

117 defmodule MyMap do TCO def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) defp do_map_tco(acc, [], _function) do acc defp do_map_tco(acc, [head tail], func) do do_map_tco([func.(head) acc], tail, func)

118 defmodule MyMap do TCO def map_tco(list, function) do Enum.reverse do_map_tco([], list, function) defp do_map_tco(acc, [], _function) do acc defp do_map_tco(acc, [head tail], func) do do_map_tco([func.(head) acc], tail, func)

119 defmodule MyMap do arg_order def map_tco(list, function) do Enum.reverse do_map_tco(list, function, []) defp do_map_tco([], _function, acc) do acc defp do_map_tco([head tail], func, acc) do do_map_tco(tail, func, [func.(head) acc])

120 map_fun = fn(i) -> i + 1 inputs = %{ "Small (10 Thousand)" => "Middle (100 Thousand)" => "Big (1 Million)" => "Bigger (5 Million)" => } TCO Enum.to_list(1..10_000), Enum.to_list(1..100_000), Enum.to_list(1..1_000_000), Enum.to_list(1..5_000_000) Benchee.run %{ "tail-recursive" => fn(list) -> MyMap.map_tco(list, map_fun), "stdlib map" => fn(list) -> Enum.map(list, map_fun), "body-recursive" => fn(list) -> MyMap.map_body(list, map_fun), "tail-rec arg-order" => fn(list) -> MyMap.map_tco_arg_order(list, map_fun) }

121 map_fun = fn(i) -> i + 1 inputs = %{ "Small (10 Thousand)" => "Middle (100 Thousand)" => "Big (1 Million)" => "Bigger (5 Million)" => } TCO Enum.to_list(1..10_000), Enum.to_list(1..100_000), Enum.to_list(1..1_000_000), Enum.to_list(1..5_000_000) Benchee.run %{ "tail-recursive" => fn(list) -> MyMap.map_tco(list, map_fun), "stdlib map" => fn(list) -> Enum.map(list, map_fun), "body-recursive" => fn(list) -> MyMap.map_body(list, map_fun), "tail-rec arg-order" => fn(list) -> MyMap.map_tco_arg_order(list, map_fun) }

122 map_fun = fn(i) -> i + 1 inputs = %{ "Small (10 Thousand)" => "Middle (100 Thousand)" => "Big (1 Million)" => "Bigger (5 Million)" => } TCO Enum.to_list(1..10_000), Enum.to_list(1..100_000), Enum.to_list(1..1_000_000), Enum.to_list(1..5_000_000) Benchee.run %{ "tail-recursive" => fn(list) -> MyMap.map_tco(list, map_fun), "stdlib map" => fn(list) -> Enum.map(list, map_fun), "body-recursive" => fn(list) -> MyMap.map_body(list, map_fun), "tail-rec arg-order" => fn(list) -> MyMap.map_tco_arg_order(list, map_fun) }

123 ##### With input Small (10 Thousand) ##### stdlib map 5.05 K body-recursive 5.00 K x slower tail-rec arg-order 4.07 K x slower tail-recursive 3.76 K x slower ##### With input Middle (100 Thousand) ##### stdlib map body-recursive x slower tail-rec arg-order x slower tail-recursive x slower ##### With input Big (1 Million) ##### body-recursive stdlib map x slower tail-rec arg-order x slower tail-recursive x slower ##### With input Bigger (5 Million) ##### tail-rec arg-order 6.68 tail-recursive x slower stdlib map x slower body-recursive x slower TCO

124 ##### With input Small (10 Thousand) ##### stdlib map 5.05 K body-recursive 5.00 K x slower tail-rec arg-order 4.07 K x slower tail-recursive 3.76 K x slower ##### With input Middle (100 Thousand) ##### stdlib map body-recursive x slower tail-rec arg-order x slower tail-recursive x slower ##### With input Big (1 Million) ##### body-recursive stdlib map x slower tail-rec arg-order x slower tail-recursive x slower ##### With input Bigger (5 Million) ##### tail-rec arg-order 6.68 tail-recursive x slower stdlib map x slower body-recursive x slower TCO

125 ##### With input Small (10 Thousand) ##### stdlib map 5.05 K body-recursive 5.00 K x slower tail-rec arg-order 4.07 K x slower tail-recursive 3.76 K x slower ##### With input Middle (100 Thousand) ##### stdlib map body-recursive x slower tail-rec arg-order x slower tail-recursive x slower ##### With input Big (1 Million) ##### body-recursive stdlib map x slower tail-rec arg-order x slower tail-recursive x slower ##### With input Bigger (5 Million) ##### tail-rec arg-order 6.68 tail-recursive x slower stdlib map x slower body-recursive x slower Big Inputs

126 Order matters! ##### With input Small (10 Thousand) ##### stdlib map 5.05 K body-recursive 5.00 K x slower tail-rec arg-order 4.07 K x slower tail-recursive 3.76 K x slower ##### With input Middle (100 Thousand) ##### stdlib map body-recursive x slower tail-rec arg-order x slower tail-recursive x slower ##### With input Big (1 Million) ##### body-recursive stdlib map x slower tail-rec arg-order x slower tail-recursive x slower ##### With input Bigger (5 Million) ##### tail-rec arg-order 6.68 tail-recursive x slower stdlib map x slower body-recursive x slower

127 TCO

128 What about Memory?

129 WIP

130 ##### With input Small (10 Thousand) stdlib map KB body-recursive KB tail-rec arg-order KB tail-recursive KB - ##### 1.00x memory usage 1.86x memory usage 1.86x memory usage ##### With input Middle (100 Thousand) ##### stdlib map 1.53 MB body-recursive 1.53 MB x memory usage tail-rec arg-order 1.80 MB x memory usage tail-recursive 1.80 MB x memory usage ##### With input Big (1 Million) ##### stdlib map MB body-recursive MB x memory usage tail-rec arg-order MB x memory usage tail-recursive MB x memory usage ##### With input Bigger (5 Million) ##### tail-rec arg-order MB tail-recursive MB x memory usage stdlib map MB x memory usage body-recursive MB x memory usage Memory

131 ##### With input Small (10 Thousand) stdlib map KB body-recursive KB tail-rec arg-order KB tail-recursive KB - ##### 1.00x memory usage 1.86x memory usage 1.86x memory usage ##### With input Middle (100 Thousand) ##### stdlib map 1.53 MB body-recursive 1.53 MB x memory usage tail-rec arg-order 1.80 MB x memory usage tail-recursive 1.80 MB x memory usage ##### With input Big (1 Million) ##### stdlib map MB body-recursive MB x memory usage tail-rec arg-order MB x memory usage tail-recursive MB x memory usage ##### With input Bigger (5 Million) ##### tail-rec arg-order MB tail-recursive MB x memory usage stdlib map MB x memory usage body-recursive MB x memory usage Memory

132 What even is a benchmark?

133 A transformation of inputs config > Benchee.init > Benchee.system > Benchee.benchmark("job", fn -> magic ) > Benchee.measure > Benchee.statistics > Benchee.Formatters.Console.output > Benchee.Formatters.HTML.output

134 RUN YOUR OWN BENCHMARKS

135 Enjoy Benchmarking Tobias pragtob.info github.com/pragtob/benchee

136 Excursion into Statistics

137 Average average = total_time / iterations

138 Standard Deviation defp standard_deviation(samples, average, iterations) do total_variance = Enum.reduce samples, 0, fn(sample, total) -> total + :math.pow((sample - average), 2) variance = total_variance / iterations :math.sqrt variance

139 Spread of Values defp standard_deviation(samples, average, iterations) do total_variance = Enum.reduce samples, 0, fn(sample, total) -> total + :math.pow((sample - average), 2) variance = total_variance / iterations :math.sqrt variance

140 Raw Run Times

141 Histogram

142 Outliers

143 Standard Deviation

144 Median defp compute_median(run_times, iterations) do sorted = Enum.sort(run_times) middle = div(iterations, 2) if Integer.is_odd(iterations) do sorted > Enum.at(middle) > to_float else (Enum.at(sorted, middle) + Enum.at(sorted, middle - 1)) / 2

145 Average

146 Mediam

147 Boxplot