Scaling Up from 1000 to 10 Nodes

Size: px

Start display at page:

Download "Scaling Up from 1000 to 10 Nodes"

Philomena Wilkerson
5 years ago
Views:

1 Scaling Up from 1000 to 10 Nodes Anton Lavrik Alert Logic, Inc. 1

2 Alert Logic vs Twitter: daily numbers (2012) Alert Logic Twitter Events received 11.5B 0.5B Volume received 5TB 70GB Logs produced? 100TB 2

3 Alert Logic vs Twitter: daily numbers (2012) Alert Logic Twitter Events received 11.5B 0.5B Volume received Logs produced 5TB 10GB 70GB 100TB 3

4 Log collection: before rewrite hosts H H H H H H H x 100K appliances A A A A A x 1000 backend Alert Logic WAN users 4

5 Log collection: goal hosts & appliances + agents H A' H' H H H H H H' A' H' A' H' H' H' H' H' H' H A' H' x 1M+ backend Alert Logic users 5

6 Log collection: goal hosts & appliances + agents H A' H' H H H H H H' A' H' A' H' H' H' H' H' H' H A' H' x 1M+ new backend Alert Logic users 6

7 Log collection: zooming in H' A' H' A' H' H' H' H' H' H' Collection Storage Analysis Web UI Search Reports Alerts 7

8 Log collection: zooming in C C A' C H' C C C clients appliance agent Load Balancing Collection Controller Collection Controller Collection Controller Persistent Queue 8

9 Log collection: zooming in clients C C C C x 100K+ LB Collection Controller Collection Controller Collection Controller 3 Persistent Queue 9

10 Log collection: zooming in clients C C C C x 100K+ HTTPS LB sticky sessions Collection Controller Collection Controller Collection Controller 3 Persistent Queue 10

11 Log collection: zooming in clients C C C C x 100K+ HTTPS LB sticky sessions Collection Controller Collection Controller Collection Controller 3 Persistent Queue 11

12 Log collection: before rewrite hosts H H H H H H H x 100K appliances A A A A A x 1000 backend Alert Logic WAN users 12

13 Log collection: zooming in clients C C C x 40K collection controller node CC CC CC controller Erlang processes x 40K Persistent Queue 13

14 Log collection: zooming in clients C C C x 40K collection controller node CC CC CC controller Erlang processes x 40K Stats Configs DBs... Persistent Queue 14

15 What changes when you scale up from 1000 to 10 nodes? 15

16 Everything! 16

17 Increases per node/cpu core request rate data volume number of open connections number of processes per VM failure rate bugs and memory leaks get exposed 17

18 Colocated environment different clients installed in different environments create different load patterns one set of clients can affect health of others can t easily upgrade/troubleshoot individual clients no downtime tolerance 18

19 Application monitoring logging is less useful for troubleshooting need real-time monitoring, dashboards need visualization 19

20 Problem: high memory usage leading to OOM kill 20

21 Hint: many processes each dealing with lots of data clients C C C x 40K collection controller node CC CC CC controller Erlang processes x 40K Persistent Queue 21

22 High memory usage: force garbage collection garbage_collect(p) 22

23 Problem: memory leak 23

24 Memory leak: searching... some luck [ garbage_collect(p) P <- processes() ]. 24

25 Memory leak: troubleshooting... [{M, P, process_info(p, [registered_name, initial_call, current_function, dictionary]), B} {P, M, B} <- lists:sublist(lists:reverse(lists:keysort(2, [case process_info(p, binary) of {_, Bins} -> SortedBins = lists:usort(bins), {_, Sizes, _} = lists:unzip3(sortedbins), {P, lists:sum(sizes), SortedBins}; _ - > {P, 0, []} end P <- processes()])), 5)]. 25

26 Memory leak: troubleshooting... [ {M, P, process_info(p, [registered_name, initial_call, current_function, dictionary]), B} {P, M, B} <- lists:sublist(lists:reverse(lists:keysort(2, [ case process_info(p, binary) of {_, Bins} -> SortedBins = lists:usort(bins), {_, Sizes, _} = lists:unzip3(sortedbins), {P, lists:sum(sizes), []}; _ -> {P, 0, []} end P <- processes() ] )), 5) ]. 26

27 Memory leak: searching... [{ ,< >, [{registered_name,alcollect_client_sup}, {initial_call,{proc_lib,init_p,5}}, {current_function,{gen_server,loop,6}}, {dictionary,[{'$ancestors',[alcollect_sup,lmcollect_sup, < >]}, {'$initial_call', {supervisor,alcollect_client_sup,1}}]}], []}, { ,< >, [{registered_name,gproc}, {initial_call,{proc_lib,init_p,5}}, {current_function,{gen_server,loop,6}}, {dictionary,[{'$ancestors',[gproc_sup,< >]}, {'$initial_call',{gproc,init,1}}]}], []}, 27

28 Memory leak: searching... victory! supervisor:terminate_child(lmcollect_sup, lmcollect_ls_config). application:stop(lmcollect)

29 Memory leak: sub-binary references process heaps ets tables message queues driver queues 29

30 Memory leak: preventing binary references 1. binary:copy(x) 2. -define(copy_binaries(x), binary_to_term(term_to_binary(x))). 30

31 Problem: supervisors shutdown after children exceed max restart rate 31

32 Preventing supervisor shutdowns Delay child crashes: 2 options 1. supervisor2 2. manually insert delays for each gen_server s Mod:handle_X handle_call() -> try do_handle_call() catch Class:Reason -> timer:sleep(1000), exit({crash, Class, Reason, get_stracktrace()}) end 32

33 Problem: stall while using gen_udp 33

34 Problem: stall while using gen_udp 34

35 usr/erlang/erts-5.8.5/src/prim_inet.erl send(s, Data, OptList) when is_port(s), is_list(optlist) ->?DBG_FORMAT("prim_inet:send(~p, ~p)~n", [S,Data]), try erlang:port_command(s, Data, OptList) of false -> % Port busy and nosuspend option passed?dbg_format("prim_inet:send() -> {error,busy}~n", []), {error,busy}; true -> receive {inet_reply,s,status} ->?DBG_FORMAT("prim_inet:send() -> ~p~n", [Status]), Status end catch error:_error ->?DBG_FORMAT("prim_inet:send() -> {error,einval}~n", []), {error,einval} end. 35

36 Preventing gen_server stall when using gen_udp 1. gen_server2 2. asynchronous gen_udp:send - gen_udp:send(socket, Address, Port, SyslogMessage); + try erlang:port_command(socket, + [[((Port) bsr 8) band 16#ff, + (Port) band 16#ff], + [A band 16#ff, B band 16#ff, + C band 16#ff, D band 16#ff], + SyslogMessage])of + true -> ok + catch... +handle_info({inet_reply, _, ok}, State) -> + {noreply, State}; 3. fix gen_udp! 36

37 Problem: client-side latencies: 50%-ile 37

38 Client-side latencies: 90%-ile 38

39 Client-side latencies: 99%-ile 39

40 Client-side latencies: max 40

41 Summary 41

42 Erlang is a very efficient tool* 42

43 Erlang is a very efficient tool* * you need to know your tools 43

44 Excellent runtime for running highly concurrent applications + soft real-time Simple and powerful language Fault-tolerance Stable Rich troubleshooting capabilities Growing community 44

45 A dream: Erlang/OSP (Open Server Platform) 45

46 revise OTP establish OSP principles and building blocks packaging and deployment: e.g. get rid of reltool.config logging out of the box: standard error, syslog (including sasl) http client that works well clients for popular databases: MySQL, Postgres alternative distributed Erlang: cluster membership, leader election, distributed locking 46

47 Questions? Thank you 47

48 Optional: if there s enough time and interest 48

49 Problem: httpc - Erlang/OTP HTTP client 49

50 httpc bug: mixes up responses Server responds with 200: Oct 23 02:20:36 lm-scale-api-01 al-hostmeta[17814]: ALH00002I "GET /host_metadata/1710/ e1924d5bfc58ac9576d6b31f261e0d137d56e142 HTTP/1.1" performed in 4 ms Client magically receives 204: Oct 23 02:20:36 lm-scale-lmcollect-02 al-lmcollect[9436]: ALL00000T al_httpc:61: http get e1924d5bfc58ac9576d6b31f261e0d137d56e142 performed in ms Oct 23 02:20:36 lm-scale-lmcollect-02 al-lmcollect[9436]: AHM00001E Error querying host metadata for key 1710, e1924d5bfc58ac9576d6b31f261e0d137d56e142: {unknown_http_response, {{"HTTP/1.1",204,"No Content"},[{"date","Tue, 23 Oct :20:36 GMT"},{"server","MochiWeb/1.0 (Any of you quaids got a smint?)"}, {"content-length","0"}],<<>>}} 50

51 httpc bug: mixes up responses Server responds with 204: Oct 23 02:20:38 lm-scale-api-01 al-hostmeta[17814]: ALH00002I "PUT /host_metadata/1712/ fd09bc458d5ee0a250668cf9bf41b0bea HTTP/1.1" performed in 4 ms Client receives 404: Oct 23 02:20:38 lm-scale-lmcollect-02 al-lmcollect[9436]: ALL00000T al_httpc:86: http put fd09bc458d5ee0a250668cf9bf41b0bea performed in ms Oct 23 02:20:38 lm-scale-lmcollect-02 al-lmcollect[9436]: AHM00000E Error updating host metadata for key 1712, fd09bc458d5ee0a250668cf9bf41b0bea : {unknown_http_response, {404,<<"No metadata for key">>}} 51

Performance Optimization 101. Louis-Philippe Gauthier Team AdGear Trader

Performance Optimization 101. Louis-Philippe Gauthier Team AdGear Trader Performance Optimization 101 Louis-Philippe Gauthier Team leader @ AdGear Trader Exercise HTTP API server API GET /date - returns today s date GET /time - returns the unix time in seconds HTTP API server