Operational Efficiency Hacks. John Allspaw Operations Engineering, Flickr

Transcription

1 Operational Efficiency Hacks John Allspaw Operations Engineering, Flickr

2 who am I? Manage the Flickr Operations group Wrote a geeky book:

3 Efficiencies

4 Efficiencies Doing more with the robots you ve got

5 Efficiencies Doing more with the robots you ve got Doing more with the humans you ve got

6 Some optimization rules

7 Some optimization rules - Don t rely on being able to tweak anything.

8 Some optimization rules - - Don t rely on being able to tweak anything. Don t waste too much time tuning when you have no evidence it ll matter.

9 Optimization rules performance tuning gains time spent tuning

10 Optimization rules We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Knuth, (or Hoare)

11 however...

12 Optimization rules

13 Optimization rules That doesn t give us an excuse to be lazy and inefficient.

14 Optimization rules That doesn t give us an excuse to be lazy and inefficient.

15 Optimization rules That doesn t give us an excuse to be lazy and inefficient. We lean on the experience of people in the community for evidence that tuning(s) might be a worthwhile thing to do.

16 Optimization rules Yet we should not pass up our opportunities in that critical 3 percent. Knuth, (or Hoare)

17 So... stop somewhere in here obvious tuning wins OMG I'm time for no reason

18 Our Context

19 Our Context - 24 TB of MySQL data

20 Our Context TB of MySQL data 32k/sec of MySQL writes

21 Our Context TB of MySQL data 32k/sec of MySQL writes 120k/sec of MySQL reads

22 Our Context TB of MySQL data 32k/sec of MySQL writes 120k/sec of MySQL reads 6 PB of photos

23 Our Context TB of MySQL data 32k/sec of MySQL writes 120k/sec of MySQL reads 6 PB of photos 10TB storage eaten per day

24 Our Context TB of MySQL data 32k/sec of MySQL writes 120k/sec of MySQL reads 6 PB of photos 10TB storage eaten per day 15,362 service monitors (alerts)

25 Infrastructure Hacks

26 Infrastructure Hacks - Examples of what changing software can do (plain old-fashioned performance tuning)

27 Infrastructure Hacks - - Examples of what changing software can do (plain old-fashioned performance tuning) Examples of what changing hardware can do (yay for Mr. Moore!)

28 Leaning on compilers (synthetic PHP benchmarks, not real-world) ( ICC-Benchmark.html)

29 PHP (real-world) php to php migration

30 Can now handle more with less same taste, less filling

31 Image Processing

32 Image Processing , Flickr was using ImageMagick for image processing (version 6.1.9)

33 Image Processing , Flickr was using ImageMagick for image processing (version 6.1.9) Changed to GraphicsMagick, about 15% faster at the time (version 1.1.5)

34 Image Processing , Flickr was using ImageMagick for image processing (version 6.1.9) Changed to GraphicsMagick, about 15% faster at the time (version 1.1.5) Only need a subset of ImageMagick features anyway for our purposes

35 Image Processing - OpenMP support ( - Allows parallelization of processing jobs, using multiple cores working on the same image - Some algorithms have more parallelization than others

36 Image Processing - Test script large-ish DSLR photos Cascade resizing each to 6 smaller sizes, semi-typical for Flickr s workload Each resize processed serially

37 Image Processing compiler differences (GM version , non-openmp)

38 Image Processing OpenMP differences OpenMP advantage (gcc 4.1.2, on quad core Xeon 2.00GHz)

39 Image Processing CPU differences

40 Diagonal Scaling

41 Diagonal Scaling - Vertically scaling your already horizontallyscaled nodes

42 Diagonal Scaling - - Vertically scaling your already horizontallyscaled nodes a.k.a. tech refresh

43 Diagonal Scaling Vertically scaling your already horizontallyscaled nodes a.k.a. tech refresh a.k.a. Moore s Law Surfing

44 Diagonal Scaling

45 Diagonal Scaling We replaced 67 old webservers with 18 new :

46 Diagonal Scaling We replaced 67 old webservers with 18 new : servers CPUs per server RAM per server drives per server total power peak GB 1x80GB GB 1x146GB

47 Diagonal Scaling We replaced 67 old webservers with 18 new : servers CPUs per server RAM per server drives per server ~70% LESS power total power peak GB 1x80GB U LESS rack space GB 1x146GB

48 Diagonal Scaling

49 Diagonal Scaling We replaced 23 old image processing boxes with 8 new

50 Diagonal Scaling We replaced 23 old image processing boxes with 8 new server photos/min rack total power peak

51 Diagonal Scaling We replaced 23 old image processing boxes with 8 new server photos/min rack ~75% FASTER total power peak U LESS rack space 65% LESS power

52 Diagonal Scaling We replaced 23 old image processing boxes with 8 new server photos/min rack ~75% FASTER 15U LESS rack space total power peak % LESS power

53 Diagonal Scaling We replaced 23 old image processing boxes with 8 new server photos/min rack ~75% FASTER 15U LESS rack space total power peak % LESS power

54 Diagonal Scaling We replaced 23 old image processing boxes with 8 new server photos/min rack ~75% FASTER 15U LESS rack space total power peak % LESS power from this to this

55 What do you do with old/slow machines?

56 What do you do with old/slow machines? - Liquidate

57 What do you do with old/slow machines? - Liquidate - Re-purpose as dev/staging/etc

58 What do you do with old/slow machines? - Liquidate - Re-purpose as dev/staging/etc - offline tasks

59 Offline Tasks

60 Offline Tasks - Out-of-band/asynchronous queuing and execution system, for non-realtime tasks

61 Offline Tasks - - Out-of-band/asynchronous queuing and execution system, for non-realtime tasks See here:

62 Offline Tasks - - Out-of-band/asynchronous queuing and execution system, for non-realtime tasks See here:

63 Offline Tasks Out-of-band/asynchronous queuing and execution system, for non-realtime tasks See here: See Myles Grant talk about it more here:

64 Offline Tasks Out-of-band/asynchronous queuing and execution system, for non-realtime tasks See here: See Myles Grant talk about it more here:

65 Runbook Hacks WTF HAPPENED LAST NIGHT?!

66 Why?

67 Why? As infrastructure grows, try to keep the Humans:Machines ratio from getting out of hand

68 Why? As infrastructure grows, try to keep the Humans:Machines ratio from getting out of hand Some of the How:

69 Why? As infrastructure grows, try to keep the Humans:Machines ratio from getting out of hand Some of the How: - teach machines to build themselves

70 Why? As infrastructure grows, try to keep the Humans:Machines ratio from getting out of hand Some of the How: - teach machines to build themselves - teach machines to watch themselves

71 Why? As infrastructure grows, try to keep the Humans:Machines ratio from getting out of hand Some of the How: - teach machines to build themselves - teach machines to watch themselves - teach machines to fix themselves

72 Why? As infrastructure grows, try to keep the Humans:Machines ratio from getting out of hand Some of the How: - teach machines to build themselves - teach machines to watch themselves - teach machines to fix themselves - reduce MTTR by streamlining

73 Automated Infrastructure

74 Automated Infrastructure - If there is only one thing you do, automatic configuration and deployment management should be it.

75 Automated Infrastructure - If there is only one thing you do, automatic configuration and deployment management should be it. - See: - Opscode/Chef ( - Puppet ( - System Imager/Configurator (

76 Conguration Management Codeswarm

77 Time Machine time is cheaper than human time. If a failure results in some commands being run to fix it, make the machines do it. (i.e., don t wake people up for stupid things!)

78 Aggregate Monitoring

79 Aggregate Monitoring Don t care about single nodes, only care about delta change of metrics/faults - Warn ( ) on X % change - Page (wake up) on Y % change

80 Aggregate Monitoring Don t care about single nodes, only care about delta change of metrics/faults - Warn ( ) on X % change - Page (wake up) on Y % change High and low water marks for some metrics

81 Self-Healing

82 Self-Healing Make service monitoring fix common failure scenarios, notify us later about it.

83 Self-Healing Make service monitoring fix common failure scenarios, notify us later about it.

84 Self-Healing Make service monitoring fix common failure scenarios, notify us later about it. Daemons/processes run on machines, will take corrective action under certain conditions, and report back with what they did.

85 Self-Healing Make service monitoring fix common failure scenarios, notify us later about it. Daemons/processes run on machines, will take corrective action under certain conditions, and report back with what they did.

86 Self-Healing Make service monitoring fix common failure scenarios, notify us later about it. Daemons/processes run on machines, will take corrective action under certain conditions, and report back with what they did. Can greatly reduce your mean time to recovery (MTTR)

87 Self-Healing Make service monitoring fix common failure scenarios, notify us later about it. Daemons/processes run on machines, will take corrective action under certain conditions, and report back with what they did. Can greatly reduce your mean time to recovery (MTTR)

88 Basic Apache Example

89 Basic Apache Example 1. Webserver not running?

90 Basic Apache Example 1. Webserver not running? 2. Under certain conditions, try to start it, and that this happened. (I ll read it tomorrow)

91 Basic Apache Example 1. Webserver not running? 2. Under certain conditions, try to start it, and that this happened. (I ll read it tomorrow) 3. Won t start? Assume something s really wrong, so don t keep trying ( that, too)

92 MySQL Self-Healing

93 MySQL Self-Healing Some MySQL Issues fixed by the machines

94 MySQL Self-Healing Some MySQL Issues fixed by the machines

95 MySQL Self-Healing Some MySQL Issues fixed by the machines - Kill long-running SELECT queries (marked safe to kill)

96 MySQL Self-Healing Some MySQL Issues fixed by the machines - Kill long-running SELECT queries (marked safe to kill) - Queries not safe to kill are marked by the application as NO KILL in comments

97 MySQL Self-Healing Some MySQL Issues fixed by the machines - Kill long-running SELECT queries (marked safe to kill) - Queries not safe to kill are marked by the application as NO KILL in comments - Run EXPLAIN on killed queries, and report the results

98 MySQL Self-Healing Some MySQL Issues fixed by the machines - Kill long-running SELECT queries (marked safe to kill) - Queries not safe to kill are marked by the application as NO KILL in comments - Run EXPLAIN on killed queries, and report the results - Keep track of the query types and databases that need the most killing, produce a DBs that Suck report

99 MySQL Self-Healing

100 MySQL Self-Healing Some MySQL Replication issues fixed by the machines, by error

101 MySQL Self-Healing Some MySQL Replication issues fixed by the machines, by error - Skip errors that can safely be skipped and restart slave threads

102 MySQL Self-Healing Some MySQL Replication issues fixed by the machines, by error - Skip errors that can safely be skipped and restart slave threads - Force refetch of replication binlogs on: (ER_PARSE_ERROR)

103 MySQL Self-Healing Some MySQL Replication issues fixed by the machines, by error - Skip errors that can safely be skipped and restart slave threads - Force refetch of replication binlogs on: (ER_PARSE_ERROR) - Re-run queries on: (ER_LOCK_WAIT_TIMEOUT) (ER_LOCK_DEADLOCK)

104 Troubleshooting

105 Code and Config Deploy Logs

106 Code and Config Deploy Logs 1. ESSENTIAL

107 Code and Config Deploy Logs 1. ESSENTIAL 2. MANDATORY

108 Communications

109 Communications Internal IRC - For ongoing discussions - Logged, so infinite scrollback

110 Communications Internal IRC - For ongoing discussions - Logged, so infinite scrollback IM Bot (built on libyahoo2.sf.net) - For production changes - Broadcasts all to all contacts - Logged, and injected into IRC - IM Status = who is in primary/secondary on-call

111 Communications Internal IRC - For ongoing discussions - Logged, so infinite scrollback IM Bot (built on libyahoo2.sf.net) - For production changes - Broadcasts all to all contacts - Logged, and injected into IRC - IM Status = who is in primary/secondary on-call All of IRC and IM Bot slurped into a search index

112

113 when

114 when what

115 when what detailed what*

116 when what detailed what* *also points to what commands should be used to back out the changes

117 when what detailed what* who *also points to what commands should be used to back out the changes

118 when what detailed what* who *also points to what commands should be used to back out the changes

119 when what detailed what* who time of last deploy at top of ganglia *also points to what commands should be used to back out the changes

120

121

122 IM Bot (timestamps help correlation)

123 IM Bot (timestamps help correlation)

124 IM Bot (timestamps help correlation) all IRC, IM bot into searchable history

125 Morals of Our Stories

126 Morals of Our Stories - Optimizations can be a Very Good Thing

127 Morals of Our Stories - - Optimizations can be a Very Good Thing Weigh time spent optimizing against expected gains

128 Morals of Our Stories Optimizations can be a Very Good Thing Weigh time spent optimizing against expected gains Lean on others for how much expected gains mean for different scenarios

129 Morals of Our Stories Optimizations can be a Very Good Thing Weigh time spent optimizing against expected gains Lean on others for how much expected gains mean for different scenarios Plain old-fashioned intuition

130 Some Wisdom Nuggets Jon Prall s 85 WebOps Rules: operations-rules-to-live-by.html

131 Questions?