Michelin Starred Cooking with Chef

Transcription

1 Michelin Starred Cooking with Chef Jon Cowie,

2 What?

3 What? Chef at Etsy

4 What? Chef at Etsy Familiarity and Understanding

5 What? Chef at Etsy Familiarity and Understanding Critical Approach and Experimentation

6 What? Chef at Etsy Familiarity and Understanding Critical Approach and Experimentation Use The Source

7 What? Chef at Etsy Familiarity and Understanding Critical Approach and Experimentation Use The Source A liberal sprinkling of screwups

8 What? Chef at Etsy Familiarity and Understanding Critical Approach and Experimentation Use The Source A liberal sprinkling of screwups Open Sourced Goodness - We re all here!

9 What? Chef at Etsy Familiarity and Understanding Critical Approach and Experimentation Use The Source A liberal sprinkling of screwups Open Sourced Goodness - We re all here! [x] =

10 Opscode is Orange, Velocity is Blue. In Soviet Russia, Cookbook writes you.

11 Chef at Etsy

12 Our Setup

13 Our Setup Open Source chef server

14 Our Setup Open Source chef server Backup to Opscode Platform

15 Our Setup Open Source chef server Backup to Opscode Platform ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)

16 Our Setup Open Source chef server Backup to Opscode Platform ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac) KVM & lxc virts, self hosted too.

17 Our Setup Open Source chef server Backup to Opscode Platform ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac) KVM & lxc virts, self hosted too. Never test in production!

18 Our Setup Open Source chef server Backup to Opscode Platform ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac) KVM & lxc virts, self hosted too. Never test in production! Many chefs don t spoil our soup

19 Our Setup Open Source chef server Backup to Opscode Platform ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac) KVM & lxc virts, self hosted too. Never test in production! Many chefs don t spoil our soup

20 Familiarity and Understanding

21 not (C) Alexander McQueen

22

23 Insight

24 Insight Simplicity

25 Insight Simplicity Standards

26 Insight

27 Insight You should never have to say I don t know.

28 Insight You should never have to say I don t know. What, where, when, why, how long?

29 Insight You should never have to say I don t know. What, where, when, why, how long? Easy, and I ll show you how!

30

31 Chef Dashboard

32 Chef Dashboard Chef handler sends metrics to graphite [4]

33 Chef Dashboard Chef handler sends metrics to graphite [4] git clone git://github.com/etsy/chef-handlers.git

34 Chef Dashboard Chef handler sends metrics to graphite [4] git clone git://github.com/etsy/chef-handlers.git Set your graphite server s URL in graphite.rb

35 Chef Dashboard Chef handler sends metrics to graphite [4] git clone git://github.com/etsy/chef-handlers.git Set your graphite server s URL in graphite.rb Add the following to client.rb

36 Chef Dashboard Chef handler sends metrics to graphite [4] git clone git://github.com/etsy/chef-handlers.git Set your graphite server s URL in graphite.rb Add the following to client.rb require "<clonedir>/graphite.rb" graphite_handler = GraphiteReporting.new report_handlers << graphite_handler exception_handlers << graphite_handler

37 Chef Dashboard Chef handler sends metrics to graphite [4] git clone git://github.com/etsy/chef-handlers.git Set your graphite server s URL in graphite.rb Add the following to client.rb require "<clonedir>/graphite.rb" graphite_handler = GraphiteReporting.new report_handlers << graphite_handler exception_handlers << graphite_handler Etsy dashboards framework [5]

38 [12:54:01] <irccat> Chef run failed on gfernandez.vm.ny4dev.etsy.com [12:54:02] <irccat> [12:54:02] <irccat> [12:54:07] <irccat> Chef run failed on buildtest11.ny4dev.etsy.com [12:54:07] <irccat> [12:54:07] <irccat>

39 Chef irccat Alerts

40 Chef irccat Alerts Chef handler send fails to irc[4] via irccat [6]

41 Chef irccat Alerts Chef handler send fails to irc[4] via irccat [6] git clone git://github.com/etsy/chef-handlers.git

42 Chef irccat Alerts Chef handler send fails to irc[4] via irccat [6] git clone git://github.com/etsy/chef-handlers.git Set your irccat[6] server s URL in logtoirc.rb

43 Chef irccat Alerts Chef handler send fails to irc[4] via irccat [6] git clone git://github.com/etsy/chef-handlers.git Set your irccat[6] server s URL in logtoirc.rb Add the following to client.rb

44 Chef irccat Alerts Chef handler send fails to irc[4] via irccat [6] git clone git://github.com/etsy/chef-handlers.git Set your irccat[6] server s URL in logtoirc.rb Add the following to client.rb require "<clonedir>/logtoirc.rb" exception_handlers << Etsy::LogToIRC.new

45 ~ > knife node lastrun buildtest11.ny4dev.etsy.com Status failed Elapsed Time Start Time :06: End Time :06: Recipe Action Resource Type Resource Backtrace <snip> Exception Chef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version el5 of php not found. Did you specify both version and release? (version-release, e.g fc6)

46 ~ > knife search node 'lastrun_debug_formatted_exception:chef\: \:Exceptions\:\:Package*' -a lastrun.debug.formatted_exception 5 items found id: masterrestore.ny4.etsy.com lastrun.debug.formatted_exception: Chef::Exceptions::Package: package[postgresql-server] (postgresql::server-8.3 line 1) had an error: Installed package postgresql-server pgdg_id is newer than candidate package postgresql-server pgdg_id.rhel5 id: buildtest11.ny4dev.etsy.com lastrun.debug.formatted_exception: Chef::Exceptions::Package: package[php] (php::buildtest line 20) had an error: Version el5 of php not found. Did you specify both version and release? (version-release, e.g fc6) <snip>

47 Chef lastrun Info

48 Chef lastrun Info Chef handler and knife plugin [7]

49 Chef lastrun Info Chef handler and knife plugin [7] gem install knife-lastrun

50 Chef lastrun Info Chef handler and knife plugin [7] gem install knife-lastrun Add the following to client.rb

51 Chef lastrun Info Chef handler and knife plugin [7] gem install knife-lastrun Add the following to client.rb require "lastrun_update" handler = LastRunUpdateHandler.new report_handlers << handler exception_handlers << handler

52 Chef lastrun Info Chef handler and knife plugin [7] gem install knife-lastrun Add the following to client.rb require "lastrun_update" handler = LastRunUpdateHandler.new report_handlers << handler exception_handlers << handler knife node lastrun <nodename>

53 Simplicity

54 Simplicity Think of yourself at 3AM!

55 Simplicity Think of yourself at 3AM! Please, won t you think of the new guy?

56 Simplicity Think of yourself at 3AM! Please, won t you think of the new guy? Minimize the logics!

57 Simplicity Think of yourself at 3AM! Please, won t you think of the new guy? Minimize the logics! As few logical steps from start to finish as possible.

58 Simplicity - Not! Date: Mon Dec :07:18 GMT+0000 (GMT) Subject: so close to death # Don't install v2 on search or Cent 5.6 nodes -if node[:fqdn]!~ /\b(^(preprod-)?search[0-9]{2} ny4dev\.etsy\.com ^(preprod-)? giftsweb[0-9]{2} ^db(shard spare data)[0-9]{2} ^qa-web01 ^devsearch[0-9]{2} ^nagios01 ^webnest[0-9]{2} ^prodking[0-9]{2} ^sandboxweb[0-9]{2} ^virt((0[5-9]) (1[0-9])) ^msysmgr[0-9]{2} ^msysmta[0-9]{2} ^dbconvo[0-9]{2} ^dbshowcase01 atlasweb[0-9]{2} devnagios[0-9]{2} cimaster02 worker[0-9]{2} ^ganglia[0-9]{2} ^imgcache[0-9]{2} imgconvert[0-9]{2} ^imgwriter[0-9]{2} dev-img02 ^datacache04 ^graphite01 ^graphite03 ^webutil03 ^webutil04 ^statsd01 ^maintweb[0-9]{2} ^(dev- preprod-)?convosearch[0-9]{2} deployinator[0-9]{2} ^wpadmin01 ^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("web56") == false and node.role?("preprodweb56") == false and node.role?("princess53") == false +if node[:fqdn]!~ /\b(^(preprod-)?search[0-9]{2} ny4dev\.etsy\.com ^(preprod-)? giftsweb[0-9]{2} ^db(shard spare data)[0-9]{2} ^qa-web01 ^devsearch[0-9]{2} ^nagios01 ^webnest[0-9]{2} ^prodking[0-9]{2} ^sandboxweb[0-9]{2} ^virt((0[5-9]) (1[0-9])) ^msysmgr[0-9]{2} ^msysmta[0-9]{2} ^dbconvo[0-9]{2} ^dbshowcase01 atlasweb[0-9]{2} devnagios[0-9]{2} cimaster02 worker[0-9]{2} ^ganglia[0-9]{2} ^imgcache[0-9]{2} imgconvert[0-9]{2} ^imgwriter[0-9]{2} dev-img02 ^datacache04 ^graphite01 ^graphite03 ^webutil03 ^webutil04 ^statsd01 ^maintweb[0-9]{2} ^(dev- preprod-)?convosearch[0-9]{2} deployinator[0-9]{2} ^wpadmin01 ^(preprod-)?dbtasks[0-9]{2})\b/ and node.role?("web56") == false and node.role?("preprodweb56") == false and node.role?("princess53") == false and node.role?("auth") == false

59 Simplicity - Better! if node.chef_environment == "libmemcached_upgrade" package "libmemcached" do version " " action :install end <snip> else package "libmemcached" do version " " action :install end <snip> end

60 Simplicity - Complexity

61 Simplicity - Complexity Sometimes you need complex behaviour

62 Simplicity - Complexity Sometimes you need complex behaviour Don t fight it, try to abstract it.

63 Simplicity - Complexity Sometimes you need complex behaviour Don t fight it, try to abstract it. Case in point: Syslog-ng refactor

64 Case Study: Syslog-ng

65 Case Study: Syslog-ng 36 recipes

66 Case Study: Syslog-ng 36 recipes 30 versions of syslog-ng.conf

67 Case Study: Syslog-ng 36 recipes 30 versions of syslog-ng.conf 27 manually configured files in /etc/syslogng.d on central server

68 Case Study: Syslog-ng 36 recipes 30 versions of syslog-ng.conf 27 manually configured files in /etc/syslogng.d on central server Edge cases and exceptions galore

69 Case Study: Syslog-ng

70 Case Study: Syslog-ng Down to:

71 Case Study: Syslog-ng Down to: 2 recipes (one client, one server)

72 Case Study: Syslog-ng Down to: 2 recipes (one client, one server) 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d)

73 Case Study: Syslog-ng Down to: 2 recipes (one client, one server) 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d) Attributes in roles

74 Case Study: Syslog-ng Down to: 2 recipes (one client, one server) 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d) Attributes in roles Not open sourced yet, sorry :(

75 Case Study: Syslog-ng "syslog": { "group": "preprod_web", "items": { "web_apache_access_log": { "source": "/var/log/httpd/access_log", "source_program_override": "APACHEACCESS: ", "destination": "<snip>/access.log", "destination_filters": [ "host('^preprod-web')", "match('apacheaccess')" ], "destination_options": [ "template_escape(no)" ] }, } }

76 Remember, No Panacea!

77 Remember, No Panacea! A new package hits the repo.

78 Remember, No Panacea! A new package hits the repo. Are you in control of when it goes out?

79 Remember, No Panacea! A new package hits the repo. Are you in control of when it goes out? Memcached Outage

80 Remember, No Panacea! A new package hits the repo. Are you in control of when it goes out? Memcached Outage Do you know what services are going to restart and when?

81 Remember, No Panacea! A new package hits the repo. Are you in control of when it goes out? Memcached Outage Do you know what services are going to restart and when? Image Service Outage

82 Standards

83 Standards Not going to talk about testing [8]!

84 Standards Not going to talk about testing [8]! But I don t have time for standards!

85 Standards - No Time!

86 Standards - No Time! I won t say Make Time, but you should...

87 Standards - No Time! I won t say Make Time, but you should... For a quick win, try Foodcritic

88 Standards - No Time! I won t say Make Time, but you should... For a quick win, try Foodcritic Good out of the box rules

89 Standards - No Time! I won t say Make Time, but you should... For a quick win, try Foodcritic Good out of the box rules Jenkins integration in seconds

90 Standards - No Time! I won t say Make Time, but you should... For a quick win, try Foodcritic Good out of the box rules Jenkins integration in seconds Supports custom rules

91 Standards - No Time! I won t say Make Time, but you should... For a quick win, try Foodcritic Good out of the box rules Jenkins integration in seconds Supports custom rules Plays well with others

92 Foodcritic

93 Foodcritic gem install foodcritic

94 Foodcritic gem install foodcritic foodcritic <cookbook_repo>

95 Foodcritic gem install foodcritic foodcritic <cookbook_repo> Simple Jenkins job:

96 Foodcritic gem install foodcritic foodcritic <cookbook_repo> Simple Jenkins job: #!/usr/bin/env rvm-shell foodcritic -f correctness.

97 Standards at Etsy

98 Standards at Etsy style not correctness [9]

99 Standards at Etsy style not correctness [9] ETSY001 - Package or yum_package resource used with :upgrade action

100 Standards at Etsy style not correctness [9] ETSY001 - Package or yum_package resource used with :upgrade action ETSY002 - Execute resource used to run git commands

101 Standards at Etsy style not correctness [9] ETSY001 - Package or yum_package resource used with :upgrade action ETSY002 - Execute resource used to run git commands ETSY003 - Execute resource used to run curl or wget commands

102 Standards at Etsy style not correctness [9] ETSY001 - Package or yum_package resource used with :upgrade action ETSY002 - Execute resource used to run git commands ETSY003 - Execute resource used to run curl or wget commands ETSY004 - Execute resource defined without conditional or action :nothing

103 Standards at Etsy style not correctness [9] ETSY001 - Package or yum_package resource used with :upgrade action ETSY002 - Execute resource used to run git commands ETSY003 - Execute resource used to run curl or wget commands ETSY004 - Execute resource defined without conditional or action :nothing ETSY005 - Action :restart sent to a core service

104 Standards at Etsy style not correctness [9] ETSY001 - Package or yum_package resource used with :upgrade action ETSY002 - Execute resource used to run git commands ETSY003 - Execute resource used to run curl or wget commands ETSY004 - Execute resource defined without conditional or action :nothing ETSY005 - Action :restart sent to a core service ETSY006 - Execute resource used to run chef-provided command

105 Standards at Etsy style not correctness [9] ETSY001 - Package or yum_package resource used with :upgrade action ETSY002 - Execute resource used to run git commands ETSY003 - Execute resource used to run curl or wget commands ETSY004 - Execute resource defined without conditional or action :nothing ETSY005 - Action :restart sent to a core service ETSY006 - Execute resource used to run chef-provided command ETSY007 - Package or yum_package resource used to install core package without specific version number

106 Standards at Etsy style not correctness [9] ETSY001 - Package or yum_package resource used with :upgrade action ETSY002 - Execute resource used to run git commands ETSY003 - Execute resource used to run curl or wget commands ETSY004 - Execute resource defined without conditional or action :nothing ETSY005 - Action :restart sent to a core service ETSY006 - Execute resource used to run chef-provided command ETSY007 - Package or yum_package resource used to install core package without specific version number

107 Standards at Etsy

108 Standards at Etsy ETSY001 - Written after Memcached Outage

109 Standards at Etsy ETSY001 - Written after Memcached Outage Package or yum_package resource used with :upgrade action

110 Standards at Etsy ETSY001 - Written after Memcached Outage Package or yum_package resource used with :upgrade action package "memcached" do action :upgrade end

111 Standards at Etsy

112 Standards at Etsy ETSY005 - Written after a total Image service outage

113 Standards at Etsy ETSY005 - Written after a total Image service outage Action :restart sent to a core service

114 Standards at Etsy ETSY005 - Written after a total Image service outage Action :restart sent to a core service cookbook_file "/etc/httpd/conf.d/myvhost.conf" do source "myvhost.conf" notifies :restart, resources(:service => "httpd") end

115 Critical Approach and Experimentation

116 CA&E

117 CA&E Chef is by necessity generic

118 CA&E Chef is by necessity generic...so don t take Opscode s word for it.

119 CA&E Chef is by necessity generic...so don t take Opscode s word for it. If it doesn t work well for you, change it!

120 CA&E Chef is by necessity generic...so don t take Opscode s word for it. If it doesn t work well for you, change it! Case Study - Etsy Environments rollout

121 Environments Rollout

122 Environments Rollout We knew we needed them

123 Environments Rollout We knew we needed them Simple enough, right?

124 Environments Rollout We knew we needed them Simple enough, right? Let s look at the workflow...

125 Env: Standard Workflow

126 Env: Standard Workflow knife cookbook show php

127 Env: Standard Workflow knife cookbook show php Change version number in metadata.rb

128 Env: Standard Workflow knife cookbook show php Change version number in metadata.rb Change version constraint in foo.json

129 Env: Standard Workflow knife cookbook show php Change version number in metadata.rb Change version constraint in foo.json Commit and push changes to git

130 Env: Standard Workflow knife cookbook show php Change version number in metadata.rb Change version constraint in foo.json Commit and push changes to git knife cookbook upload php --freeze

131 Env: Standard Workflow knife cookbook show php Change version number in metadata.rb Change version constraint in foo.json Commit and push changes to git knife cookbook upload php --freeze knife environment from file foo.json

132 Env: Our Issues

133 Env: Our Issues 41 people with Chef repo access

134 Env: Our Issues 41 people with Chef repo access Most with knife keys

135 Env: Our Issues 41 people with Chef repo access Most with knife keys All editing the same 2 files with every change...

136 Env: Solutions?

137 Env: Solutions? Go with it and hope for the best?

138 Env: Solutions? Go with it and hope for the best? Don t use environments?

139 Env: Solutions? Go with it and hope for the best? Don t use environments? Write a totally new workflow?

140 Env: Solutions? Go with it and hope for the best? Don t use environments? Write a totally new workflow? Tweak the existing one with some tooling?

141 Env: Solution!

142 Env: Solution! Tweak with some tooling!

143 Env: Solution! Tweak with some tooling! Presenting knife-spork [9]...

144 Spork: Workflow

145 Spork: Workflow Wrapper around standard environments workflow

146 Spork: Workflow Wrapper around standard environments workflow check - cookbook versioning

147 Spork: Workflow Wrapper around standard environments workflow check - cookbook versioning bump - increment version component

148 Spork: Workflow Wrapper around standard environments workflow check - cookbook versioning bump - increment version component upload - upload and freeze

149 Spork: Workflow Wrapper around standard environments workflow check - cookbook versioning bump - increment version component upload - upload and freeze promote - set env constraints

150 Spork: Check $> knife spork check apache2 Checking versions for cookbook apache2... Current local version: Remote versions (Max. 5 most recent only): *1.0.6, frozen 1.0.5, frozen <snip> DANGER: Your local cookbook has same version number as the starred version above! Please bump your local version or you won't be able to upload.

151 Spork: Bump $> knife spork bump apache2 <major minor patch manual> Bumping patch level of the apache2 cookbook from to 1.0.7

152 Spork: Upload $> knife spork upload apache2 Uploading and freezing apache2 [1.0.7] upload complete

153 Spork: Promote $> knife spork promote foo php Adding version constraint php = Saving changes into foo.json Promotion complete! Please remember to upload your changed Environment file to the Chef Server. --- $> knife spork promote foo php --remote Adding version constraint php = Saving changes into foo.json Uploading foo to server

154 Spork

155 Spork Worked well, avoided the issues it was designed for

156 Spork Worked well, avoided the issues it was designed for Subsequently evolved

157 Spork Worked well, avoided the issues it was designed for Subsequently evolved A lot of input & work came from Devs

158 Spork Worked well, avoided the issues it was designed for Subsequently evolved A lot of input & work came from Devs Organic evolution

159 Spork Worked well, avoided the issues it was designed for Subsequently evolved A lot of input & work came from Devs Organic evolution Open sourced, of course

160 Spork: Evolution

161 Spork: Evolution Safety Checks

162 Spork: Evolution Safety Checks Extra Features

163 Spork: Safety Checks

164 Spork: Safety Checks Before promoting, check version is uploaded...

165 Spork: Safety Checks $> knife spork promote php --remote <snip> WARNING: It looks like you have multiple cookbook paths defined so I can't tell if you're running inside a git repo. Checking that php version exists on the server before promoting (any error means it hasn't been uploaded yet)... ERROR: The object you are looking for could not be found Response: Cannot find a cookbook named php with version

166 Spork: Safety Checks Before promoting, check version is uploaded... Check if you re promoting changes to more than you thought...

167 Spork: Safety Checks WARNING: You're about to promote changes to several cookbooks: WARNING: ganglia: = changed to = installerz: = changed to = php: = changed to = Are you sure you want to continue? (Y/N) N You said no, so I'm done here. Would you like to reset your local development.json to match the server?? (Y/N) Y <snip> development.json reset.

168 Spork: Features

169 Spork: Features Default Environments

170 Spork: Features Default Environments Git support

171 Spork: Features Default Environments Git support Chat notifications + Gist (upload and promote)

172 Spork: Features Default Environments Git support Chat notifications + Gist (upload and promote) IRCCat

173 Spork: Features Default Environments Git support Chat notifications + Gist (upload and promote) IRCCat Hipchat (courtesy of Secondmarket)

174 Features: Chat Notifications [19:43:46] <irccat> CHEF: pmcdonnell uploaded and froze cookbook immount version [19:44:00] <irccat> CHEF: pmcdonnell uploaded environment production [19:44:01] <irccat> CHEF: pmcdonnell uploaded environment development

175 Spork: Features Default Environments Git support Chat notifications + Gist (upload and promote) IRCCat Hipchat (courtesy of Secondmarket) Graphite (on promote)

176 Features: Graphite

177 Spork: Features Default Environments Git support Chat notifications + Gist (upload and promote) IRCCat Hipchat (courtesy of Secondmarket) Graphite (on promote) Foodcritic (on upload)

178 Features: Foodcritic $> knife spork upload system <snip> Lint checking system... ERROR: Lint check failed. Halting upload. ERROR: Lint check output: ERROR: ETSY003: Execute resource used to run curl or wget commands: /Users/jcowie/dev/etsy/chef/cookbooks/ system/recipes/dev-ssl.rb:41

179 Use the Source! tiny.cc/velocity2012

180 We re hiring! BoF on Tuesday or just come and say Hi :)