WELCOME

Transcription

1

2 WELCOME

3 Josh

4 Josh Kalderimis

5 @j2h

6 github.com/joshk

7

8

9 #38ish

10

11 Wellington

12

13 NEW ZEALAND

14

15

16

17

18

19

20 Amsterdam

21

22

23

24

25 but now...

26

27

28

29

30 before we get going...

31

32

33 -35

34 -35 WAT!!

35 Desconstruindo Travis

36 LOGGING

37 METRICS

38 MONITORING

39 part theory

40 part code

41 part internal tools

42 part mustache

43 ?

44

45

46

47

48

49 IT S NOT HOW GOOD YOUR APP IS, IT S HOW GOOD YOU WANT IT TO BE

50

51

52

53 1,000,000,000 rps

54 newrelic, airbrake, errbit, exception notifier, metriks, statsd, graphite, and librato

55 internal tools

56 services are your friends

57

58

59 Travis CI

60 Travis CI Distributed, continuous integration for the open-source community

61 we run your tests for you!

62 free

63 open source

64 another piece of infrastructure!

65 we are for builds what RubyGems is for libraries

66 19,300+ projects

67 7,000+ builds a day

68 496,068 build requests

69 1,459,846 total test runs

70 11+ languages

71

72 started for fun

73 instant

74 live

75 modern

76 hackable

77 little did you know...

78 Travis united us

79

80

81

82 sven mustache pic

83 sven mustache

84

85

86

87 BUT

88 NO MORE MUSTACHE

89 INSTEAD

90 SAY HELLO TO THE ONSIE!

91

92

93 FOCUS JOSH!!!

94 Travis Architecture

95

96

97

98

99 mini S.O.A.

100 we share resources

101 4 deployable apps

102 4 different environments

103 Org : staging production

104 Org : staging production Pro : staging production

105 16 different deployments

106 measuring and monitoring is a challenge

107 but all the more important

108

109 LOGGING

110 simple

111 concise

112 Started GET "/" for at :28: Processing by HomeController#index as HTML Rendered text template within layouts/application (0.0ms) Rendered layouts/_assets.html.erb (2.0ms) Rendered layouts/_top.html.erb (2.6ms) Rendered layouts/_about.html.erb (0.3ms) Rendered layouts/_google_analytics.html.erb (0.4ms) Completed 200 OK in 79ms (Views: 78.8ms ActiveRecord: 0.0ms)

113 one line

114 Request Type

115 Request Type URL

116 Request Type URL Format

117 Request Type URL Format Action

118 Request Type URL Format Action Status

119 Request Type URL Format Action Status Total Duration

120 Request Type URL Format Action Status Total Duration View Duration

121 Request Type URL Format Action Status Total Duration View Duration DB Duration

122 GET /jobs/ json format=json action=jobs#show status=200 duration=58.33 view=40.43 db=15.26

123 GET /jobs/ json format=json action=jobs#show status=200 duration=58.33 view=40.43 db=15.26

124 one line

125 lograge

126 non web apps?

127 use different log levels

128 debug

129 info

130 error

131 DEBUG LOTS

132 info a bit

133 error exceptions

134 @marcelokanzaki LOG ALL THE THINGS

135 Travis Hub

136 1,100 rpm

137 Build Finished

138 Notifications

139 in background threads

140 E [githubcommitstatus] Could not update the PR status on d18173d510eadf063917db62956f25edab9f70c1 (the server responded with status 40 I Travis::Task::GithubCommitStatus#run for #<Build id= > I Travis::Event::Handler::GithubCommitStatus#notify(build:finished) for #<Build id=22 I Travis::Hub::Handler::Job#update for #<Job id=" "> I Travis::Task::Pusher#run for #<Job id= > I Travis::Event::Handler::Pusher#notify(job:test:finished) for #<Job::Test id= > I Travis::Task::Pusher#run for #<Build id= > I Travis::Event::Handler::Pusher#notify(build:finished) for #<Build id= > I Travis::Event::Handler:: #notify(build:finished) for #<Build id= > I Travis::Event::Handler::Irc#notify(build:finished) for #<Build id= > I Travis::Event::Handler::Webhook#notify(build:finished) for #<Build id= > I Travis::Event::Handler::Campfire#notify(build:finished) for #<Build id= > I [github] Successfully commented on I Travis::Task::Github#run for #<Build id= > I Travis::Event::Handler::Github#notify(build:finished) for #<Build id= > I Travis::Task::GithubCommitStatus#run for #<Build id= >

141 E [githubcommitstatus] Could not update the PR status on d18173d510eadf063917db62956f25edab9f70c1 (the server responded with status 40 I Travis::Task::GithubCommitStatus#run for #<Build id= > I Travis::Event::Handler::GithubCommitStatus#notify(build:finished) for #<Build id=22 I Travis::Hub::Handler::Job#update for #<Job id=" "> I Travis::Task::Pusher#run for #<Job id= > I Travis::Event::Handler::Pusher#notify(job:test:finished) for #<Job::Test id= > I Travis::Task::Pusher#run for #<Build id= > I Travis::Event::Handler::Pusher#notify(build:finished) for #<Build id= > I Travis::Event::Handler:: #notify(build:finished) for #<Build id= > I Travis::Event::Handler::Irc#notify(build:finished) for #<Build id= > I Travis::Event::Handler::Webhook#notify(build:finished) for #<Build id= > I Travis::Event::Handler::Campfire#notify(build:finished) for #<Build id= > I [github] Successfully commented on I Travis::Task::Github#run for #<Build id= > I Travis::Event::Handler::Github#notify(build:finished) for #<Build id= > I Travis::Task::GithubCommitStatus#run for #<Build id= >

142 I Travis::Event::Handler::Irc #notify(build:finished) for #<Build id= >

143 I [class] [method + extra info] for #<[model] id=[id]>

144 E [class] (error message)

145 Travis Worker

146 5 VMs per worker

147 5 jobs per worker

148 goal is different

149 easy to see what s going on

150 live

151 I [ruby-2:worker] starting job slug:nedomas/ta id: I [ruby-2:build:remote] about to run I [ruby-2:worker:virtual_box] ruby-2 started with process id : 5772 I [ruby-2:shell:session] starting ssh session to : I [ruby-2:build:job:test:ruby] about to run I [ruby-2:build:job:test:ruby] about to perform I [ruby-2:build:job:test:ruby] about to chdir I [ruby-2:build:job:test:ruby] about to export I [ruby-2:build:job:test:ruby] about to checkout I [ruby-2:build:job:test:ruby] about to setup I [ruby-2:build:job:test:ruby] about to run_command(:install, "bundle

152 a more verbose form of info

153 ruby-2 I [ruby-2:worker] starting job slug:nedomas/ta id: I [ruby-2:build:remote] about to run I [ruby-2:worker:virtual_box] ruby-2 started with process id : 5772 I [ruby-2:shell:session] starting ssh session to : I [ruby-2:build:job:test:ruby] about to run I [ruby-2:build:job:test:ruby] about to perform I [ruby-2:build:job:test:ruby] about to chdir I [ruby-2:build:job:test:ruby] about to export I [ruby-2:build:job:test:ruby] about to checkout I [ruby-2:build:job:test:ruby] about to setup I [ruby-2:build:job:test:ruby] about to run_command(:install, "bundle

154 tail -f log/worker.log grep "\[ruby-2:"

155 we can improve it further!

156 @marcelokanzaki CONSOLIDATE ALL THE LOGGING

157 rsync to a single server

158 Graylog2 (

159 ruby / java elasticsearch

160 papertrail (

161

162 archiving notifications webhooks

163 we <3 papertrail

164

165 EXCEPTIONS

166 checking logs for errors is no fun

167 use an exception app or service or lib

168 exception_notifier (

169 airbrake (

170 errbit (

171 haystack

172

173

174

175 errors as a todo list

176 errors as a todo list

177 errors as a natural part of our app

178 don t ignore them

179 but don t fret over them either

180 your customer

181 support s

182 teh twitters

183 users are a great notification system

184 METRICS

185

186

187 NewRelic is powerful

188

189

190 AMQP subscriptions

191 using NewRelic for messaging?

192 WAT!!

193 def handle... end new_relic :handle

194 proxy

195 controller instrumentation

196 trace

197

198

199 NewRelic isn t just for web apps

200

201 METRICS

202 collecting

203 collecting sending

204 collecting sending aggregating

205 collecting sending aggregating visualizing

206

207 Collecting

208 alias_method

209 wraps methods

210 times them

211 and sends the data to NewRelic

212 ActiveSupport Notifications

213 ActiveSupport Notifications

214 ActiveSupport Notifications

215 publish `events`

216 instrument `code`

217 subscribe to both

218 one event type

219 multiple using a regex

220 publish one time events

221 instrument yield a block and time it

222

223 we use it to publish three events

224 received

225 completed

226 failed

227 meta programming

228 def handle... end instrument :handle

229 ActiveSupport Notifications is metrics agnostic

230 it s general purpose

231 it s all about the subscriber

232

233 Sending and Aggregating

234 statsd

235 statsd

236 node.js

237 udp

238 pusher.received:1 c

239 pusher.completed:320 ms

240 echo "pusher.completed:320 ms" nc server port

241 statsd-ruby

242 statsd = Statsd.new(host, port)

243 statsd.increment 'pusher' statsd.timing 'pusher', 320

244 155 lines of code

245 metriksd

246 Eric Lindvall (Papertrail)

247 ruby eventmachine

248 metriks

249 counter = Metriks.counter('pusher') counter.increment

250 timer = Metriks.timer('pusher') timer.time do work end

251 part aggregator, part sender

252 reporters

253 logs

254 T13:19:00+00:00 app[web.1]: metriks: time= name=active_record.reads type=timer count= one_minute_rate= five_minute_rate= fifteen_minute_rate= mean_rate= min= max= mean= stddev= median= th_percentile=

255 AS Notifications Metriks

256 a quick way to get started

257 chokes up our logs with metrics

258 AS Notifications Metriks?

259 Vizualising

260 graphite (

261

262 librato (

263

264

265

266

267 tasseo (

268

269 AS Notifications Metriks Webhook App

270 AS Notifications collecting Metriks aggregating & sending visualizing Webhook App aggregating & sending

271 we can do better!

272 metriksd_reporter and metriksd

273 AS Notifications collecting Metriks sending visualizing Metriksd aggregating & sending

274

275 Internal Tools

276 Travis Admin

277 sinatra

278 bootstrap

279 @konstantinhaase

280

281

282

283 EVENT MONITOR

284

285 @drogus

286 Ember.js

287 Redis Pub/Sub

288

289

290

291

292 Travis Instrumentation

293 ActiveSupport Notifications

294 travis/support

295 open source the Event Monitor?

296

297 IT S NOT HOW GOOD YOUR APP IS, IT S HOW GOOD YOU WANT IT TO BE

298

299

300 1,000,000,000 rps

301 newrelic, papertrail, metriks, and librato

302 internal tools

303 services are your friends

304

305 @travisci irc.freenode.net#travis

306

307

308 THANK YOU

309 Questions?