BRING THE NOISE! MAKING SENSE OF A HAILSTORM OF METRICS. Abe Jon

Transcription

1 BRING THE NOISE! MAKING SENSE OF A HAILSTORM OF METRICS Abe Jon

2 Ninety minutes is a long time. This talk: ~10 ~25 ~30 ~10 ~15 - motivations - skyline - oculus - demo! - questions

3 Ninety minutes is a long time. This talk: But we have some sweet stuff to show you. ~10 ~25 ~30 ~10 ~15 - motivations - skyline - oculus - demo! - questions

4 Background and Motivations

5

6 1.5 billion page views $117 million of goods sold 950 thousand users

7 1.5 billion page views $117 million of goods sold 950 thousand users (in december 12)

8 We practice continuous deployment.

9 de ploy /diˈploi/ Verb To release your code for the world to see, hopefully without breaking the Internet

10 250+ committers. Everyone deploys.

11 Day one: DEPLOY

12

13 30+ DEPLOYS A DAY (~8 commits per deploy!)

14 30 deploys a day? Is that safe?

15 We optimize for quick recovery by anticipating problems...

16 ...instead of fearing human error.

17 Can t fix what you don t measure! - W. Edwards Deming

18 not homemade homemade! Ganglia graphite Nagios Skyline StatsD Supergrep Oculus

19 Real time error logging Text

20 Not all things that break throw errors. - Oscar Wilde

21 StatsD

22 StatsD::increment( foo.bar )

23 If it moves, graph it!

24 If it moves, graph it! we would graph them

25 If it doesn t move, graph it anyway (it might make a run for it)

26 DASHBOARDS!

27 [ , 20] [ , 20] [ , 20] [ , 60] [ , 20] [ , 20] [ , 20] [ , 20] [ , 20] [ , 20] [ , 20]

28 DASHBOARDS! x 250,000

29

30 lol nagios

31 ...but there are also unknown unknowns - there are things we do not know we don t know.

32 Unknown anomalies

33 Unknown correlations

34 Kale.

35 Kale: - leaves - green stuff

36 Kale: - leaves SKYLINE - green stuff OCULUS

37 Q). How do you analyze a timeseries for anomalies in real time?

38 A). Lots of HTTP requests to Graphite s API!

39 Q). How do you analyze a quarter million timeseries for anomalies in real time?

40 SKYLINE

41 SKYLINE

42 A real time anomaly detection system

43 Real time?

44 Kinda.

45 StatsD Ten second resolution

46 Ganglia One minute resolution

47 Best case: ~ 10s ( ~ 1min

48 Takes about 90 seconds with our throughput. (

49 Still faster than you would have discovered it otherwise. (

50 Memory > Disk

51

52 Q). How do you get a quarter million timeseries into Redis on time?

53 STREAM IT!

54 Graphite s relay agent original graphite backup graphite

55 Graphite s relay agent [statsd.numstats, [ , 73421]] [statsd.numstats, [ , 82345]] [statsd.numstats, [ , 80611]] pickles original graphite backup graphite

56 Graphite s relay agent [statsd.numstats, [ , 73421]] [statsd.numstats, [ , 82345]] [statsd.numstats, [ , 80611]] pickles original graphite skyline

57 We import from Ganglia too.

58 Storing timeseries

59 Minimize I/O Minimize memory

60 redis.append() - Strings - Constant time - One operation per update

61 JSON?

62 => get statsd.numstats [ , 51],

63 => get statsd.numstats [ , 51], [ , 23],

64 => get statsd.numstats [ , 51], [ , 23], [ , 45],

65 OVER HALF CPU time spent decoding JSON

66 [1,2]

67 Stuff we care about [ 1, 2 ] Extra junk

68 MESSAGEPACK

69 MESSAGEPACK A binary-based serialization protocol

70 Things we care about \x93\x01\x02 Array size (16 or 32 bit big endian integer)

71 Things we care about \x93\x01\x02 \x93\x02\x03 Array size (16 or 32 bit big endian integer)

72 CUT IN HALF Run Time + Memory Used

73 ROOMBA.PY CLEANS THE DATA

74 Wait...you wrote this in Python?

75 Great statistics libraries Not fun for parallelism

76 The Analyzer Assign Redis keys to each process Process decodes and analyzes

77 The Analyzer Anomalous metrics written as JSON setinterval() retrieves from front end

78

79 What does it mean to be anomalous?

80 Consensus model

81 Implement everything you can get your hands on

82 Basic algorithm: A metric is anomalous if its latest datapoint is over three standard deviations above its moving average.

83 Grubb s test, ordinary least squares

84 Histogram binning

85 Four horsemen of the modelpocalypse

86 1. Seasonality 2. Spike influence 3. Normality 4. Parameters

87 Anomaly?

88 Nope.

89 Spikes artificially raise the moving average Bigger moving average Text Anomaly Anomaly missed :( detected (yay!)

90 Real world data doesn t necessarily follow a perfect normal distribution.

91 Too many metrics to fit parameters for them all!

92 A robust set of algorithms is the current focus of this project.

93 Q). How do you analyze a quarter million timeseries for correlations?

94 OCULUS

95 Image comparison is expensive and slow

96 Use raw timeseries instead of raw graphs [[975, ], [643, ], [750, ], [992, ], [580, ], [586, ], [649, ], [548, ], [901, ], [633, ]]

97 HARD PROBLEMS Naming Things Cache Invalidation Numerical Comparison?

98 HARD PROBLEMS Naming Things Cache Invalidation Numerical Comparison?

99 Euclidian Distance

100 Dynamic Time Warping (helps with phase shifts)

101 We ve solved it!

102 O(N 2 )

103 O(N 2 ) x 250k

104 Too slow!

105 doesn t

106 No need to run DTW on all 250k.

107 Discard obviously dissimilar metrics.

108 Shape Description Alphabet sharpdecrement flat increment sharpincrement flat flat shapdecrement

109 Shape Description Alphabet (normalization step) sharpdecrement flat increment sharpincrement flat flat shapdecrement

110

111 Search for shape description fingerprint in Elasticsearch

112 Run DTW on results as final polish

113 O(N 2 ) on ~10k metrics

114 Still too slow.

115 Fast DTW - O(N) coarsen project refine

116 Elasticsearch Details Phrase search for first pass scores across shape description fingerprints

117 Elasticsearch Details Phrase search for first pass scores across shape description fingerprints Custom FastDTW and euclidian distance plugins to score across the remaining filtered timeseries

118 Elasticsearch Structure { :id => statsd.numstats, :fingerprint => sdec inc sinc sdec, :values => " " }

119 Mappings Specify tokenizers Untouched fields

120 First pass query :match => { :fingerprint => { :query => sdec inc sinc sdec inc, :type => "phrase", } } :slop => 20 shape description fingerprint

121 Refinement query {:custom_score => { :query => <first_pass_query>, :script => "oculus_dtw", raw timeseries :params => { :query_value => , :query_field => "values.untouched", }, }

122 KALE StatsD Graphite Ganglia Elasticsearch Skyline Flask Sinatra Resque

123 Populating Elasticsearch

124 resque workers ES Index

125 Too slow to update and search

126 Webapp New Index Last Index

127 Sinatra frontend Queries ES Renders results

128 Collections

129 devops <3

130

131 Special thanks to: Dr. Neil Gunther, PerfDynamics Dr. Brian Whitman, Echonest Burc Arpat, Facebook Seth Walker, Etsy Rafe Colburn, Etsy Mike Rembetsy, Etsy John Allspaw, Etsy

132 Thanks!