Avoiding alerts overload from microservices

Size: px

Start display at page:

Download "Avoiding alerts overload from microservices"

Herbert Clarke
5 years ago
Views:

1 Avoiding alerts overload from microservices Sarah Wells Principal Engineer, Financial

5 Knowing when there s a problem isn t

6 You only want an alert when you need to take action

7 @sarahjwells Hello

10 1

11 1 2

12 2 1 3

14 @sarahjwells Monitoring this system

15 @sarahjwells Microservices make it worse

16 microservices (n,pl): an efficient device for transforming business problems into distributed transaction

17 The services *themselves* are

18 @sarahjwells There s a lot of complexity around them

19 @sarahjwells Why do they make monitoring harder?

20 @sarahjwells You have a lot more services

21 99 functional microservices 350 running

22 52 non functional services 218 running

23 @sarahjwells That s 568 separate services

24 If we checked each service every

25 @sarahjwells 817,920 checks per day

26 @sarahjwells What about system checks?

27 @sarahjwells 16,358,400 checks per day

28 One-in-a-million issues would hit us 16 times every

29 Running containers on shared VMs reduces this to 92,160 system checks per

30 @sarahjwells For a total of 910,080 checks per day

31 @sarahjwells It s a distributed system

32 @sarahjwells Services are not independent

38 You have to change how you think about

39 How can you make it better?

40 1. Build a system you can

41 @sarahjwells The basic tools you need

42 @sarahjwells Log aggregation

44 Logs go missing or get delayed more

45 Which means log based alerts may miss

46 @sarahjwells Monitoring

48 @sarahjwells Limitations of our nagios integration

49 @sarahjwells No service-level view

50 Default checks included things we couldn t

51 @sarahjwells A new approach for our container stack

52 @sarahjwells We care about each service

54 @sarahjwells We care about each VM

56 @sarahjwells We care about unhealthy instances

57 Monitoring needs aggregating

58 @sarahjwells SAWS

59 Built by Silvano Dossan See our Engine room blog:

60 "I imagine most people do exactly what I do - create a google filter to send all Nagios s straight to the

61 "Our screens have a viewing angle of about 10

62 "It never seems to show the page I

63 @sarahjwells Code at:

64 @sarahjwells Dashing

67 @sarahjwells Graphing of metrics

73 The things that make those tools

74 Effective log aggregation needs a way to find all related

75 Transaction ids tie all microservices together

76 @sarahjwells Make it easy for any language you use

77 @sarahjwells

78 Services need to report on their own

79 The FT healthcheck standard GET health

80 The FT healthcheck standard GET health returns 200 if the service can run the healthcheck

81 The FT healthcheck standard GET health returns 200 if the service can run the healthcheck each check will return "ok": true or "ok": false

84 Knowing about problems before your clients

85 Synthetic requests tell you about problems early

86 2. Concentrate on the stuff that

87 It s the business functionality you should care

89 We care about whether content got published successfully

91 When people call our APIs, we care about speed

92 we also care about errors

94 But it's the end-to-end that matters

95 If you just want information, create a dashboard or report

96 Checking the services involved in a business

97 / health?categories=lists-publish

100 @sarahjwells 3. Cultivate your alerts

101 Make each alert great

102 Splunk Alert: PROD - MethodeAPIResponseTime5MAlert Business Impact The methode api server is slow responding to requests. This might result in articles not getting published to the new content platform or publishing requests timing

103 Splunk Alert: PROD - MethodeAPIResponseTime5MAlert Business Impact The methode api server is slow responding to requests. This might result in articles not getting published to the new content platform or publishing requests timing

104 Technical Impact The server is experiencing service degradation because of network latency, high publishing load, high bandwidth utilization, excessive memory or cpu usage on the VM. This might result in failure to publish articles to the new content

105 Splunk Alert: PROD Content Platform Ingester Methode Publish Failures Alert There has been one or more publish failures to the Universal Publishing Platform. The UUIDs are listed below. Please see the run book for more information. _time transaction_id uuid Mon Oct 12 07:43: tid_pbueyqnsqe

106 Splunk Alert: PROD Content Platform Ingester Methode Publish Failures Alert There has been one or more publish failures to the Universal Publishing Platform. The UUIDs are listed below. Please see the run book for more information. _time transaction_id uuid Mon Oct 12 07:43: tid_pbueyqnsqe

107 Splunk Alert: PROD Content Platform Ingester Methode Publish Failures Alert There has been one or more publish failures to the Universal Publishing Platform. The UUIDs are listed below. Please see the run book for more information. _time transaction_id uuid Mon Oct 12 07:43: tid_pbueyqnsqe

108 Splunk Alert: PROD Content Platform Ingester Methode Publish Failures Alert There has been one or more publish failures to the Universal Publishing Platform. The UUIDs are listed below. Please see the run book for more information. _time transaction_id uuid Mon Oct 12 07:43: tid_pbueyqnsqe

109 Make sure you can't miss an alert

110 Ops Cops keep an eye on our

111 @sarahjwells Use the right communication channel

112 @sarahjwells It s not

113 Slack integration

114

115 Support isn t just getting the system

116

117 @sarahjwells You build it, you run it?

118 @sarahjwells Review the alerts you get

119 If it isn't helpful, make sure you don't get sent it again

120 See if you can improve it om/

121 @sarahjwells When you didn't get an alert

122 What would have told you about this?

123 @sarahjwells

124 Setting up an alert is part of fixing the problem code test

125 System boundaries are more difficult Severin.stalder [CC BY-SA 3.0 ( via Wikimedia Commons

126 Make sure you would know if an alert stopped

127 Add a unit test public void shouldincludetriggerwordsforpublishfailurealertinsplunk() { }

128 Deliberately break things

129 Chaos snail

130 It s going to change: deal with

131 Out of date information can be worse than

132 @sarahjwells Automate updates where you can

133 @sarahjwells Find ways to share what s changing

134 In summary: to avoid alerts

135 1 Build a system you can support

136 2 Concentrate on the stuff that matters

137 Cultivate your alerts 3

138 A microservice architecture lets you move

139 But there s an associated operational

140 Make sure it s a cost you re willing to

141 @sarahjwells Thank you

Excel 2010 Formulas Not Work In 2003 Copying Down Correctly

Excel 2010 Formulas Not Work In 2003 Copying Down Correctly Invest a few minutes and learn how to correctly use relative and absolute cell How relative and absolute cell references work in conditional