The Day the DNS Died

Transcription

1 The Day the DNS Died Jeremy Blosser, Principal Operations Engineer 1

2 Introduction SparkPost, aka Message Systems, is a high-volume, transactional software and services vendor. 2

3 Introduction SparkPost, aka Message Systems, is a high-volume, transactional software and services vendor. We send a lot of Over 30% of the world s non-spam is sent using our software. 15B messages/month sent via our cloud offering. 3

4 Introduction SparkPost, aka Message Systems, is a high-volume, transactional software and services vendor. We send a lot of Over 30% of the world s non-spam is sent using our software. 15B messages/month sent via our cloud offering. That requires a lot of DNS: 8,000 queries/second. 20Mb/s+ sustained traffic just for DNS queries. Several different resolution paths. 4

5 Introduction But DNS is easy, right? 5

6 Introduction But DNS is easy, right? 6

7 Outline Introduction Previous DNS Design(s) May 2017 Outage New DNS Design Lessons Learned / Remembered References Questions? 7

8 Previous DNS Design(s) 8

9 Version 1, Centralized Internal Resolver Cluster 9

10 Version 1, Centralized Internal Resolver Cluster 10

11 Version 1, Centralized Internal Resolver Cluster 11

12 Version 1, Centralized Internal Resolver Cluster 12

13 Version 1, Centralized Internal Resolver Cluster 13

14 Version 1, Centralized Internal Resolver Cluster 14

15 Version 1, Centralized Internal Resolver Cluster 15

16 Version 1, Centralized Internal Resolver Cluster 16

17 Version 1, Centralized Internal Resolver Cluster 17

18 Version 1, Centralized Internal Resolver Cluster 18

19 Version 1, Centralized Internal Resolver Cluster 19

20 Version 1, Centralized Internal Resolver Cluster 20

21 Version 2, AWS VPC Resolver 21

22 Version 2, AWS VPC Resolver 22

23 Version 1.5, Centralized Internal Resolver Cluster 23

24 Version 1.5, Centralized Internal Resolver Cluster 24

25 Version 3.14, Centralized Internal Resolver Cluster 25

26 May

27 May 2017 Outage A day like any other day until... 27

28 May 2017 Outage A day like any other day until... 28

29 May 2017 Outage A day like any other day until... 29

30 May 2017 Outage A day like any other day until... 30

31 May 2017 Outage A day like any other day until... 31

32 May 2017 Outage A day like any other day until... 32

33 May 2017 Outage DNS Cluster Aggregate CPU 33

34 May 2017 Outage MTA Cluster Aggregate CPU 34

35 May 2017 Outage MTA Cluster Aggregate CPU 35

36 May 2017 Outage Mail Delivery (one customer) 36

37 May 2017 Outage (Near) Total Impact Sending mail 37

38 May 2017 Outage (Near) Total Impact Sending mail - (most) customer mail injection not impacted 38

39 May 2017 Outage (Near) Total Impact Sending mail - (most) customer mail injection not impacted App/DB traffic 39

40 May 2017 Outage (Near) Total Impact Sending mail - (most) customer mail injection not impacted App/DB traffic Metrics 40

41 May 2017 Outage (Near) Total Impact Sending mail - (most) customer mail injection not impacted App/DB traffic Metrics Config management (partial) 41

42 May 2017 Outage (Near) Total Impact Sending mail - (most) customer mail injection not impacted App/DB traffic Metrics Config management (partial) Admin logins 42

43 May 2017 Outage (Near) Total Impact Sending mail - (most) customer mail injection not impacted App/DB traffic Metrics Config management (partial) Admin logins 43

44 May 2017 Outage 44

45 May 2017 Outage Diagnosing Blind 45

46 May 2017 Outage Diagnosing Blind Lack of insight into our DNS Unable to reach support systems 46

47 May 2017 Outage Diagnosing Blind Lack of insight into our DNS Unable to reach support systems Is it throttling (again)? Is it capacity (again)? 47

48 May 2017 Outage Diagnosing Blind Lack of insight into our DNS Unable to reach support systems Is it throttling (again)? - Central forward to VPC Resolver Immediately overrun Is it capacity (again)? - Add capacity Immediately affected 48

49 May 2017 Outage Mitigation Repoint individual instances to VPC Resolver - Edit resolv.conf 49

50 May 2017 Outage resolv.conf Limited to 3 entries Always tried top to bottom Limited practical retry Read on app startup - Changes require restarts 50

51 May 2017 Outage Mitigation Repoint individual instances to VPC Resolver - Edit resolv.conf, with restarts - Provided breathing room Main resolver cluster recovered as load was removed App tier recovery: 2 hours Major customer mail recovery: 4-5 hours Time to full recovery: 7 hours 51

52 May 2017 Outage Mitigation Webhook SQS Queued Messages 52

53 May 2017 Outage Diagnosis Asymmetric DNS packet flow - Tcpdump - AWS Network Flow Logs Average 300 responses per 5000 queries (94% failure) 53

54 May 2017 Outage The Cause? 54

55 May 2017 Outage The Cause? Connection Tracking 55

56 May 2017 Outage The Cause? [Undocumented] Connection Tracking 56

57 May 2017 Outage The Cause? [Undocumented] Connection Tracking 57

58 May 2017 Outage After Action Conclusions Incident response process was functional Ability to respond via the process was compromised Limits of iteration New DNS design required 58

59 New DNS Design 59

60 New DNS Design Requirements Resolve all needed name sources Modifiable without changing resolv.conf Avoid throttling No conntrack Multi cluster / isolate components Distributed across resolver clusters Minimize latency Effective caching Respect TTLs Increase DNS profiling and monitoring 60

61 New DNS Design 61

62 New DNS Design Network Configuration Dedicated VPC for isolation Open Security Groups with stateless ACLs Separate resolver clusters to isolate impacts Query traffic favors same Availability Zone 62

63 New DNS Design 63

64 New DNS Design Resolver (Unbound) Configuration Instance and service tuning Multiple network interfaces per instance Multiple IPs per interface serve-expired enabled 64

65 New DNS Design OS Configuration Two local cache services routes to resolvers in same AZ routes to resolvers in other AZs dnsmasq Configuration Max concurrency Max cache size /etc/resolv.conf points to: direct resolver IP 65

66 New DNS Design 66

67 Lessons Learned / Remembered AWS main service model is pull, not push 67

68 Lessons Learned / Remembered AWS main service model is pull, not push Not all cloud provider limits are apparent - make sure they understand your business 68

69 Lessons Learned / Remembered AWS main service model is pull, not push Not all cloud provider limits are apparent - make sure they understand your business Instrument your support services - and protect them from each other 69

70 Lessons Learned / Remembered AWS main service model is pull, not push Not all cloud provider limits are apparent - make sure they understand your business Instrument your support services - and protect them from each other resolv.conf is not agile - not even eventually consistent 70

71 Lessons Learned / Remembered AWS main service model is pull, not push Not all cloud provider limits are apparent - make sure they understand your business Instrument your support services - and protect them from each other resolv.conf is not agile - not even eventually consistent Iteration doesn t solve it all 71

72 Lessons Learned / Remembered It s always a DNS problem 72

73 Lessons Learned / Remembered It s always a DNS problem - unless it s a firewall problem 73

74 References pc.pdf work-security.html#security-group-connection-tracking ns.html 74

75 Questions? 75