Traffic Steering using RUM DNS. Abhijeet Rastogi Senior Linkedin (Edge Performance & Traffic)

Transcription

1 Traffic Steering using RUM DNS Abhijeet Rastogi Senior Linkedin (Edge Performance & Traffic)

2 My Team Public DNS CDN Operations Load balancer & reverse proxy at PoPs and DCs

3 Largest Professional Network 500M Members Countries signups/second

4 Problem Statement SiteSpeed

5 Problem Statement User Engagement SiteSpeed

6 Problem Statement SiteSpeed

7 Problem Statement User Engagement Revenue SiteSpeed

8 Agenda CDN strategy POP Strategy RUM Steering Tooling

9 Agenda CDN strategy POP Strategy RUM Steering Tooling

10 LinkedIn CDN Stats >80% >85% Cache Hit Rate Origin bandwidth offloaded

11 LinkedIn CDN partners Global China

12 Multi CDN Strategy Performance: Having multiple options allows comparing and selecting the best CDN to each network in each geography in real time Redundancy: CDNs can be pulled out of rotation for planned & unplanned events Benefits Capacity: Traffic spikes of all shapes and sizes can be handled

13 Multi CDN Strategy (India CDN Performance)

14 Multi CDN Strategy (US CDN Performance)

15 Agenda CDN strategy POP Strategy RUM Steering Tooling

16 Point of Presence (POP)

17 POP (Point of Presence) Typical DataCenter LB/RP: TCP+SSL termination Users Internet Application servers

18 POP (Point of Presence) Typical DataCenter Users LB/RP: TCP+SSL termination Application servers

19 POP (Point of Presence) Typical DataCenter Users LB/RP: TCP+SSL termination Application servers

20 POP (Point of Presence) Typical DataCenter LB/RP: TCP+SSL termination Users Total: 2000ms Application servers

21 POP (Point of Presence) Decouple LBs LB/RP: TCP+SSL termination Users Application servers TCP Window size, HTTP keepalive Internet Internet

22 POP (Point of Presence) Establish Backbone LB/RP: TCP+SSL termination Users Application servers TCP Window size, HTTP keepalive Internet Backbone

23 POP (Point of Presence) LBs become POP POP Users Application servers TCP Window size, HTTP keepalive Internet Backbone

24 POP (Point of Presence) LBs become POP Users POP Application servers

25 POP (Point of Presence) LBs become POP Users POP Application servers

26 POP (Point of Presence) LBs become POP Users POP Application servers

27 POP (Point of Presence) LBs become POP Users POP Total: 1100ms Application servers

28 POP (Point of Presence) LBs become POP Users POP 900ms gain! Application servers

29 POP (Point of Presence) Cost effective $ $ $ $

30 LinkedIn POP Footprint Seattle Dublin San Jose Chicago Frankfurt Ashburn Los Angeles London Miami Hong Kong Mumbai Dallas Sao Paulo 15 POP Locations Singapore Sydney

31 LinkedIn DataCenter Footprint Oregon Virginia Texas Singapore

32 POP Benefit Mumbai PoP Singapore & London PoP Sau Paulo PoP Mumbai & Dublin PoP Singapore PoP

33 Agenda CDN strategy POP Strategy RUM Steering Tooling

34 Multi DNS Root Servers $ dig +short +noshort NS linkedin.com sort linkedin.com IN NS dns1.p09.nsone.net. linkedin.com IN NS dns2.p09.nsone.net. linkedin.com IN NS dns3.p09.nsone.net. linkedin.com IN NS dns4.p09.nsone.net. linkedin.com IN NS ns1.p43.dynect.net. linkedin.com IN NS ns2.p43.dynect.net. linkedin.com IN NS ns3.p43.dynect.net. linkedin.com IN NS ns4.p43.dynect.net. $ NSOne DynDNS

35 Multi DNS Benefits Performance Availability

36 Multi DNS Dyn DDOS

37 Multi DNS Dyn DDOS

38 Traffic Steering How to select the best POP and CDN?

39 Traffic Steering GeoDNS

40 Traffic Steering GeoDNS

41 Traffic Steering GeoDNS Only knows resolver IP Client Local Resolver Authoritative Nameserver

42 Traffic Steering GeoDNS EDNS Client Subnet (rfc7871) # Ireland IP subnet dig +short +noshort +client= /24 media-lcdn.licdn.com 300 IN CNAME pop-tln1.media-src.linkedin.com. # India IP Subnet dig +short +noshort +client= /24 media-lcdn.licdn.com 300 IN CNAME pop-tmu1.media-src.linkedin.com.

43 Traffic Steering GeoDNS EDNS Client Subnet (rfc7871) # Ireland IP subnet dig +short +noshort +client= /24 media-lcdn.licdn.com 300 IN CNAME pop-tln1.media-src.linkedin.com. # India IP Subnet dig +short +noshort +client= /24 media-lcdn.licdn.com 300 IN CNAME pop-tmu1.media-src.linkedin.com.

44 Traffic Steering Anycast

45 Traffic Steering Anycast Region/Country Performance Illinois +20% Florida +22% Georgia +18% Pennsylvania +10%

46 Traffic Steering Anycast Region/Country Performance Illinois +20% Florida +22% Georgia +18% Pennsylvania +10% Arizona -21% Brazil -55% New York -3%

47 Traffic Steering Anycast Least # of hops!= Least latency Region/Country Performance Illinois +20% Florida +22% Georgia +18% Pennsylvania +10% Arizona -21% Brazil -55% New York -3%

48 Traffic Steering RUM (Real User Measurements) DNS

49 Public DNS with RUM

50 Traffic Steering RUM DNS

51 RUM Steering 2 Phases Measure Steer

52 RUM Steering Measurement Phase 0. Visitor

53 RUM Steering Measurement Phase 0. Visitor Site HTML: xx.css + yy.js + rum.js

54 RUM Steering Measurement Phase 0. Visitor 1. Identify Site HTML: xx.css + yy.js + rum.js One time key: i2-nqoqjnfrqy.init.cedexis-radar.net

55 RUM Steering Measurement Phase 0. Visitor 1. Identify Site HTML: xx.css + yy.js + rum.js One time key: i2-nqoqjnfrqy.init.cedexis-radar.net /providers.json

56 RUM Steering Measurement Phase 0. Visitor 1. Identify Site HTML: xx.css + yy.js + rum.js One time key: i2-nqoqjnfrqy.init.cedexis-radar.net /providers.json Key uniquely identifies this resolver RUM DNS Service

57 RUM Steering Measurement Phase 0. Visitor 1. Identify Site HTML: xx.css + yy.js + rum.js One time key: i2-nqoqjnfrqy.init.cedexis-radar.net /providers.json POP-X CDN1 CDN2 Key uniquely identifies this resolver RUM DNS Service

58 RUM Steering Measurement Phase 0. Visitor 1. Identify 2. Measure Site HTML: xx.css + yy.js + rum.js One time key: i2-nqoqjnfrqy.init.cedexis-radar.net /providers.json POP-X CDN1 CDN2 Key uniquely identifies this resolver RUM DNS Service

59 RUM Steering Measurement Phase 0. Visitor 1. Identify 2. Measure Site HTML: xx.css + yy.js + rum.js One time key: i2-nqoqjnfrqy.init.cedexis-radar.net /providers.json POP-X CDN1 CDN2 Key is geo-located! RUM DNS Service

60 RUM Steering Measurement Phase 0. Visitor 1. Identify 2. Measure Site HTML: xx.css + yy.js + rum.js One time key: i2-nqoqjnfrqy.init.cedexis-radar.net /providers.json POP-X CDN1 CDN2 Key uniquely identifies this resolver RUM DNS Service

61 RUM Steering Measurement Phase 0. Visitor 1. Identify 2. Measure 3. Report Site HTML: xx.css + yy.js + rum.js One time key: i2-nqoqjnfrqy.init.cedexis-radar.net /providers.json POP-X CDN1 CDN2 Key uniquely identifies this resolver RUM DNS Service

62 RUM Steering Measurement Phase 0. Visitor 1. Identify Client s IP address ASN Geolocation (reporting/aggregation) Performance data for client POP-X Client s resolver IP address Various clients/asns behind this resolver Performance data for various clients behind this resolver CDN1 2. Measure Site HTML: xx.css + yy.js + rum.js One time key: i2-nqoqjnfrqy.init.cedexis-radar.net /providers.json 3. Report CDN2 Key uniquely identifies this resolver RUM DNS Service

63 RUM Steering Measurement Phase

64 RUM Steering Measurement Phase

65 RUM Steering Measurement Phase

66 RUM Steering Measurement Phase

67 RUM Steering Measurement Phase RUM DNS Service

68 RUM Steering Measurement Phase RUM DNS Service

69 RUM Steering Steering Phase

70 RUM Steering Steering Phase 1. DNS Request linkedin.com RUM DNS Service

71 RUM Steering Steering Phase Resolver is identified 1. DNS Request linkedin.com RUM DNS Service

72 RUM Steering Steering Phase Resolver is identified 1. DNS Request 2. Inspect RUM linkedin.com RUM DNS Service

73 RUM Steering Steering Phase Resolver is identified 1. DNS Request 2. Inspect RUM If no ECS DNS resolver IP Aggregate performance data of ASNs behind that resolver If ECS Subnet of client to return the right DNS response with high confidence score. linkedin.com RUM DNS Service

74 RUM Steering Steering Phase Resolver is identified 1. DNS Request 2. Inspect RUM 3. DNS Response linkedin.com DNS cached RUM DNS Service

75 RUM Steering Steering Phase Resolver is identified 1. DNS Request 2. Inspect RUM 3. DNS Response linkedin.com DNS cached RUM DNS Service

76 RUM Steering RUM DNS at LinkedIn $ dig +short +noshort static.licdn.com What happens when a client resolves our LICDN domain

77 RUM Steering RUM DNS at LinkedIn $ dig +short +noshort static.licdn.com static.licdn.com. 300 IN CNAME c3e-003d.cdx.cedexis.net. Cedexis owned CNAME belonging to our account

78 RUM Steering RUM DNS at LinkedIn $ dig +short +noshort static.licdn.com static.licdn.com. 300 IN CNAME c3e-003d.cdx.cedexis.net c3e-003d.cdx.cedexis.net. 300 IN CNAME cs627.wac.epsiloncdn.net. Cedexis looks at RUM measurements and hands out the best from our CDNs

79 RUM Steering RUM DNS at LinkedIn $ dig +short +noshort static.licdn.com static.licdn.com. 300 IN CNAME c3e-003d.cdx.cedexis.net c3e-003d.cdx.cedexis.net. 300 IN CNAME cs627.wac.epsiloncdn.net. Low TTL so that clients pick up response changes faster

80 RUM Steering RUM DNS at LinkedIn $ dig +short +noshort static.licdn.com static.licdn.com. 300 IN CNAME c3e-003d.cdx.cedexis.net c3e-003d.cdx.cedexis.net. 300 IN CNAME cs627.wac.epsiloncdn.net. cs627.wac.epsiloncdn.net IN A $ CDN owned CNAME then resolves to an IP

81 CDN RUM Steering Cedexis Cedexis Akamai LinkedIn CDN Edgecast Quantil ChinaCache

82 CDN RUM Steering static.licdn.com. 300 IN CNAME c3e-003d.cdx.cedexis.net static.licdn.com. 300 IN CNAME c3e-0040.cdx.cdxcn.cn Cedexis Cedexis Akamai LinkedIn CDN Edgecast Quantil ChinaCache

83 POP RUM Steering IN CNAME c3e-003c.cdx.cedexis.net Cedexis Los Angeles Brazil London Mumbai Hong Kong Sydney

84 RUM Steering Benefits of RUM DNS

85 RUM Steering Benefits of RUM DNS Two CDNs having opposite performance throughout the day

86 RUM Steering Benefits of RUM DNS RUM DNS always picks the best of the two CDNs

87 RUM Steering Benefits of RUM DNS RUM DNS failing out a CDN automatically and shifting to others

88 RUM Steering RUM DNS vs GeoDNS

89 RUM Steering RUM DNS vs GeoDNS (DNS Resolver in US) DNS Resolver of a large chip manufacturing company.

90 RUM Steering RUM DNS vs GeoDNS (DNS Resolver in US) RUM Steering correctly to specific PoPs at different times.

91 RUM Steering RUM DNS vs GeoDNS (India) RUM DNS circumvents GeoIP inaccuracies in India GeoDNS RUM DNS

92 RUM Steering RUM DNS vs GeoDNS (Vietnam) Vietnam equidistant from Hong Kong and Singapore

93 RUM Steering RUM DNS vs GeoDNS (Vietnam) Vietnam correctly routed to a blend of POPs Singapore POP Hong Kong POP

94 RUM Steering RUM DNS vs GeoDNS (Vietnam) Viettel sees 50% latency improvement RUM DNS GeoDNS

95 RUM Steering RUM DNS vs GeoDNS (Philippines) Philippines closer to Hong Kong than Sydney. Hong Kong Philippines Sydney

96 RUM Steering RUM DNS vs GeoDNS (Philippines) Philippines Globe Telecom routed to the correct POP GeoDNS Hong Kong POP RUM DNS Sydney POP

97 RUM Steering Closing thoughts: FLAWS Cache hit ratio

98 RUM Steering Closing thoughts: FLAWS Workarounds Cache hit ratio CDN stickiness

99 RUM Steering Closing thoughts: FLAWS Workarounds Cache hit ratio CDN stickiness Cache miss latency

100 RUM Steering Closing thoughts: FLAWS Workarounds Cache hit ratio CDN stickiness Cache miss latency Measure miss latency with 2nd object

101 RUM Steering Closing thoughts: FLAWS Workarounds Cache hit ratio CDN stickiness Cache miss latency Measure miss latency with 2nd object POP load shedding

102 RUM Steering Closing thoughts Workarounds Cache hit ratio CDN stickiness Cache miss latency Measure miss latency with 2nd object POP load shedding Latency padding

103 Agenda CDN strategy POP Strategy RUM Steering Tooling

104 DNS Tool Single interface Auditing and logging Search records across zones Exposes API

105 CDN Regression tool SSL certificate Caching headers Compression honors origin cache headers

106 Purge Tool Self service Purge status tracking 5 minute SLA for site-wide purge

107 Synthetic Monitoring LinkedIn Monitor at global scale ~ ensuring our ability to serve LinkedIn through CDNs & DNS Create Selenium scripts used to simulate user behaviors/workflows (login, click a button, check for an element or string to create set of actions) Perform performance tests Troubleshooting tests Waterfalls, screenshots, scatter plot for tests 850 Unique Test

108 DNS Tool - One interface for all-things-dns CDN regression suite - CDN configuration consistency CDN Purge tool - Remove non-compliant assets from all CDNs Synthetic monitoring - keep tabs on your vendors Big Red Button - fail out of a vendor Other Tools POP turn up automation (DNS, SSL, monitoring) Visualize RUM steering decisions InCerts - provision SSL certs in couple clicks Synthetic Monitoring Akamai vs Edgecast

109 Catchpoint LinkedIn AMF Active Monitoring Framework TESTS Catchpoint Nodes Holden

110 DNS Tool - One interface for all-things-dns CDN regression suite - CDN configuration consistency CDN Purge tool - Remove non-compliant assets from all CDNs Synthetic monitoring - keep tabs on your vendors Big Red Button - fail out of a vendor Other Tools POP turn up automation (DNS, SSL, monitoring) Visualize RUM steering decisions InCerts - provision SSL certs in couple clicks Synthetic Monitoring Holden - CDN Availability Catchpoint Push API scales better Reduced time to detect issues Reduced time to recover from issues

111 Closing thoughts

112 Closing thoughts Build even more POPs

113 Closing thoughts Build even more POPs Don't depend on one provider

114 Closing thoughts Build even more POPs Don't depend on one provider Regionalize & reevaluate your vendors

115 Closing thoughts Build even more POPs Don't depend on one provider Regionalize & reevaluate your vendors Clients are your measurement agents

116 Closing thoughts Build even more POPs Don't depend on one provider Regionalize & reevaluate your vendors Clients are your measurement agents Don t stop your analysis

117 Closing thoughts Build even more POPs Don't depend on one provider Regionalize & reevaluate your vendors Clients are your measurement agents Don t stop your analysis Automate when you onboard itself

118