Advancing the Art of Internet Edge Outage Detection

Size: px

Start display at page:

Download "Advancing the Art of Internet Edge Outage Detection"

Milo Henry
5 years ago
Views:

1 Advancing the Art of Internet Edge Outage Detection ACM Internet Measurement Conference 2018 Philipp Richter MIT / Akamai Ramakrishna Padmanabhan University of Maryland Neil Spring University of Maryland Arthur Berger Akamai / MIT David Clark MIT

2 Internet Edge Outages 1

3 Internet Edge Outages Uninterrupted Internet availability becomes increasingly critical Growing interest in systems to detect Internet edge outages Regulatory Bodies, Governments, ISPs, Academic Research 1

4 Internet Edge Outages Uninterrupted Internet availability becomes increasingly critical Growing interest in systems to detect Internet edge outages Regulatory Bodies, Governments, ISPs, Academic Research Goal: Track Internet edge outages on (i) a broad scale and (ii) a detailed level 1

5 Edge Outage Detection: Existing Approaches Detecting outages in the control plane? Edge outages often invisible in BGP Deploying hardware in end user premises? Potentially highly accurate, but does not scale Actively probing addresses? Challenging to scale, unresponsive addresses, difficult to interpret 2

6 Edge Outage Detection: Existing Approaches Detecting outages in the control plane? Edge outages often invisible in BGP Deploying hardware in end user premises? Potentially highly accurate, but does not scale Actively probing addresses? Challenging to scale, unresponsive addresses, difficult to interpret This work: Passive edge outage detection based on CDN request patterns 2

7 Outline Disruption Detection Global View on Disruptions Device-Centric View on Disruptions U.S. Broadband Case Study

8 Disruption Detection using CDN Access Logs 200,000+ servers ASes 120+ countries > 3 trillion daily requests Dataset: Hourly request counts per IPv4 /24 address block 3

9 Disruption Detection using CDN Access Logs 200,000+ servers ASes 120+ countries > 3 trillion daily requests Dataset: Hourly request counts per IPv4 /24 address block Assumption: Edge outages will be reflected in absence/reduction of CDN requests. 3

10 Active IP Addresses per /24 active IPv4 addresses per hour hours typical residential /24 address block 4

11 Active IP Addresses per /24: Baseline Activity active IPv4 addresses per hour baseline value: 93 IP addresses hours typical residential /24 address block Number of addresses contacting the CDN every hour never drops below baseline value value per block. 4

12 Active IP Addresses per /24: Baseline Activity active IPv4 addresses per hour baseline value: 93 IP addresses hours typical residential /24 address block CDN content requests, update requests, widgets, beacons, etc. home network 4 icons: mavadee / flaticon

13 Active IP Addresses per /24: Baseline Activity active IPv4 addresses per hour baseline value: 93 IP addresses hours typical residential /24 address block CDN content requests, update requests, widgets, beacons, etc. home network Some 2.3M /24 address blocks (83% of all client IPs) baseline > 40 Robust signal, largely independent of user-triggered activity Dependent on a functioning network 4

14 Disruption Detection active IPv4 addresses baseline b0: sliding window alpha * min [last 168 hours] hours 5

15 Disruption Detection active IPv4 addresses baseline b0: sliding window alpha * min [last 168 hours] hours 5

16 Disruption Detection activity in next hour violates baseline criterion active IPv4 addresses baseline b0: sliding window alpha * min [last 168 hours] hours 5

17 Disruption Detection active IPv4 addresses baseline b0: sliding window alpha * min [last 168 hours] baseline b1: sliding window beta * min [last 168 hours] hours 5

18 Disruption Detection active IPv4 addresses baseline b0: sliding window alpha * min [last 168 hours] baseline b1: sliding window beta * min [last 168 hours] hours 5

19 Disruption Detection new baseline found active IPv4 addresses baseline b0: sliding window alpha * min [last 168 hours] baseline b1: sliding window beta * min [last 168 hours] hours 5

20 Disruption Detection active IPv4 addresses baseline b0: sliding window alpha * min [last 168 hours] non-steady state } baseline b1: sliding window beta * min [last 168 hours] hours 5

21 Disruption Detection active IPv4 addresses baseline b0: sliding window alpha * min [last 168 hours] non-steady state } baseline b1: sliding window beta * min [last 168 hours] hours 5

22 Did this Address Block really go offline? active IPv4 addresses ICMP responsive CDN active hour [starting ] 6

23 Did this Address Block really go offline? active IPv4 addresses ICMP responsive CDN active hour [starting ] Address Space Survey Data (provided by ISI) Ping every address in 1% of the allocated IPv4 /24s every 11 mins Some address blocks show a very steady number of responsive IPs 6

24 Did this Address Block really go offline? active IPv4 addresses ICMP responsive CDN active hour [starting ] Address Space Survey Data (provided by ISI) Ping every address in 1% of the allocated IPv4 /24s every 11 mins Some address blocks show a very steady number of responsive IPs 6

25 CDN Disruptions vs. ICMP / /24 active IPv4 addresses CDN active IPs ICMP responsive IPs active IPv4 addresses CDN active IPs ICMP responsive IPs hour / hour /24 active IPv4 addresses CDN active IPs ICMP responsive IPs active IPv4 addresses CDN active IPs ICMP responsive IPs hour hour 6

26 Global View on Disruptions hourly disrupted /24s 8K (~0.7%) 4K (~0.35%) partial /24 disrupted entire /24 disrupted 0 Apr 2017 May 2017 Jun 2017 Jul 2017 Aug 2017 Sep 2017 Oct 2017 Nov 2017 Dec 2017 Jan 2018 Feb M disruption events over the course of one year 8

27 Global View on Disruptions hourly disrupted /24s 8K (~0.7%) 4K (~0.35%) partial /24 disrupted entire /24 disrupted 0 Apr 2017 May 2017 Jun 2017 Jul 2017 Aug 2017 Sep 2017 Oct 2017 Nov 2017 Dec 2017 Jan 2018 Feb M disruption events over the course of one year At least 0.2% of the monitored space faces a disruption Natural disasters, intended Internet shutdowns Weekly pattern, mostly absent during Christmas/NYE 8

28 Global View on Disruptions Hurricane Irma hourly disrupted /24s 8K (~0.7%) 4K (~0.35%) partial /24 disrupted entire /24 disrupted 0 Apr 2017 May 2017 Jun 2017 Jul 2017 Aug 2017 Sep 2017 Oct 2017 Nov 2017 Dec 2017 Jan 2018 Feb M disruption events over the course of one year At least 0.2% of the monitored space faces a disruption Natural disasters, intended Internet shutdowns Weekly pattern, mostly absent during Christmas/NYE 8

29 Weekly Pattern fraction of disruption events Mon Tue Wed Thu Fri Sat Sun day of the week [local time] all entire /24 9

30 Weekly Pattern fraction of disruption events Mon Tue Wed Thu Fri Sat Sun day of the week [local time] all entire /24 fraction of disruption events start hour [local time] all entire /24 9

31 Weekly Pattern: Scheduled Maintenance Window fraction of disruption events Mon Tue Wed Thu Fri Sat Sun day of the week [local time] all entire /24 fraction of disruption events start hour [local time] all entire /24 9

32 Outline Disruption Detection Global View on Disruptions Device-Centric View on Disruptions U.S. Broadband Case Study

Device Perspective on Disruptions active IPv4 addresses 0 50

1 168 336 504 672 hours Established that this address block

33 Device Perspective on Disruptions active IPv4 addresses ? hours Established that this address block lost connectivity Do devices really lose Internet connectivity? Or can they connect from somewhere else? 10

34 Device-Specific Dataset for Subset of Users software ID A software ID B software ID C 10

35 Device-Specific Dataset for Subset of Users IP during IP before /24 IP after software ID A software ID B software ID C /24 disruption in /24 time / /24 Device-information for 52K entire /24 disruption events Only consider disruptions which affected an entire /24 (no activity during the disruption) 10

36 Device Perspective on Disruptions disruptions with active IDs < 1hr before N=52K 11

37 Device Perspective on Disruptions 86% disruptions with active IDs < 1hr before N=52K no active IDs during disruption expected case, devices do not have connectivity 11

38 Device Perspective on Disruptions disruptions with active IDs < 1hr before N=52K 86% 14% no active IDs during disruption active IDs during disruption from different IP address expected case, devices do not have connectivity 11

39 Device Perspective on Disruptions disruptions with active IDs < 1hr before N=52K 86% 14% no active IDs during disruption expected case, devices do not have connectivity 20% active IDs during disruption from different IP address 13% switch to/from cellular switch AS mobility and/or multi-homed devices 11

40 Device Perspective on Disruptions disruptions with active IDs < 1hr before N=52K 86% 14% no active IDs during disruption active IDs during disruption from different IP address expected case, devices do not have connectivity 20% 13% 67% switch to/from cellular switch AS same AS mobility and/or multi-homed devices? (10% of disruptions) 11

41 Anti-Disruptions: Temporary Surges in Address Activity IPs in disrupted/alternate / disrupted /24 alternate /24 time [hours] ID XYZ ID XYZ 12

42 Anti-Disruptions: Temporary Surges in Address Activity IPs in disrupted/alternate / disrupted /24 alternate /24 time [hours] ID XYZ ID XYZ Some disruptions not service outages but address reassignment! Developed mechanism to detect anti-disruptions Rank ASes by correlation of disruptions with anti-disruptions 12

43 AS-Level Disruptions and Anti-Disruptions (anti ) disrupted IPs 40K 0 40K (anti ) disrupted IPs 0 100K 250K major US cable ISP: No correlation between disruptions and anti-disruptions (pearson r= 0.02) Uruguayan ISP: Strong correlation between disruptions and anti-disruptions (pearson r= 0.63) 13

44 AS-Level Disruptions and Anti-Disruptions (anti ) disrupted IPs 40K 0 40K (anti ) disrupted IPs 0 100K 250K major US cable ISP: No correlation between disruptions and anti-disruptions (pearson r= 0.02) Uruguayan ISP: Strong correlation between disruptions and anti-disruptions (pearson r= 0.63) Anti-Disruptions can heavily skew per-as and per-country assessment of Internet reliability 13

45 Case Study: US Broadband ISPs ISP A Cable ISP B Cable ISP C Cable ISP D DSL ISP E DSL ISP F DSL ISP G DSL anti-disruption correlation % disruptions w/ intermittent activity 4% 1% 1% 0% 3% 6% 14% % /24s only disrupted during Hurricane Irma 11% 1% 2% 23% 1% 0% 3% % /24s only disrupted maintenance window 67% 54% 75% 29% 60% 71% 62% 14

46 Case Study: US Broadband ISPs ISP A Cable ISP B Cable ISP C Cable ISP D DSL ISP E DSL ISP F DSL ISP G DSL anti-disruption correlation % disruptions w/ intermittent activity 4% 1% 1% 0% 3% 6% 14% % /24s only disrupted during Hurricane Irma 11% 1% 2% 23% 1% 0% 3% % /24s only disrupted maintenance window 67% 54% 75% 29% 60% 71% 62% Most major US broadband ISPs do not show strong signs of anti-disruptions. 14

47 Case Study: US Broadband ISPs ISP A Cable ISP B Cable ISP C Cable ISP D DSL ISP E DSL ISP F DSL ISP G DSL anti-disruption correlation % disruptions w/ intermittent activity 4% 1% 1% 0% 3% 6% 14% % /24s only disrupted maintenance window 67% 54% 75% 29% 60% 71% 62% % /24s only disrupted during Hurricane Irma 11% 1% 2% 23% 1% 0% 3% All (but one) ISP has majority of all disrupted /24s during scheduled maintenance window (Mo-Fr Midnight-6AM). 14

48 Case Study: US Broadband ISPs ISP A Cable ISP B Cable ISP C Cable ISP D DSL ISP E DSL ISP F DSL ISP G DSL anti-disruption correlation % disruptions w/ intermittent activity 4% 1% 1% 0% 3% 6% 14% % /24s only disrupted maintenance window 67% 54% 75% 29% 60% 71% 62% % /24s only disrupted during Hurricane Irma 11% 1% 2% 23% 1% 0% 3% ISP A: Some 80% of all disruptions fall in the maintenance window, or are caused by force majeure (Hurricane Irma). Relevant for SLAs, policies, and reliability assessment. 14

49 Implications Outage Detection: Methodological Insights Baseline activity enables fine-granular detection of disruptions Anti-disruptions due to reassignment Can bias active and passive outage detection systems Interpreting Service Outages Majority of outages (for many ISPs) during scheduled maintenance Implications for SLAs, reporting requirements, regulations hourly disrupted /24s 8K (~0.7%) 4K (~0.35%) 0 partial /24 disrupted entire /24 disrupted Apr 2017 May 2017 Jun 2017 Jul 2017 Aug 2017 Sep 2017 Oct 2017 Nov 2017 Dec 2017 Jan 2018 Feb

76 days Wed 8/24/16 Wed 12/7/16 Daniel Wang,Shreyas Makde,Madhavi Potluri,Roua 2 Requirements analysis 11 days Wed 8/24/16 Wed 9/7/16

ID Mode Name Duration Start Finish Predecessors Resource Names 1 OpenWells Cross-Platform Mobile Application 76 days Wed 8/24/16 Wed 12/7/16 Daniel Wang,Shreyas Makde,Madhavi 2 Requirements analysis 11