INFERRING PERSISTENT INTERDOMAIN CONGESTION

Transcription

1 INFERRING PERSISTENT INTERDOMAIN CONGESTION Amogh Dhamdhere with David Clark, Alex Gamero-Garrido, Matthew Luckie, Ricky Mok, Gautam Akiwate, Kabir Gogia, Vaibhav Bajpai, Alex Snoeren, k Claffy

2 Problem: High Volume Strains Internet Technology and Economics 2

3 Manifestation: Interdomain Congestion Access ISP A B E C D F G 3

4 Manifestation: Interdomain Congestion Access ISP A B E Interconnection disputes C resulted in congestion D F G 3

5 Manifestation: Interdomain Congestion Access ISP A B E Interconnection disputes C resulted in congestion D No data was available for third-party to study these F G links 3

6 Consequences of the Problem 4

7 Consequences of the Problem Congestion on transit links affects parties other than those involved in the dispute 4

8 Consequences of the Problem Congestion on transit links affects parties other than those involved in the dispute Limited data available to regulators and researchers to increase transparency and empirical grounding of debate 4

9 Consequences of the Problem Congestion on transit links affects parties other than those involved in the dispute Limited data available to regulators and researchers to increase transparency and empirical grounding of debate Our goal: third-party inference of congestion at interdomain interconnections 4

10 Consequences of the Problem Congestion on transit links affects parties other than those involved in the dispute Limited data available to regulators and researchers to increase transparency and empirical grounding of debate Our goal: third-party inference of congestion at interdomain interconnections Scientific approach to achieving this goal involves challenges in network inference, system development and data mining 4

11 Our Contributions 5

12 Our Contributions 1. Methodology: Operationalized a lightweight method for third-party inference of interdomain congestion, conducted thorough validation 5

13 Our Contributions 1. Methodology: Operationalized a lightweight method for third-party inference of interdomain congestion, conducted thorough validation 2. System: Built data collection and analysis platform to support the entire scientific workflow, and enable others to access and further study the data (ongoing) 5

14 Our Contributions 1. Methodology: Operationalized a lightweight method for third-party inference of interdomain congestion, conducted thorough validation 2. System: Built data collection and analysis platform to support the entire scientific workflow, and enable others to access and further study the data (ongoing) 3. Observations: Studied 8 large U.S. broadband providers from ch 16 to 17 (data collection ongoing) 5

15 Method: Time Series Latency Probes (But first, an observation) Peak-hour congestion fills up router buffers, resulting in elevated latency across an interdomain link 6

16 Method: Time Series Latency Probes (But first, an observation) Peak-hour congestion fills up router buffers, resulting in elevated latency across an interdomain link How do we measure latency across an interdomain link? 6

17 Time Series Latency Probes (TSLP) VP ISP A near far ISP B dst vantage point #A #B Border routers destination 7

18 Time Series Latency Probes (TSLP) VP ISP A near far ISP B dst vantage point RTT #A TTL: n #A #B Border routers destination 7

19 Time Series Latency Probes (TSLP) VP ISP A near far ISP B dst vantage point RTT #A TTL: n #A #B Border routers destination RTT #B TTL: n+1 7

20 Time Series Latency Probes (TSLP) VP ISP A near far ISP B dst vantage point RTT #A TTL: n #A #B Border routers destination RTT #B TTL: n+1 (repeat to obtain a time series) 7

21 An Experiment with TSLP Comcast Cogent VP near far dst vantage point #A #B Border routers destination Measured interdomain link from Comcast to Cogent using VP in Comcast 8

22 RTT (ms) 1 Thu 7th An Experiment with TSLP Fri 8th RTT measurements of border routers Cogent (far) Comcast (near) Sat Sun Mon Tue Wed 9th 1th 11th 12th 13th Day of week (local time in New York) Thu 14th *Luckie, Dhamdhere, Clark, Huffaker, Claffy, Challenges in Inferring Interdomain Congestion, IMC 14 9

23 An Experiment with TSLP RTT (ms) 1 Thu 7th Fri 8th Diurnal elevation to far-side RTT measurements of border routers Cogent (far) Comcast (near) Sat Sun Mon Tue Wed 9th 1th 11th 12th 13th Day of week (local time in New York) Thu 14th *Luckie, Dhamdhere, Clark, Huffaker, Claffy, Challenges in Inferring Interdomain Congestion, IMC 14 9

24 RTT (ms) 1 Thu 7th An Experiment with TSLP Fri 8th Diurnal elevation to far-side RTT measurements of border routers Cogent (far) Comcast (near) Sat Sun Mon Tue Wed 9th 1th 11th 12th 13th Day of week (local time in New York) Thu 14th No elevation to near-side *Luckie, Dhamdhere, Clark, Huffaker, Claffy, Challenges in Inferring Interdomain Congestion, IMC 14 1

25 Pieces of the Puzzle 11

26 Pieces of the Puzzle 12

27 Pieces of the Puzzle Interdomain link Identification: Need to identify interdomain links to be able to probe them 12

28 Pieces of the Puzzle Interdomain link Identification: Need to identify interdomain links to be able to probe them Adaptive Probing: Need to be adaptive to changes in the underlying topology and routing 12

29 Pieces of the Puzzle Interdomain link Identification: Need to identify interdomain links to be able to probe them Adaptive Probing: Need to be adaptive to changes in the underlying topology and routing Identifying Congested Links: Need time-series analysis techniques to find patterns in (noisy) data that indicate congestion 12

30 Pieces of the Puzzle Interdomain link Identification: Need to identify interdomain links to be able to probe them Adaptive Probing: Need to be adaptive to changes in the underlying topology and routing Identifying Congested Links: Need time-series analysis techniques to find patterns in (noisy) data that indicate congestion Validation: Need to validate inferences. Most peering agreements are covered by NDAs 12

31 System bdrmap probing bdrmap analysis External Inputs Interactive Data Exploration TSLP Links DB Probing history Real-time Dashboards NDT Youtube TSLP target selection Longitudinal Views Loss rate VP TSLP data Time series analysis Frontend Measurement and ANalysis of Internet Congestion Backend 13

32 System Interdomain link identification bdrmap probing bdrmap analysis External Inputs Interactive Data Exploration TSLP Links DB Probing history Real-time Dashboards NDT Youtube TSLP target selection Longitudinal Views Loss rate VP TSLP data Time series analysis Frontend Measurement and ANalysis of Internet Congestion Backend 13

33 System bdrmap probing bdrmap analysis External Inputs Interactive Data Exploration TSLP Links DB Probing history Real-time Dashboards NDT Youtube TSLP target selection Adaptive Probing Longitudinal Views Loss rate VP TSLP data Time series analysis Frontend Measurement and ANalysis of Internet Congestion Backend 13

34 System bdrmap probing bdrmap analysis External Inputs Interactive Data Exploration TSLP Links DB Probing history Real-time Dashboards NDT Youtube TSLP target selection Longitudinal Views Loss rate Identifying Congested Links VP TSLP data Time series analysis Frontend Measurement and ANalysis of Internet Congestion Backend 13

35 System bdrmap probing bdrmap analysis External Inputs Interactive Data Exploration TSLP Links DB Probing history Real-time Dashboards NDT Validation Youtube TSLP target selection Longitudinal Views Loss rate VP TSLP data Time series analysis Frontend Measurement and ANalysis of Internet Congestion Backend 13

36 System bdrmap probing bdrmap analysis External Inputs Interactive Data Exploration TSLP Links DB Probing history Real-time Dashboards NDT Youtube TSLP target selection Longitudinal Views Loss rate VP TSLP data Time series analysis Frontend Visualization Measurement and ANalysis of Internet Congestion Backend 13

37 Identifying Interdomain Links ISP A B E VP C D F G 14

38 Identifying Interdomain Links ISP A B VP E We focus on interdomain links of C network hosting a D measurement VP F G 15

39 Identifying Interdomain Links ISP A B E C D F G *Luckie, Dhamdhere, Huffaker, Clark, Claffy, bdrmap: Inference of Borders Between IP Networks, IMC 16 16

40 Identifying Interdomain Links ISP A B E bdrmap: C traceroute to all D routed IPv4 prefixes F G *Luckie, Dhamdhere, Huffaker, Clark, Claffy, bdrmap: Inference of Borders Between IP Networks, IMC 16 16

41 Identifying Interdomain Links ISP A B E C D F G *Luckie, Dhamdhere, Huffaker, Clark, Claffy, bdrmap: Inference of Borders Between IP Networks, IMC 16 17

42 Identifying Interdomain Links ISP A B E D C bdrmap: Infer router ownership to identify interdomain links at the router-level F G *Luckie, Dhamdhere, Huffaker, Clark, Claffy, bdrmap: Inference of Borders Between IP Networks, IMC 16 17

43 Identifying Congested Links Focus on persistently congested links Look for periods of elevated latency that correlate across days (autocorrelation method) 18

44 Autocorrelation method Day 1 Day 2 Day 3 Day 5 Far-side latency 19

45 Autocorrelation method Day 1 Day 2 Day 3 Day 5 Far-side latency 19

46 Autocorrelation method Day 1 Day 2 # Days Day 3 Sum number of days showing elevation in each 15-min interval Day 5 Far-side latency 19

47 Autocorrelation method Day 1 Day 2 # Days Day 3 Infer an interval of recurring congestion Day 5 Far-side latency

48 Autocorrelation method Day 1 Day 2 # Days Day 3 Map inferred interval onto each day Day 5 Far-side latency 21

49 Autocorrelation method Day 1 Day 2 # Days Day 3 Map inferred interval onto each day Day 5 Reject elevated intervals that do not overlap with Far-side latency recurring interval 21

50 Heavy Emphasis on Validation (Critical due to political nature of inferences) 22

51 Heavy Emphasis on Validation (Critical due to political nature of inferences) Validation of inference method - Correlation with loss - Correlation with throughput - Correlation with YouTube streaming 22

52 Heavy Emphasis on Validation (Critical due to political nature of inferences) Validation of inference method - Correlation with loss - Correlation with throughput - Correlation with YouTube streaming Validation of specific inferences - Ground truth from operators 22

53 Heavy Emphasis on Validation (Critical due to political nature of inferences) Validation of inference method - Correlation with loss - Correlation with throughput - Correlation with YouTube streaming Validation of specific inferences - Ground truth from operators 23

54 Heavy Emphasis on Validation (Critical due to political nature of inferences) Validation of inference method - Correlation with loss - Correlation with throughput - Correlation with YouTube streaming Brief overview in this talk See paper for full details Validation of specific inferences - Ground truth from operators 23

55 RTT 1 minimu 3 1 Does TSLP inference correlate with loss? TSLP latency RTT Loss 15-min rate (5-min minimum average) (ms) Loss rate (5-min average) : 6: 18: 18: 6: 6: 18: UTC Time 18: UTC Time 6: 6: 18: 18: Far Far RTT Loss Near Near RTT Loss 6: Far Loss Near Loss 6: 24

56 RTT 1 minimu 3 1 Does TSLP inference correlate with loss? TSLP latency RTT Loss 15-min rate (5-min minimum average) (ms) Loss rate (5-min average) : 6: 18: 18: 6: Diurnal latency elevation to far side indicates congestion 6: 18: UTC Time 18: UTC Time 6: 6: 18: 18: Far Far RTT Loss Near Near RTT Loss 6: Far Loss Near Loss 6: 24

57 Does TSLP inference correlate with loss? TSLP latency RTT 15-min minimum (ms) Far RTT Near RTT Loss Rate Loss rate (5-min average) : 18: 6: 18: UTC Time 6: 18: Far Loss Near Loss 6: 25

58 Does TSLP inference correlate with loss? TSLP latency RTT 15-min minimum (ms) Far RTT Near RTT Loss Rate Loss rate (5-min average) : 18: 6: 18: UTC Time 6: 18: Far Loss Near Loss 6: Far loss rate exceeds near loss rate during periods of inferred congestion 25

59 Does TSLP inference correlate with loss? TSLP latency RTT 15-min minimum (ms) Far RTT Near RTT Loss Rate Loss rate (5-min average) : 18: 6: 18: UTC Time 6: 18: Far Loss Near Loss 6: Far loss rate during inferred congestion exceeds far loss rate during non-congested period 26

60 Does TSLP inference correlate with loss? TSLP latency RTT 15-min minimum (ms) Far RTT Near RTT Loss Rate Loss rate (5-min average) : 18: 6: 18: UTC Time 6: 18: Far Loss Near Loss 6: In 81% of cases, far loss rate was higher during congested periods and higher than near loss rate 27

61 Does TSLP inference correlate with throughput? M-lab NDT server Approach: throughput ISP B measurements from Ark far side VP to M-lab NDT server traversing congested congested link interdomain link near side ISP A Ark VP 28

62 Does TSLP inference correlate with throughput? M-lab NDT server Approach: throughput ISP B measurements from Ark far side VP to M-lab NDT server traversing congested congested link interdomain link near side Ark VP ISP A Challenge: difficult to find NDT servers that cover specifically observed interconnections 28

63 Does TSLP inference correlate with throughput? RTT 15-min minimums (ms) Far RTT Near RTT Throughput (Mbps) : 18: 6: 18: 6: UTC Time 18: 6: Download Throughput 18: 6: 29

64 Does TSLP inference correlate with throughput? RTT 15-min minimums (ms) Far RTT Near RTT Throughput (Mbps) : 18: 6: 18: 6: UTC Time 18: 6: Download Throughput 18: 6: Lower throughput during periods inferred congested 29

65 Does TSLP inference correlate with throughput? RTT 15-min minimums (ms) Throughput (Mbps) : 18: 6: 18: Avg. throughput 27Mbps during uncongested periods 6: UTC Time 18: 6: 18: Far RTT Near RTT Download Throughput 6: Avg. throughput 8Mbps during congested periods 3

66 Validation: Operator Feedback 31

67 Validation: Operator Feedback Validated our inferences with operators from two large U.S. access ISPs 31

68 Validation: Operator Feedback Validated our inferences with operators from two large U.S. access ISPs ISP A: 7 links (all inferred congested) ISP B: links (1 inferred congested, 1 uncongested) 31

69 Validation: Operator Feedback Validated our inferences with operators from two large U.S. access ISPs ISP A: 7 links (all inferred congested) ISP B: links (1 inferred congested, 1 uncongested) Our inferences were correct in each case: no false positives or false negatives 31

70 Longitudinal Study Collecting data since ch 16 Focused on interdomain links of 8 large access ISPs in the U.S. to their transit providers and peers from 16 to 17 Driving questions: - How prevalent is interdomain congestion? - Which transit/content providers are most often congested to access providers? - Can we characterize trends over time? 32

71 What Did We Find (so far)? No evidence of widespread (pervasive) congestion between 16 and 17 Small fraction of peers of the 8 studied access providers showed evidence of congestion Certain transit providers (e.g., TATA) and content providers (e.g. Google) most often showed evidence of congestion Interesting dynamics of interdomain congestion See paper for details 33

72 34 Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Percent of congested day-links over time

73 35 Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Percent of congested day-links over time Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox AT&T

74 35 Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Comcast-Google congestion increased and subsided over time Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Percent of congested day-links over time Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox AT&T

75 36 Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Time Warner - TATA congestion subsided in ember 16 Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon AT&T TWC

76 37 Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC AT&T - TATA congestion increased and subsided over time Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Percent of congested day-links over time

77 38 Providers: Zayo NTT Tata Google Netflix Providers: Telia XO Vodafone Level3 Cox Verizon CenturyLink AT&T Comcast TWC Interdomain congestion evolves over time. Need for ongoing measurements!

78 Public Access to Data We are publicly releasing our data via an interactive visualization system (based on Grafana) And API access to the time series data (based on InfluxDB) 39

79 Interactive Visualization Longitudinal view of a single link, April - November 17

80 Interactive Visualization Zooming in for more detail 41

81 Interactive Visualization Zooming in for more detail 41

82 Takeaways We have developed a lightweight method and system to provide third-party visibility into interdomain congestion We hope that our data can provide empirical grounding to debates over interconnection performance Contact us for access to the data: manic-info@caida.org 42

83 Host a Measurement VP! We are always looking for volunteers to host VPs! Contact us: manic-info@caida.org 43

84 Thanks! Questions? 44