On low-latency-capable topologies, and their impact on the design of intra-domain routing

Transcription

1 On low-latency-capable topologies, and their impact on the design of intra-domain routing Nikola Gvozdiev, Stefano Vissicchio, Brad Karp, Mark Handley University College London (UCL)

2 We want low latency!

3 We want low latency! In the datacenter

4 We want low latency! In the datacenter In the enterprise WAN [B4, BwE, SWAN ] operator controls both WAN and sources so demands are predictable

5 We want low latency! In the datacenter In the enterprise WAN [B4, BwE, SWAN ] operator controls both WAN and sources so demands are predictable In the ISP this talk ISP operator does not control sources

6 How do we get low latency in a loaded ISP network?

7 How do we get low latency in a loaded ISP network? The topology? Topology must offer diverse lowlatency paths

8 How do we get low latency in a loaded ISP network? The topology? Topology must offer diverse lowlatency paths The routing? and routing system must make good use of those low-latency paths

9 How do we get low latency in a loaded ISP network? The topology? Topology must offer diverse lowlatency paths The routing? and routing system must make good use of those low-latency paths

10 All possible topologies connecting a set of PoPs Venn diagram

11 Shortest-path routing doesn t yield lowest latency on all topologies Other topologies SP routing gives lowest possible latency

12 Shortest-path routing doesn t yield lowest latency on all topologies Other topologies SP routing gives lowest possible latency

13 Shortest-path routing doesn t yield lowest latency on all topologies capacity: 1.5 Other topologies SP routing gives lowest possible latency

14 Shortest-path routing doesn t yield lowest latency on all topologies capacity: 1.5 Other topologies demands: 1 each SP routing gives lowest possible latency

15 Shortest-path routing doesn t yield lowest latency on all topologies capacity: 1.5 Other topologies demands: 1 each SP routing gives lowest possible latency

16 Shortest-path routing doesn t yield lowest latency on all topologies capacity: 1.5 Other topologies demands: 1 each Can t achieve lower latency on a tree! SP routing gives lowest possible latency

17 Better Topologies May Need Better Routing capacity: 1.5 demands: 1 each Let s improve the topology, let s add redundancy!

18 Better Topologies May Need Better Routing capacity: 1.5 demands: 1 each Congestion inflates latency if both aggregates don t fit.

19 Better Topologies May Need Better Routing capacity: 1.5 demands: 1 each Congestion control makes aggregates fit; hurts throughput

20 Better Topologies May Need Better Routing capacity: 1.5 Other topologies demands: 1 each SP routing gives lowest possible latency

21 Better Topologies May Need Better Routing Other topologies SP routing gives lowest possible latency

22 Better Topologies May Need Better Routing Other topologies Modern TE (B4, MPLS-TE) give lowest possible latency

23 Better Topologies May Need Better Routing capacity: 1.5 Other topologies demands: 1 each Modern TE (B4, MPLS-TE) give lowest possible latency

24 Better Topologies May Need Better Routing capacity: 1.5 Other topologies demands: 1 each Route as much as possible on shortest paths Modern TE (B4, MPLS-TE) give lowest possible latency

25 Better Topologies May Need Better Routing capacity: 1.5 Other topologies demands: 1 each Traffic is split, low-latency top path fully loaded Modern TE (B4, MPLS-TE) give lowest possible latency

26 Do Even Better Topologies Need Even Better Routing? Do any topologies fall in this region? Other topologies Modern TE (B4, MPLS-TE) give lowest possible latency

27 Do Even Better Topologies Need Even Better Routing? Do any topologies fall in this region? If so, do any of them have a greater potential to provide low latency? Other topologies Modern TE (B4, MPLS-TE) give lowest possible latency

28 Do Even Better Topologies Need Even Better Routing? Do any topologies fall in this region? If so, do any of them have a greater potential to provide low latency? Why does current routing do poorly on those topologies? Other topologies Modern TE (B4, MPLS-TE) give lowest possible latency

29 Limitations of Today s Routing Proof by example. Consider this real-world ISP topology Central European network of GTS, 2010

30 1.0 CDF Shortest path CDF over 100 runs of SP routing on synthetic traffic matrices on GTS s topology Fraction of flows that encounter congestion

31 1.0 CDF Shortest path Fraction of all flows in each traffic matrix that cross at least one congested link Fraction of flows that encounter congestion

32 SP does poorly, as expected 1.0 CDF Shortest path Each point is a run of the routing system on a different traffic matrix Fraction of flows that encounter congestion

33 B4 for the win sort of Jain, Sushant, et al. "B4: Experience with a globally-deployed software defined WAN." ACM SIGCOMM CDF 0.8 Shortest path B Fraction of flows that encounter congestion

34 Where does greedy routing such as B4 You are here! go wrong?

35 Where does greedy routing such as B4 go wrong? Let s focus on a small part of the network

36 Limitations of greedy routing

37 Limitations of greedy routing 1.Allocate as much as possible on shortest path

38 Limitations of greedy routing G Local flows between Veszprem (V) and Gyor (G) V 1.Allocate as much as possible on shortest path

39 Limitations of greedy routing Through flows G Local flows between Veszprem (V) and Gyor (G) V 1.Allocate as much as possible on shortest path

40 Limitations of greedy routing G First link on V->G aggregate s shortest path fills eastbound V 1.Allocate as much as possible on shortest path

41 Limitations of greedy routing G First link on V->G aggregate s shortest path fills eastbound V 1.Allocate as much as possible on shortest path 2.Allocate to longer paths

42 Limitations of greedy routing G First link on V->G aggregate s shortest path fills eastbound? V 1.Allocate as much as possible on shortest path 2.Allocate to longer paths

43 Limitations of greedy routing First link on V->G V->G s second aggregate s shortest best path is already full, using it results in congestion!? G V path fills eastbound 1.Allocate as much as possible on shortest path 2.Allocate to longer paths

44 Limitations of greedy routing

45 Limitations of greedy routing Rings embedded in a topology can trigger this problem with greedy routing

46 B4 for the win sort of 1.0 CDF 0.8 Shortest path B Fraction of flows that encounter congestion

47 Can we do better? 0.8 CDF Yes, a placement which both avoids congestion and minimizes propagation delay does exist! 1.0 Shortest path B4 Latency-optimal Fraction of flows that encounter congestion

48 Can we do better? 0.8 CDF Yes, a placement which both avoids congestion and minimizes propagation delay does exist! 1.0 Shortest path B4 Latency-optimal to low latency. Are other So GTS is amenable topologies? Fraction of flows that encounter congestion

49 How might we quantify a topology s potential for low latency under load? Want a metric to capture a topology s inherent potential for low latency Should be: traffic matrix-agnostic routing algorithm-agnostic

50 How might we quantify a topology s potential for low latency under load? Want a metric to capture a topology s inherent potential for low latency Should be: traffic matrix-agnostic routing algorithm-agnostic Want to capture two things: topology s potential for routing around congestion hot spots without incurring long propagation delay

51 How might we quantify a topology s potential for low latency under load? Want a metric to capture a topology s inherent potential for low latency Should be: traffic matrix-agnostic We want a metric that rewards routing algorithm-agnostic alternate paths with short Want to capture two things: propagation delay topology s potential for routing around congestion hot spots without incurring long propagation delay

52 Alternate Path Availability (APA) Y Gbps

53 Alternate Path Availability (APA) Shortest path: T mstotal propagation delay Y Gbps SP capacity Y Gbps

54 Alternate Path Availability (APA) Shortest path: T mstotal propagation delay Y Gbps SP capacity Y Gbps Exclude each link on the shortest path; can we route Y Gbps over one or more alternative paths with delay < 1.4 T?

55 Alternate Path Availability (APA) Shortest path: T mstotal propagation delay Y Gbps SP capacity Y Gbps Exclude each link on the shortest path; can we route Y Gbps over one or more alternative paths with delay < 1.4 T?

56 Alternate Path Availability (APA) Shortest path: T mstotal propagation delay Y Gbps SP capacity Links on shortest path 1 5 Y Gbps Links with viable alternative paths Exclude each link on the shortest path; can we route Y Gbps over one or more alternative paths with delay < 1.4 T?

57 Alternate Path Availability (APA) Shortest path: T mstotal propagation delay Y Gbps SP capacity 1 5 Y Gbps Exclude each link on the shortest path; can we route Y Gbps over one or more alternative paths with delay < 1.4 T?

58 Alternate Path Availability (APA) Shortest path: T mstotal propagation delay Y Gbps SP capacity 2 5 Y Gbps Exclude each link on the shortest path; can we route Y Gbps over one or more alternative paths with delay < 1.4 T?

59 Alternate Path Availability (APA) Shortest path: T mstotal propagation delay Y Gbps SP capacity 3 5 Y Gbps Exclude each link on the shortest path; can we route Y Gbps over one or more alternative paths with delay < 1.4 T?

60 Alternate Path Availability (APA) Shortest path: T mstotal propagation delay Y Gbps SP capacity 3 5 Y Gbps Path too long or not enough capacity Exclude each link on the shortest path; can we route Y Gbps over one or more alternative paths with delay < 1.4 T?

61 Alternate Path Availability (APA) Shortest path: T mstotal propagation delay Y Gbps SP capacity 4 5 Y Gbps Exclude each link on the shortest path; can we route Y Gbps over one or more alternative paths with delay < 1.4 T?

62 Alternate Path Availability (APA) Shortest path: T mstotal propagation delay Y Gbps SP capacity 45 = 0.8 Y Gbps For this PoP pair 80% of the links on the SP have an alternate path with acceptable low latency Exclude each link on the shortest path; can we route Y Gbps over one or more alternative paths with delay < 1.4 T?

63 Low-latency path diversity (LLPD) 1. Compute APA for all PoP pairs

64 Low-latency path diversity (LLPD) 1. Compute APA for all PoP pairs 2. Compute LLPD = Fraction of PoP pairs with good path availability

65 Low-latency path diversity (LLPD) 1. Compute APA for all PoP pairs 2. Compute LLPD = = Fraction of PoP pairs with good path availability number of PoP pairs with APA 0.7 total number of PoP pairs

66 Low-latency path diversity (LLPD) 1. Compute APA for all PoP pairs Empirically derived; metric not sensitive to picking different values 2. Compute LLPD = = Fraction of PoP pairs with good path availability number of PoP pairs with APA 0.7 total number of PoP pairs

67 fraction of pairs congested 90th percentile Median 0.5 GTS LLPD real-world ISP topologies, ranked by low-latency path diversity (LLPD)

68 fraction of pairs congested 90th percentile Median 0.5 GTS LLPD Generate TMs for each topology; plot fraction of (Src,Dst) PoP pairs in each TM that crosses at least one congested link

69 fraction of pairs congested Shortest path routing congests links 90th percentile Median 0.5 GTS LLPD Two points per topology: median TM and 90th percentile TM; line shows spread of distribution

70 fraction of pairs congested Shortest path routing congests links 90th percentile Median 0.5 GTS LLPD Networks with high LLPD offer lots of alternative paths à shortest path routing experiences congestion

71 fraction of pairs congested Shortest path routing congests links 90th percentile Median 0.5 GTS LLPD Networks with LLPD here. offer lots of alternative Nohigh surprises What paths à shortest pathabout routing experiences congestion B4?

72 Better at using alternative paths fraction of pairs congested B4 congests networks with high potential for low latency 90th percentile Median 0.5 GTS LLPD

73 total prop delay of all flows total prop delay if all flows routed on SP latency stretch Better at using alternative paths fraction of pairs congested B4 congests networks with high potential for low latency GTS GTS th percentile Median LLPD

74 latency stretch Better Propagation delay Better Congestion fraction of pairs congested B4 congests networks with high potential for low latency GTS GTS th percentile Median LLPD

75 latency stretch Better at using alternative paths Some flows routed on non-shortest paths fraction of pairs congested B4 congests networks with high potential for low latency GTS GTS th percentile Median LLPD

76 latency stretch Better at using alternative paths Some flows routed on non-shortest paths Still incurs congestion, and precisely on high-llpd networks! fraction of pairs congested B4 congests networks with high potential for low latency GTS GTS th percentile Median LLPD

77 fraction of pairs congested B4 congests networks with high potential for low latency GTS GTS latency stretch Better at using 0.5 alternative paths Some flows routed on non-shortest paths 1.0 Still incurs 1.2 congestion, and 90th percentile Need a different routing scheme. How 1.4 precisely on Median about one that prioritizes avoiding high-llpd networks! congestion above all else?llpd

78 Minimizing utilization avoids congestion

79 Minimizing utilization avoids congestion Spread traffic out to leave spare capacity in case traffic levels increase A well-known technique called MinMax Does not care about propagation delay 33%

80 Minimizing utilization avoids congestion Spread traffic out to leave spare capacity in case traffic levels increase A well-known technique called MinMax Does not care about propagation delay How does MinMax do? 33%

81 latency stretch Minimizes utilization, designed to avoid congestion fraction of pairs congested MinMax inflates propagation delay GTS th percentile Median LLPD

82 latency stretch Minimizes utilization, designed to avoid congestion Routes some flows on paths with high propagation delay fraction of pairs congested MinMax inflates propagation delay GTS th percentile Median LLPD

83 latency stretch Minimizes utilization, designed to avoid congestion Routes some flows on paths with high propagation delay fraction of pairs congested MinMax inflates propagation delay 0.5 One extreme of the design space. The other one? 1.0 GTS th percentile Median LLPD

84 latency stretch Minimizes prop delay and avoids congestion Maximizes utilization of links on low-delay paths fraction of pairs congested Latency-optimal placement GTS th percentile Median LLPD

85 Assume it is possible to compute this at scale, more about that later latency stretch Minimizes prop delay and avoids congestion Maximizes utilization of links on low-delay paths fraction of pairs congested Latency-optimal placement GTS th percentile Median LLPD

86 fraction of pairs congested latency stretch latency stretch fraction of pairs congested Two extremes of congestion-free routing GTS th percentile Median LLPD Minimize utilization (MinMax) GTS th percentile Median LLPD Minimize propagation delay and avoid congestion

87 fraction of pairs congested latency stretch latency stretch fraction of pairs congested Two extremes of congestion-free routing GTS th percentile Median LLPD Minimize utilization (MinMax) GTS th percentile Median LLPD Minimize propagation delay and avoid congestion

88 entile fraction of pairs congested GTS latency stretch latency stretch fraction of pairs congested Two extremes of congestion-free routing GTS th percentile Median th percentile Median LLPD Minimize utilization (MinMax) GTS GTS th percentile Median LLPD Minimize propagation delay and avoid congestion

89 entile fraction of pairs congested GTS GTS th percentile Median 1.4 latency stretch latency stretch fraction of pairs congested Two extremes of congestion-free routing GTS GTS th percentile Median 90th percentile 0.1 GTS x 0.5 difference sees in propagation delay! LLPD LLPD Median Minimize utilization (MinMax) Minimize propagation delay and avoid congestion

90 entile fraction of pairs congested GTS GTS th percentile Median 1.4 latency stretch latency stretch fraction of pairs congested Two extremes of congestion-free routing GTS GTS th percentile Median 90th percentile 0.1 GTS x 0.5 difference sees in propagation delay! LLPD LLPD Median Minimize utilization Minimize propagation delay on a single traffic matrix Let s focus (MinMax) and avoid congestion

91 Two extremes of congestion-free routing CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) link utilization 0.8 Significant delta in prop delay, but mean utilization the same 1.0

92 Two extremes of congestion-free routing CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) link utilization Links on short prop-delay paths in demand in latency-optimal placement

93 Two extremes of congestion-free routing 1.00 CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) link utilization

94 Two extremes of congestion-free routing 1.00 CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) link utilization All possible congestion-free routing solutions lie in this range

95 Two extremes of congestion-free routing 1.00 CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) link utilization % utilization? On an ISP?

96 A minute from a core link 4 3 Gbps Source: CAIDA time (s) +3.2e3

97 A minute from a core link 4 3 Gbps Mean rate. Could run traffic through a path with this capacity, but long queuing delay time (s) +3.2e3

98 A minute from a core link 4 3 Gbps Need to allocate headroom to allow for variability time (s) +3.2e3

99 The headroom dial 1.00 CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) link utilization Not feasible because of variability

100 The headroom dial 1.00 CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) link utilization MinMax is one extreme of the headroom dial

101 The headroom dial 1.00 CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) 7% headroom 27% headroom link utilization

102 The headroom dial 1.00 CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) 7% headroom 22% headroom link utilization

103 The headroom dial 1.00 CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) 7% headroom 15% headroom link utilization

104 The headroom dial 1.00 CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) 7% headroom link utilization

105 The headroom dial 1.00 CDF Latency-optimal (mean 0.32) MinMax (mean 0.30) 2% headroom link utilization Need to allow the minimal amount of headroom to cope with variability

106 Towards a low-latency routing system

107 Towards a low-latency routing system Compute latency-optimal routing solution, sans headroom expressed the problem as one big linear program (largely straightforward) efficient iterative solution: add paths, solve, repeat 400+ nodes, less than one second (vs. tens of minutes )

108 Towards a low-latency routing system Compute latency-optimal routing solution, sans headroom expressed the problem as one big linear program (largely straightforward) efficient iterative solution: add paths, solve, repeat 400+ nodes, less than one second (vs. tens of minutes ) Tune headroom dial to drive routing as close as possible to optimal solution while avoiding congestion predict how aggregates will statistically multiplex on a path by convolving their past demands

109 Towards a low-latency routing system Compute latency-optimal routing solution, sans headroom expressed the problem as one big linear program (largely straightforward) efficient iterative solution: add paths, solve, repeat 400+ nodes, less than one second (vs. tens of minutes ) More details in the paper! Tune headroom dial to drive routing as close as possible to optimal solution while avoiding congestion predict how aggregates will statistically multiplex on a path by convolving their past demands

110 Are high-llpd networks viable? A routing system may not be able to unlock the lowlatency potential of a topology LLPD indicates that a topology has good potential for low latency

111 Are high-llpd networks viable? A routing system may not be able to unlock the lowlatency potential of a topology LLPD indicates that a topology has good potential for low latency But will anyone ever really build a modern WAN with high LLPD?

112 Are high-llpd networks viable? fraction of pairs congested Repeated SP experiment, but added Google s network 90th percentile Median 0.5 Google LLPD 0.8

113 Are high-llpd networks viable? fraction of pairs congested Repeated SP experiment, but added Google s network 90th percentile Median Google Modern0.5high-LLPD network! LLPD 0.8

114 Are high-llpd networks viable? fraction of pairs congested Repeated SP experiment, but added Google s network 90th percentile Median Modern0.5high-LLPD network! Google B4 however, does great on that network! Could it be because the routing and the LLPDtopology co-evolved?

115 What topologies would people build if they knew the routing system would always extract the best from it?

116 Conclusions To achieve low latency: topology must provide low-latency paths the routing system must use them effectively State-of-the-art routing falters on high-llpd topologies precisely those with best potential for low latency Practical routing approach for high-llpd topologies: Efficient LP solution for optimal traffic placement Tune headroom dial to avoid congestion (but as little toward MinMax as possible)

117 System Design Simple, centralized design

118 System Design Simple, centralized design But measure what?

119 Measurements Only need measurements per aggregate, not per flow! Need to know enough to figure out both long and short-term variability for each aggregate Sampling traffic level 10 times per second is enough to capture short-term variability due to TCP s congestion control since RTTs in the ISP are long (order of 100ms) Sampling 10 times per second well within reach of recent hardware [DevoFlow SIGCOMM 2011]

120 What about prioritization? If you can you should definitely prioritize delaysensitive traffic but identifying this traffic in the ISP setting may not be trivial, since no single operator controls all sources also, what about bandwidth-hungry low-latency traffic (e.g., VR)