Detailed Analysis of ISIS Routing Protocol on the Qwest Backbone:

Transcription

1 Detailed Analysis of ISIS Routing Protocol on the Qwest Backbone: A recipe for subsecond ISIS convergence Cengiz Alaettinoglu cengiz@packetdesign.com Stephen Casner casner@packetdesign.com Packet Design 1

2 Why subsecond convergence? Increased network reliability Support for multi service traffic Voice over IP, ATM over IP, TDM over IP,... Lower cost/complexity compared to layer 2 protection schemes like SONET Packet Design 2

3 Where are we today? Current IP re route times are typically tens of seconds We need to do better. There are two choices: Figure out what s wrong with IP routing and fix it Replace IP routing with something else Packet Design 3

4 Qwest Backbone ISIS Analysis Collected multiple week long ISIS packet traces Identified problem areas: causes of ISIS churn and stability sequence of events and delays during routing convergence Conclude with a recipe for achieving subsecond ISIS convergence Packet Design 4

5 Qwest ISIS Findings Summary The backbone is extremely stable 3 out of the 4 week long data collection periods have no route change on our path the churn is caused by few problem links Convergence times Hard to find a link failure to diagnose Convergence as fast as today s technology allows Can be improved to subsecond Packet Design 5

6 ISIS Routing Protocol 101: Link State Packets Router Link / adjacency An LSP describes the local topology of a router List of adjacencies Hello protocol Packet Design 6

7 ISIS Routing Protocol 101: Flooding LSPs are flooded link propagation delays + per hop processing delays Packet Design 7

8 ISIS Routing Protocol 101: Shortest Path Calculation = A router constructs the current topology Dijkstra s SPF algorithm determines the routes Packet Design 8

9 ISIS Routing Protocol 102: Convergence after a Topology Change Detection Link up/down or peer reachability Hardware detection is fast & preferred Software detection using an HELLO protocol is slower but is a backup Propagation Flood a Link State Packet (LSP) Link propagation delays + per hop processing delay Rate limiting may slow propagation New Route Computation Run Dijkstra s Shortest Path First (SPF) algorithm CPU resource intensive Rate limiting may delay SPF computation & consistency Packet Design 9

10 Our Measurement Setup Collection Host ISIS HELLOs + capture traceroute every 5s sk gig eth Burbank R Qwest Backbone Houston R Collection Host UDP stream (tg) packet per avg 6 msec Collection host is passive peer, sends no LSPs Multi vendor backbone: Ciscos, Junipers,... All point to point backbone: OC48, OC192,... Packet Design 10

11 Our Typical Path R1 R3 Burbank R2 R4 Los Angeles R1 R1 Sunnyvale R1 R2 Houston 6 hops Packet Design 11

12 ISIS Churn and Instability Churn: number of LSPs received over a time period requires SPF calculations consumes CPU resources Busy CPU may cause HELLO packet misses can falsely bring adjacencies down increases churn Instability churn => busy CPU => HELLO misses => more churn =>... rate limits are for avoiding this instability Packet Design 12

13 HELLO Packets Interval between hello packets (msecs) Oct 06 Oct 07 Oct 08 Oct 09 Oct 10 Oct 11 Oct 12 Excellent HELLO behavior Packet Design 13

14 Churn 100 Total Churn 100 Physical Churn Number of LSPs per minute Oct 06 Oct 07 Oct 08 Oct 09 Oct 10 Oct 11 Oct 12 Number of LSPs per minute Oct 06 Oct 07 Oct 08 Oct 09 Oct 10 Oct 11 Oct 12 High churn region Very stable network: Average rate for total churn: 1 LSP every 2 seconds 97.6% of the LSPs are state refreshes Packet Design 14

15 Churn by Routers & Links Number of LSPs generated Problem routers Number of Flaps Problem links Percentage of Routers Percentage of Links About 800 LSPs per week per router is for refreshes This can be configured to be less Packet Design 15

16 One Atypical Router LSP sequence number times its usual churn Oct 06 Oct 07 Oct 08 Oct 09 Oct 10 Oct 11 Oct 12 This router is responsible for the initial high churn Packet Design 16

17 An Unstable Link Up Down 06:00 06:10 06:20 06:30 06:40 06:50 07:00 September 24, 2001 No protection against instability Goes on for a day Opposite of fast convergence requirement 30 seconds to go down, 8 seconds to go up Packet Design 17

18 Dealing with Instability SPF & LSP propagation rate limits don t reduce churn does keep CPU from melting by ignoring change To reduce churn without impacting convergence: Asymetric up/down filters for fast convergence detect bad news fast slow down on good news Adaptive filters linear or exponential adaptation to level of instability Less CPU intensive incremental SPF algorithms Packet Design 18

19 An Example Adaptive Filter Up Up 20 mins 20 mins Down Down 06:00:00 06:15:00 06:30:00 06:45:00 07:00:00 06:00:00 06:15:00 06:30:00 06:45:00 07:00:00 An example exponential filter with 20 minute max penalty It reduces the churn without hurting convergence Packet Design 19

20 ISIS Convergence Delay Time from the physical change to new routing tables Failure/repair detection LSP propagation Delay due to SPF interval SPF computation Packet Design 20

21 What Happens During Convergence? Routers perform SPF while their views of the network are not consistent, causing: routing loops black hole routes suboptimal routes If fast convergence, this is not an issue. But, Convergence times are not fast On high speed links, lots of packets are affected New services are less tolerant Packet Design 21

22 A Link Failure Delay and Loss packets R1 Packet Loss Delay R2 R3 Burbank R4 Los Angeles R1 R1 Sunyvale R1 R2 Houston 8.5 seconds Why so long? 07:05:40 07:05:55 07:06:10 07:06:25 October 10, 2001 Packet Design 22

23 Slow Link Failure Detection Delay, Loss, and LSP reception Packet Loss Delay Burbank LSP Los Angeles LSP 3 seconds, link layer detection 20 seconds, HELLO protocol 07:05:40 07:05:55 07:06:10 07:06:25 October 10, 2001 Packet Design 23

24 LSP Propagation Delay LSP propagation times (msecs) R1 R2 R3 Burbank R4 Los Angeles R1 R1 Sunyvale R1 R2 Houston Difference of 2 clocks LSP propagations are rate limited Their scheduling may be improved 0 Oct 06 Oct 07 Oct 08 Oct 09 Oct 10 Oct 11 Oct 12 Packet Design 24

25 SPF Rate Limiting spf interval parameter delays SPF computation default: SPF computation after 5 seconds from the change visible in our convergence delay example Two goals: to contain the CPU load no more than one SPF computation per 5 seconds to capture 2 4 LSPs reporting the failure in one SPF run fails to do this in our case Packet Design 25

26 Why Loss and Delay at Link Repair? Delay and Packet Loss Packet Loss Delay 227 packets lost R1 R3 Burbank R2 R4 Routing loop? Los Angeles R1 R1 Sunyvale R2 R1 Houston 1.5s 07:06:30 07:06:35 07:06:40 07:06:45 October 10, 2001 Packet Design 26

27 TTLs Confirm the Routing Loop 0.19 Delay TTL Delay, TTL Routing loop Longer path ttl=11 ttl=8 Convergence ttl=60,...,10 ttl=8 1.6s 1.5s 07:06:30 07:06:35 07:06:40 07:06:45 October 10, 2001 Packet Design 27

28 spf interval Spreads SPFs Delay, TTL and LSP receptions Delay TTL Router1 SPF Router2 SPF LA s lsp 5secs Bbank s lsp 5secs Some lsp 5secs Router3 SPF 3 routers start timers after 3 different LSPs 07:06:30 07:06:35 07:06:40 07:06:45 October 10, 2001 Packet Design 28

29 SPF Computation Times SPF Computation Times (microseconds) Dijkstra SPF Incremental SPF avg=1069usec avg=13usec Qwest topology and events using equal cost multiple paths On average 84 times faster Percentage of SPF runs Packet Design 29

30 SPF Scaling SPF Computation times (milliseconds) Dijkstra SPF Incremental SPF Source: Haobo Yu Using random topology and events Incremental SPF lets the network grow very large Number of Routers Packet Design 30

31 Where does the time go? Detection times were several seconds, must be improved LSP Propagation times were subsecond, but still much larger than link propagation delays SPF rate limiting spf interval causes most of the convergence delay spf interval spreads SPFs, groups wrong set of LSPs into the same SPF Packet Design 31

32 A Recipe for Subsecond ISIS Convergence Vendors: fast link failure detection Hardware detection is preferred Vendors have fast failure detection solution for MPLS fast reroute It will benefit convergence immediately Vendors: adaptive and asymmetric Up/Down filters It will reduce the ISIS churn w/o hurting convergence Operators: eliminate current LSP & SPF rate limits Adaptive asymmetric filters make it safe Vendors: incremental SPF algorithm A must for avoiding CPU meltdowns even as the network gets bigger Packet Design 32

33 Acknowledgements We would like to thank Chris Sieber, Paul Mabey and Shankar Rao of Qwest for providing access to their network for our study and for their review of our results. We would also like to thank Haobo Yu, Van Jacobson, Kathleen Nichols, and Kedar Poduri of Packet Design for their contributions to this work. Packet Design 33