FC LS Slovakia Expo Cisco Systems, Inc. All rights reserved. Cisco Public

Transcription

1 1

2 Fast Convergence for Link State Protocols Scott Sturgess 2

3 FAST CONVERGENCE: Concept and Types of Failure 3

4 FC Concept Convergence is the ability of the network to restore from a link or node failure by finding and installing for each known IP prefix an alternate path in the forwarding table Fast convergence is the ability to do this within one second 4

5 FC Concept Fast Convergence is not all about the IGP Other components need to react fast as well: Hardware/software for link failure detection Hardware/software for installing prefixes in RIB/FIB and distributed FIB Faster hardware results in faster convergence Lots of optimizations are system specific 5

6 Example ISP Core Site A R1 R2 R4 Site B R6 R3 R5 Metric 200 Metric 3 Metric 2 R7 Site C Traffic flows (TF) AB (Single Path) BC (Load balanced path) AC (Single Path) 6

7 ISP Core: Link Failure R5-R7 A R1 R2 R4 B R6 R3 R5 X Metric 200 Metric 3 Metric 2 R7 C Impact on traffic flows AB: R1R3R5R6 NO CHANGE BC: LB(R6R4R7:R6R5R7) SP(R6R4R7) AC: R1R3R5R7 R1R2R4R7 One link failure might completely change the path for a given src/dst pair 7

8 ISP Core: Link Failure R5-R7 A R1 R2 R4 B R6 R3 R5 X Metric 200 Metric 3 Metric 2 R7 C Impacted routers (routers that have changed their FIB for TF) R1, R5: FIB change for destination C (R3 only changes metric in RIB!) R6: Change for C (LB SP) R7: Change for A and B (LB SP) Not every router is updating the same amount of prefixes in its RIB 8

9 Rerouting Node and Failure Node A failure node is the node directly connected to the link failure A rerouting node is a node that can direct traffic to an alternate path without causing a temporary routing loop (micro-loop) There might be several different rerouting nodes for a certain destination. 9

10 Remote and Local Failure For a local failure, the first rerouting node is the node directly connected to the link failure. For a remote failure, the first rerouting node is not directly connected to the link failure. 10

11 Rerouting Node and Failure Node A R1 R2 R4 R6 B R3 R5 X Metric 200 Metric 3 Metric 2 R7 C failure nodes = first rerouting node = R5, R7 Other nodes (R1,R6) could be potential first rerouting nodes as well due to different factors (processing delay, RIB order not same, ), however in lab testing it is advised to exclude such scenarios in order to ease troubleshooting. 11

12 Rerouting Node and Failure Node A We ve removed link R4R5 R1 R2 R4 R6 B R3 R5 X Metric 200 Metric 3 Metric 2 R7 C failure nodes = R5, R7 First rerouting nodes = R3, R7, R6 For src/dst AC, if R5 converges dest C before R3, then a temp. loop occurs and traffic will still drop until R3 or R1 have converged. 12

13 Recursive Prefixes: BGP IPv4 or VPNv4 AS1 R1 R2 AS2 R4 AS3 R6 R3 R5 Metric 200 Metric 3 Metric 2 R7 AS4 How does convergence work for prefixes that are not known by the IGP? Example BGP IPv4 or VPNv4 BGP prefixes are recursive prefixes: they use the forwarding information of their BGP next-hop CRR = CEF Recursion Resolution = a process that needs to assure that BGP prefixes use the forwarding information of their BGP next-hop CRR = 0 means that traffic loss to BGP prefixes is same as traffic loss for their BGP next-hop on which these prefixes recurse 13

14 Recursive Prefixes: BGP IPv4 AS1 ibgp ebgp R1 R2 AS2 R4 R6 ebgp AS3 R3 R5 Metric 200 Metric 3 Metric 2 ibgp R7 ebgp AS4 R6,R7 learn about IPv4 BGP prefixes in AS1 via ibgp session with R1: By default BGP-NH for these prefixes is link R1-AS1. This link is advertised in the IGP as passive-interface R1 could be configured with next-hop self (loop0) When R5R7 fails, R7 will find out that path to BGP-NH changes and updates the FIB. However BGP IPv4 table does not change (BGP prefixes are stable and BGP-NH doesn t change). How does R7 change path for BGP prefixes? CRR process will change the FIB for all BGP prefixes as it knows that their next-hop has changed. 14

15 Recursive Prefixes: BGP VPNv4 VPN1 ibgp ebgp, OSPF, R1 R2 R3 AS2 R4 R5 R6eBGP, OSPF, VPN3 Metric 200 Metric 3 Metric 2 ibgp R6,R7 learn about VPNv4 BGP prefixes in VPN1 via ibgp session with R1: By default BGP-NH for these prefixes is loop0. R7 ebgp, OSPF, When R5R7 fails, R7 will find out that path to BGP-NH changes and updates the FIB. How does R7 change path for VPNv4 prefixes? VPN4 CRR process will change the FIB for all VPNv4 prefixes as it knows that their next-hop has changed. 15

16 FAST CONVERGENCE: Shortest Path First Algorithm 16

17 Shortest Path First Algorithm Also known as Dijkstra Computes the shortest path to all reachable nodes We maintain three lists UNKNOWN list: All nodes start on this list TENTative list: PATHS list: All nodes to which a path has been found which is not yet known to be minimum cost Also called the candidate list All nodes to which we have calculated final paths with definitive cost Also called the known list 17

18 Shortest Path First Algorithm We execute N steps Typically N is the number of nodes in the network; during each step we find the path(s) to one node from root At each step: Find the node amongst all nodes on TENT that has the lowest cost, and move it from TENT into PATHS Find all neighbours reachable from that node and move them into TENT Find all prefixes advertised by the node that was moved to PATHS and install them in the RIB 18

19 Shortest Path First Calculation (How It Works) R1 10 R R R6 100 R3 R2 5 R4 100 R5 TENT_LIST: R3(15),R4(15),R5(15), R2(10), R3(100), R6(25), R1 R6 PATH_LIST: R1(0), R2(10), R3(15), R4(15), R5(15), R6(25), R3 R5 19

20 SPF and PRC Basic Implementation Any change (link, node, leave) Recompute the whole SPT and the whole RIB Decouple SPT and RIB Called SPF Called PRC If any topology change (node, link) Recompute SPT and the RIB If only an IP prefix changes Keep the SPT, just update the RIB for the nodes whose prefixes have changed 20

21 SPF, PRC & LSP/LSA Generation Initial wait before SPF, PRC and LSP/LSA generation 5.5 seconds for SPF & PRC 50 msec for LSP/LSA generation Configurable minimum interval between consecutive events By default is 5 seconds for LSP/LSA generation 10 seconds for SPF & PRC calculation 21

22 Exponential Backoff Timer IS-IS and OSPF throttle their main events SPF computation PRC computation LSP/LSA generation Throttling slows down convergence Not throttling can cause melt-downs Find a compromise The goal is to react fast to changes but then to backoff under constant churn to avoid network meltdown 22

23 Exponential Backoff Timer Extended syntax spf-interval <a> [<b> <c>] <a> seconds between SPF runs (seconds) <b> milliseconds between first trigger and SPF <c> milliseconds between first and second SPF Same syntax for prc-interval lsp-gen-interval 23

24 Exponential Backoff Timer Example: spf-interval We decide to run an SPF (a) (b) (c) Wait 100 msecs, then run SPF (b=100) Wait at least 1 second before running a second SPF if needed (c = 1000) If we need to run a 3rd SPF, right after, wait at least 2 seconds (c = 2c) Wait at least 4 sec before next SPF, then 8 sec, then 10 sec, 10 sec, (c= MIN(2c, a)) When the network calms down, and there were no triggers for at least twice the minimum interval (20 secs in this example), then go back to fast behavior (after 100 msecs initial wait) 24

25 Incremental SPF (i-spf) Basic Operation We keep the unchanged part of the tree We rebuild only the affected parts of the tree Re-attach affected parts of the tree to the unchanged part of the tree More information is kept in the SPT Have to store and compare current and new information The primary lists we maintain are: Parents List Neighbour List Memory impact for using i-spf (but not significant) Based on the changed information, the SPT is modified in order to reflect the changes 25

26 Incremental SPF R R3 R2 5 R R4 5 R R5 R6 NEW_LSP_LIST: R4-LSA, R6-LSA, ORPHAN_LIST: R6, TENT_LIST: R3(15),R4(15),R5(15), R6(115), R2(10), R3(100), R6(25), R1 R6 PATH_LIST: R1(0), R2(10), R3(15), R4(15), R5(15), R6(25), R6(115), R3 R5 26

27 Incremental (ispf) Gain of incremental SPF depends on how far (topologically) the change happens from the calculating node If the change affects only a small part of the topology, gain is significant we were able to run SPF and update the RT for the 1000 node network in less then 10 ms If the change is close to the calculating node and affect almost the whole topology, there will be no gain in ispf 27

28 FAST CONVERGENCE: Measuring Convergence Time 28

29 Convergence Time Defined T1 Convergence T2 T3 A R1 R2 R3 R4 R5 L R6 B Metric 200 Metric 3 Metric 2 Assume a flow from Src A to Dest C T1: when L dies, the best path is impacted: loss of traffic T2: when the network converges, traffic reaches the destination again Loss of Connectivity for flow AC: T2 T1, called convergence hereafter R7 C 29

30 Prefix Distribution with LoC Convergence time R5 Tmax Ti T Prefix As prefixes are updated in a sequential order, convergence time will be different for each destination that needs a RIB change. LoC for prefix 2000 might be much higher than for prefix 1. 30

31 What Makes This Graph? Convergence time R5 Tmax Ti T Prefix Initial delay (Ti-T1) = Link detection << 10ms (POS p2p) LSP/LSA gen << 10 ms Flooding + propagation = fn (#hops, fiber distance) SPF = f(#nodes, Topology) Slope = (distributed) FIB update time 31

32 LoC-ked in a Formula LoC= Ax+B B = Initial delay A = Rewrite/prefix x = number of (important) prefixes that need to reroute To construct the graph Ax+B, one needs to send traffic to multiple destinations When sending 1000 pps to each src/dst, count the number of lost packets to know the convergence time in milliseconds. 32

33 Reducing Convergence Time Following factors have an impact on Ax + B (example Cisco 12000): PRP better than GRP PRP2 is 2.6x faster than GRP (E3, E4p, E6) better than E2 IP better than MPLS ispf better than full SPF Use latest code 33

34 !Only in ISIS! Reducing Convergence Time with CSCdy18429 Tmax(voip) Ti T1 Tmax(voip) Ti T Before After Voip gateways Loopbacks p2p links (config)# interface e0 (config-if)# isis tag 17 (config-if)# router isis (config-router)# ip route priority high tag 17 Prefix ordering is an important feature to reduce convergence time: Use ISIS tagging to bring a prefix to the beginning of the curve Loopbacks (/32) are automatically ordered after tagged prefixes Without prefix ordering feature, traffic to important destinations could see the worst-case convergence time. Using prefix ordering traffic loss to these prefixes will be significantly reduced because the router will rewrite the FIB entries for these destinations first. 34

35 FAST CONVERGENCE: Design and Configuration 35

36 Components Contributing to Loss of Connectivity (LoC) Connectivity is Lost Whilst: Failure is detected New LSP is originated, propagated, and received SPF is run Routing table/fib is updated LCs are updated BGP recursion is fixed LoC(p) = D + O + QSP + (h * F) + SPF(n) + RIB(p) + DD + CRR 36

37 Design Fundamentals Things to know about your network Total number of nodes, total number of prefixes Link types: Fast link detection on POS, GE p2p Other GE or multi-access needs BFD for fast detection Do we care for loopbacks (/32) and other important traffic? Do we care for node failures or SRLG? Redundancy within areas Reduce number of prefixes: Configure point-to-point for GE Advertise passive only (new design) 37

38 Infrastructure CRS/12k: Full Isolation Control vs Data Plane RP CPU dedicated to IGP/BGP PRP2 is 2.5 faster than GRP LC CPU dedicated to control of the forwarding HW LC CPU protected against any through traffic Optimization of the download to linecards 12.0S/12k: ip cef linecard ipc service-timer 50 IOS-XR, 12k/CRS: optimized by default IOS-XR offers massive control plane scaling distributed processing (drp) 38

39 Infrastructure OS scheduling priority-aware: IOS and IOS-XR low granularity: IOS-XR by default (4ms) IOS: process-max-time 50 (default = 200ms) 39

40 Failure Detection How Fast? POS or DPT or GE with LoS sonet/sdh HW failure detection < 10ms SW time to notify ISIS 31S1: P90 < 10ms, P100 = 50ms Otherwise: BFD 12k/31S1: max 100 session per LC, minimum transmit interval of 50ms and minimum holdtime = 150ms Fast Hello s NOT recommended minimum hold time is 1 second Use BFD instead! 40

41 Failure Detection How Much Control? POS UP=>DN: pos delay trigger line/path <ms> 0m or 60/110ms depending on sonet/sdh protection DN=>UP: 10sec fixed confirmation Carrier-Delay delay between the failure detection/clearance and the interface status change redundant for PoS IOS: default value = 2sec, suggested tuning (0, 10 s of ms) IOS-XR: default = 0ms 41

42 POS: Link Failure Detection t 0 : defect occurs t 2 -t 3 : hold-off for any pos delay triggers t 3 -t 4 : carrier delay t 0 t 1 t 2 t 3 t 4 t 5 t 1 : hardware detects the defect t 4 : int down in IOS t 2 : driver registers defect as a failure t 5 : adjacency down 42

43 Failure Detection - Further Stability IP Event Dampening int foo x/y dampening Immediate signalling of link DOWN information Controlled signalling of link UP information control = if the link status changed frequently in the recent past, even if the lower layers report the link as up, the IP event dampening layer will report the link as down to the L3 clients. This stabilizes the network. Same algorithm as BGP route flap dampening 43

44 IP Event Dampening Algorithm Illustration Actual interface state Maximum penalty Accumulated penalty Suppress threshold Reuse threshold Interface state seen by routing protocols 44

45 Link Failure Detection Assure fast link failure detection while preserving stability: POS: Fast detection, native dampening GE ptp: Fast detection, no native dampening Multi-access: Use BFD! In all cases, configure: Fast hellos not recommended if BFD available Dead interval is 1 second for IS-IS and OSPF (config-if)# ip dampening (config-if)# carrier-delay msec 0 45

46 Point-to-Point GE Back-to-back configuration direct fiber connection with or without optical/dwdm isis network point-to-point command stops DIS election ip ospf network point-to-point command stops DR election Fiber cut => speedy LoS detection no need for fast hellos auto-negotiation ON unidir failure is advertised in the other direction int gig 1/0 carrier-delay msec 0 ip dampening negotiation auto isis network point-to-point ip ospf network point-to-point 46

47 ISIS FC and NSF/SSO When NSF/SSO is included in the design, a good objective is to avoid losing the hello adjacency during a valid switch-over. Testing has indicated that the hold time should not be configured to less than 4 seconds to achieve this (ie. interval 1sec and multiplier 4) This applies to both IS-IS and OSPF 47

48 Advertisement of the Failure How Fast? ISIS/OSPF must regenerate its LSP/LSA and must flood it upon Local changes of adjacency status, prefix connectivity, or redistribution How fast is a regeneration? Under stress, 31S1: P100 < 12ms 48

49 Flooding Does Geography Matter? Number of hops between the failure and the rerouting node is small very rarely larger than 5 Structured Network design leads to failure containment into limited region of the network very rarely larger than 30ms of light propagation Reference: ICI research project - study of a WW SP Ex: SP in UK should account for 5 hops and 15ms of propagation CPU computation is small 31S1: P90 < 2ms per hop, P100 < 50ms per hop Source of flooding delay: Pacing 49

50 LSP/LSA Generation Interval Per node, capture all link failures in one new LSP/LSA Uses exponential back-off timers (IW, max, increment) When one link failure happens at a time, use min IW value When multiple link failures could occur at the same time (SRLG), use time as high as duration to detect all link failures, example 50 ms Increment, guideline is to use 20 ms Max wait: default 5 seconds is ok (config-router)# router isis (config-router)# lsp-gen interval (config-router)# router ospf (config-router)# timers throttle lsa all

51 Flooding When pacing is applied flooding time increases Default pacing is 33 ms Objective is to flood all important LSP/LSAs without delay, hence without pacing. Works different for ISIS vs OSPF ISIS: Pacing is applied on a per LSP basis Pacing timer can be tuned but not recommended because of fast-flood command OSPF: Pacing is applied on a per Update packet basis One update packet may contain multiple LSAs (config-router)# router isis (config-router)# fast-flood <value> (config-router)# router ospf (config-router)# timers pacing flood <ms> 51

52 SPF How Fast? SPT = shortest-path tree the set of the shortest paths from the local node to any remote node Incremental-SPF offers the most optimum computation if leaf change: SPT(new)=SPT(old) if link failure not on the previous SPT: SPT(new)=SPT(old) if link failure on the previous SPT if not local link: we only recompute the part of the old SPT that has been impacted if local link: we recompute the whole SPT router isis ispf level-2 52

53 Measurement: Full SPT Computation PRP1 - wide metric full SPT [ms] y = x R 2 = GRP: ~ 100us / node PRP1: ~ 59us / node PRP2: ~ 45us / node nodes Full SPT computation (wide metric) with Local RIB update 600 nodes => 35 ms i-spf will most of the time be much smaller than this 53

54 Network Convergence Incremental SPF Time it takes to run the SPF with the transit link flap Link flap SPF Time in msec ispf Number of nodes X

55 (i)spf: InitWait, Increment, MaxWait Enable ispf to reduce calculation time Init-wait: When only considering link failures init-wait can be set to min 1 msec When considering node failure, init-wait should be tuned to allow time to receive all LSPs/LSAs from neighbors within init-wait interval: msec Increment, MaxWait: Conservative 50ms, 1000ms (config-router)# router isis (config-router)# ispf (config-router)# spf-interval (config-router)# prc-interval (config-router)# router ospf (config-router)# ispf (config-router)# timers throttle spf

56 RIB/FIB How Fast? The more prefixes in the IGP, the more RIB/FIB updates may be necessary RIB/FIB update is measured on a per entry change and depends on RP cpu speed (PRP2 is 2.5 faster than GRP) IP vs MPLS (MPLS requires more update time per entry) A rule of thumb PRP2, IP2IP: 66 us per entry PRP2, IP2MPLS: 150us per entry 56

57 RIB/FIB How Fast? By default, RIB process (called by IS-IS and OSPF) is responsible for updating the RIB when prefixes are changing Better if ISIS and OSPF update the RIB prefixes directly themselves when their local oif goes down: ip routing protocol purge interface Default behaviour for IOS-XR 57

58 RIB/FIB How Much Control? Prioritization (IS-IS only): 3 priority levels: H, M, L: the most important ( H ) prefixes are updated first, then the important ( M ) prefixes then the rest ( L ) By default, the /32 s are in M and the rest are L Classification based on tags to customize whether a prefix is in H or M or L ISIS: 12.0(27)S and IOS-XR 3.2 interface foo ;; VoIP gateway ip router isis isis tag 17 router isis ip route priority high tag 17 58

59 System Optimizations Lower process-max-time Reduce distribution delay Bring CRR to 0 Other Check memory constraint LCs (config)# service internal (config)# process-max-time 60 (config)# ip cef linecard ipc service-timer 50 (config)# ip cef table loadinfo force (config)# ip routing protocol purge interface 59

60 Optimised OSPF Configuration Example service internal int pos carrier-delay msec 0 ip dampening int gig 1/0 carrier-delay msec 0 ip dampening ospf network point-to-point negotiation auto router ospf 1 ispf timers throttle lsa all timers lsa arrival 20 timers pacing flood 15 timers throttle spf ip routing protocol purge interface process-max-time 60 ip cef linecard ipc service-timer 50 ip cef table loadinfo force ip 60

61 Optimised IS-IS Configuration Example service internal int pos carrier-delay msec 0 ip dampening int gig 1/0 carrier-delay msec 0 ip dampening isis network point-to-point negotiation auto isis tag 17 router isis ip route priority high tag 17 fast-flood ispf prc-interval spf-interval lsp-gen-interval ip routing protocol purge interface process-max-time 60 ip cef linecard ipc service-timer 50 ip cef table loadinfo force 61

62 Conclusion traffic loss (msec) c12k-prp1-eng4p-pre28s-isis-ipip-cpuload-1mppspr60-ipcs50-bgp160-ip2ip-remote-nolb Agilent measurements % % % 300 average % prefix nr 62

63 Conclusion Sub-1s is Conservatively met 500 pref, IP2IP, PRP1/Eng4+ P50 = 200ms and P100 = 350ms IOS-XR has leveraged the FC research project and took the optimized behaviours as default 63

64 Q and A 64