Scaling BGP BRKRST Luc De Ghein Technical Leader Services

Transcription

1

2 Scaling BGP Luc De Ghein Technical Leader Services

3 Agenda Goal & Introduction Scale Challenges Memory Utilization Full mesh ibgp Update Groups Slow Peer RR Problems & Solutions Deployment Multi-Session MPLS VPN OS Enhancements 3

4 We re Gonna Need a Bigger Boat" Jaws 4

5 Goal of This session Present causes of scale challenges Present solutions for scaling No scaling numbers What you can control the most: Buy a bigger box Design the network properly 5

6 Scale Challenges BGP is robust BGP is simple The basis is simple/knobs allow for anything BGP is adaptable and is multi-protocol BGP allows policy We need to overcome the following: Newer services: new AFs More prefixes Larger scale: more (BGP) routers More multipath More resilience PIC Edge, Best External leading to more prefixes/paths 6

7 Growing Tables of BGP Internet sees ~ 2 updates per sec AS IPv4 IPv6 Source: bgp.potaroo.net 7

8 More Services Using BGP future IPv4 IDR IPv4 enterprise MPLS VPN BGP FC PIC BGP flowspec services IPv6 IDR multicast 6PE mvpn 6VPE BGP FC Path Diversity BGP PW Signalling (VPLS) BGP monitoring protocol VPN4DC mvpn Auto Discovery C-signalling BGP - LS AIGP BGP MAC Signaling (EVPN) scaling Inter-AS MPLS VPN DMVPN Unified MPLS 8

9 Services = Address Families For Your Reference IPv4 IPv6 unicast multicast vpn multicast in overlay Layer 2 IPv4 unicast IPv6 unicast vpnv4 unicast nsap unicast IPv4 Flowspec IPv4 multicast IPv6 multicast vpnv4 multicast l2vpn vpls IPv6 Flowspec IPv4 MVPN IPv6 MVPN vpnv6 unicast l2vpn evpn vpnv4 Flowspec IPv4 MDT vpnv6 multicast vpnv6 Flowspec IPv4 tunnel rtfilter unicast 9

10 Memory Utilization

11 High Memory Utilization Prefixes, Paths, Attributes BGP stores Prefixes many Paths could be many per prefix Attributes could be many per path AS 200 {200} AS 10 AS 20 {10 200} AS 100 {20 200} R1#show bgp ipv4 unicast /24 Paths: (3 available, best #3, table default) from ( ) Origin IGP, localpref 100, valid, external Community: internet 200:1 200:2 200: (metric 11) from ( ) Origin IGP, metric 0, localpref 100, valid, internal Community: 100:1 100:2 no-export from ( ) Origin IGP, localpref 100, valid, external, best Community: internet 20:20 R1#show bgp ipv4 unicast paths Address Hash Refcount Metric Path 0x6C8EE i 0x6C8ED i 0x6C8EEA i {200} {20 200} 11

12 High Memory Utilization - Solutions Prefixes Aggregation Filter Partial routing tables instead of full routing tables Paths Reduce # peerings Use RRs instead of ibgp full mesh Attributes Reduce # attributes Filter (extended) communties (SOO, cost,...) Typically inbound on ebgp peering Limit your own communities Newer types (ibgp PE-CE, AIGP,...) 12

13 High Memory Utilization Soft Reconfiguration Inbound vs Route Refresh soft reconfiguration inbound route refresh AS 10 AS 20 AS 10 AS 20 inbound filter Filtered prefixes are stored: much more memory used Support only on router itself Changed filter: re-apply policy to table with filtered prefixes inbound filter Filtered prefixes are dropped Recommendation: Use soft reconfiguration inbound only on ebgp peering when you need to know what the peer previously advertised and you filtered out Support needed on peer, but this a very old feature Changed filter: router sends out route refresh request to peer to get the full table from peer again 13

14 Full Mesh ibgp

15 Full Mesh ibgp Is a Full Mesh Scalable? Per BGP standard: ibgp needs to be full mesh Total ibgp sessions = n * (n-1) / 2 Sessions per BGP speaker = n - 1 session Two solutions 1. Confederations peers 2. Route reflectors 15

16 Confederations AS 100 Create # of sub-as inside the larger confederation Conferation AS looks like normal AS to the outside subas R5 R6 subas R1 R2 R3 R4 R13 subas R8 R9 Full mesh ibgp still needed inside subas R7 subas R12 R11 R10 No full mesh needed between subas (it s ebgp) R14 Every BGP peer needs to be in a subas Each subas can have different IGP with next-hop-self within confed No connectivity needed between any subas s Flexible confed ebgp peerings Redundancy needed vs increased memory/cpu But full mesh between subas s is not needed 16 R16 R15 confed ebgp ibgp ebgp

17 Confederations Notes Loop avoidance similar to normal AS We introduce a confederation part in the AS_PATH Two new segment types: 3: AS_CONFED_SEQUENCE (ordered) ( ) 4: AS_CONFED_SET (unordered) [ ] They never go out of the Confederation PATH inside confederation For Your Reference e.g. ( ) Attributes do not change across confed-ebgp We keep NEXT_HOP, LOCAL_PREF, MED Implications: Decisions is consistent (Example: One path with higher LOCAL_PREF wins across the confederation) Since NEXT_HOP does not change, IGP should be one across the confederation Decision process does not change Prefer ebgp paths over ibgp AND confederation-ebgp 17

18 Solve The ibgp Full Mesh Route Reflector (RR) RR A route reflector is an ibgp speaker that reflects routes learned from ibgp peers to other ibgp peers, called RR clients ibgp full mesh is turned into hub-and-spoke RR is the hub in a hub-and-spoke design router bgp neighbor R1 route-reflector-client neighbor R2 route-reflector-client RR OR RR router bgp no bgp client-to-client reflection neighbor R1 route-reflector-client neighbor R2 route-reflector-client RR clients are regular ibgp peers R1 R2 R3 R1 R2 R3 If client-to-client reflection can be turned off RR clients are interconnected 18

19 Route Reflector What s Possible? ibgp ebgp AS 101 R5 AS 100 non-client Any router can peer ebgp RR RR R1 R2 R3 R4 cluster cluster clients 19

20 How Many RRs? More RRs = more redundancy But also = more management work and more memory 2 RRs for each service set is ok More is possible e.g. 2 RRs for IPv4 2 RRs for vpnv

21 Route Reflector - Cluster Redundancy needed, min of 2 RRs per cluster Cluster = RR and its clients R5 Full mesh between RRs and non-clients should be kept small RR RR RR RR R1 R2 R3 R4 Client can peer with RRs in multiple clusters 21

22 Route Reflector Hierarchical RRs Chain RRs to keep the full mesh between RRs and non-clients small Make RRs clients of other RRs An RR is a RR and RR client at the same time RR RR & RR client ibgp topology should follow physical topology Prevents suboptimal routing, blackholing, and routing loops RRs in top tier need to be fully meshed Tier 1 Tier 2 There is no limit to the amount of tiers Tier 3 22

23 Route Reflector Loop Prevention For Your Reference Because we have RRs and same prefix can be advertised multiple times within ibgp cloud: loop prevention needed in ibgp cloud Two BGP attributes, two ways Originator ID Set to the router ID of the router injecting the route into the AS Set by the RR Cluster List Each route reflector the route passes through adds their cluster-id to this list Cluster-id = Router ID by default bgp cluster-id x.x.x.x command to set cluster-id Router discard routes if: If ORIGINATOR-ID = our ROUTER-ID If CLUSTER_LIST contains our CLUSTER-ID 23

24 Route Reflector Loop Prevention For Your Reference RR1 and RR2 have different cluster-ids AS Path: {65000} Originator-ID: RRC1 Cluster-List: {RR1} RR1 RR2 AS Path: {65000} RRC1 AS Path: {65000} Originator-ID: RRC1 Cluster-List: RRC2 {RR1, RR2} AS Path: {65000} loop detected 24

25 Route Reflector Same Cluster-ID or Not? RR1 has only 1 path for routes from RRC2 RR1 and RR2 have the same cluster-id If one link RR to RR-client fails ibgp session remains up, it is between loopback IP addresses RR1 RR2 If different cluster-id: RR1 stores the path from RR2 RR1 uses additional CPU and memory Potentially for many routes RRC1 RRC2 Using the same or different cluster-id for RRs in one set? Different cluster-id Additional memory and processor overhead on RR Same cluster-id Less redundant paths 25

26 Route Reflector Route Advertisement prefix coming from ebgp peer prefix coming from RR client RR & RR client RR prefix coming from a non-client ibgp ebgp RR sends prefix to clients and non-clients (and sends to other ebgp peers) RR reflects prefix to clients and sends to non-clients (and sends to ebgp peers) RR reflects prefix to clients (and sends to ebgp peers) 26

27 The Need for Virtual Route Reflectors A dedicated RR does IGP BGP A dedicated RR needs CPU memory primary primary primary backup backup backup service 1 set RRs service 2 set RRs service 3 set RRs primary backup service 1 set vrrs service 2 set vrrs service 3 set vrrs More BGP services: dedicated RR virtualized and multiplied: vrr Benefits: Scalability (64b OS) Performance (Multi core) Independence of operation: reload/update/changes version (VM) Same BGP implementation and software version as deployed on the Edge (XE/XR) vmotion/management (Hypervisor) 27 CSR1000v (IOS-XE) 64-bit XRv (IOS-XR) 32-bit

28 Scalability / Performance Evolution of Route-Reflector s From 7200 to vrr 64b OS IOS XE - ASR1K 64b OS IOS-XR o 64b Linux ASK9K VSM with 4 vrr: 4 x 8 core CPU IOS-XR ASR9001 IOS-7200 IOS XR with 8 RP / RR 64b OS vios-xe o VMware Integration 28

29 BGP RR Scale - Selective RIB Download To block some or all of the BGP prefixes into the RIB (and FIB) Only for RR which is not in the forwarding path Saves on memory and CPU Implemented as filter extension to table-map command For AFs IPv4/6 not needed for AFs vpnv4/6 Benefit ASR testing indicated 300% of RR-client session scaling (in order of 1000s) configuration no BGP prefixes in RIB no BGP prefixes in FIB router bgp 1 address-family ipv4 table-map block-into-rib filter route-map block-into-rib deny 10 RR1#show ip route bgp RR1# RR1#show ip cef RR1# route-policy block-into-rib if destination in (...) then drop else pass end-if configuration IOS-XR router bgp 1 address-family ipv4 unicast table-policy block-into-rib 29

30 Multi-Cluster ID router bgp 1 no bgp client-to-client reflection intra-cluster cluster-id no bgp client-to-client reflection intra-cluster cluster-id An RR can belong to multiple clusters On an IBGP neighbor of RR: configure cluster IDs on a per-neighbor basis The global cluster ID is still there Intra-cluster client-to-client reflection can be disabled, when clients are meshed Can be disabled for all clusters or per cluster More work -sending more updates- for RR clients Less work -sending fewer updates- for RRs cluster ID 1 PE1 PE2 no reflection RR1 no reflection cluster ID 2 PE3 PE4 Each set of peers in cluster ID has its own update group Loop-prevention mechanism is modified Taking into account multiple cluster IDs 30

31 Comparing Confederation vs. RR For Your Reference Confederation Route Reflector Loop prevention AS Confederation Set Originator/Cluster ID Break up a single AS Sub-AS (possibly with separate IGPs) Clusters Redundancy Multiple connections between sub-as Client connects to several reflectors External Connections Anywhere in the network Anywhere in the network Multilevel Hierarchy RRs within sub-as Clusters within clusters Policy Control Scalability Along outside borders and between sub-as Dampening possible on ebgp confed Medium; still requires full ibgp within each sub-as Along outside border Very high Migration Very difficult (impossible in some situations) But easy when merging two companies Moderately easy Deployment of (new) features Decentralized Central on RR Note: Route reflectors can exist inside subas 31

32 Update Groups

33 Grouping of BGP Neighbors Optimization Can Be Achieved Configuration/administration Performance/scalability peer groups templates, session-groups, af-groups, neighbor-group CLI only BGP neighbors with same outbound policy will be put in the same update group regardless if peer-groups are defined templates are defined neighbors are individually defined update groups Dynamic grouping BGP of peers according to common outbound policy networks that have the same best-path attributes can be grouped into the same message improving packing efficiency BGP formats the update messages once and then replicates to all members of the update group replication instead of formatting updates per neighbor: efficiency dynamic = policy changes, update group membership changes AF independent : a peer can belong to different update groups in different address families 33

34 Peer-groups vs Templates peer-groups router bgp 1 neighbor ibgp-peers peer-group neighbor ibgp-peers remote-as 1 neighbor ibgp-peers description peering to ibgp speakers neighbor ibgp-peers update-source Loopback0 neighbor peer-group ibgp-peers neighbor peer-group ibgp-peers! address-family ipv4 no neighbor activate no neighbor activate exit-address-family! address-family vpnv4 neighbor ibgp-peers send-community both neighbor activate neighbor activate exit-address-family Oldest configuration peer policy for policy commands (AD dependent) peer session for global/session commands (AF independent) templates router bgp 1 template peer-policy ibgp-policy send-community both inherit peer-policy nhs 10 exit-peer-policy! template peer-policy nhs next-hop-self exit-peer-policy! template peer-session ibgp-peers remote-as 1 description peering to ibgp speakers update-source Loopback0 exit-peer-session Newest configuration nesting possible! neighbor inherit peer-session ibgp-peers address-family vpnv4 neighbor activate neighbor send-community extended neighbor inherit peer-policy ibgppolicy exit-address-family More flexible (nesting, multiple inheritance) 34 Less configuration

35 Update Groups and Route Reflectors Update groups are very usefull on all BGP speakers but mostly on RR due to # of peers equal outbound policy ibgp typically has no outbound policy RRs have large number of ibgp peers in one update group Turn the issue into a solution Create one BGP update once and replicate for all Assuming RR does not change attributes on ibgp sessions a peer can belong to different update groups in different address families RR#show bgp ipv4 unicast neighbors For address family: IPv4 Unicast Session: BGP table version 3201, neighbor version 3201/0 Output queue size : 0 Index 2, Advertise bit 0 Route-Reflector Client 2 update-group member For address family: VPNv4 Unicast Session: BGP table version 1, neighbor version 1/0 Output queue size : 0 Index 3, Advertise bit 0 3 update-group member 35

36 Update Group on RR RR#show bgp ipv4 unicast update-group 2 BGP version 4 update-group 2, internal, Address Family: IPv4 Unicast BGP Update version : 3201/0, messages 0 Route-Reflector Client Topology: global, highest version: 3201, tail marker: 3201 Format state: Current working (OK, last not in list) Refresh blocked (not in list, last not in list) Update messages formatted 2013, replicated 24210, current 0, refresh 0,limit 2000 RR Number of NLRIs in the update sent: max 812, min 0 Minimum time between advertisement runs is 0 seconds Has 101 members:

37 Update Group Replication RR IOS format replicate 1 BGP update BGP update BGP update BGP update BGP update... n BGP update RR#show ip bgp replication 2 Current Next Index Members Leader MsgFmt MsgRepl Csize Version Version / /0 update group 2 total # of members formatting according to leader s policy # of formatted messages # of replications 37 size of cache

38 Adaptive Message Cache Size Cache = place to store formatted BGP message, before they are send Update message cache size throttles update groups during update generation and controls transient memory usage Is now adaptive Variable (change over time) queue depth from 100 to 5000 Number of peers in an update groups Installed system memory Type of address family Type of peers in an update group Benefits Update groups with large number of peers get larger update cache Allows routers with bigger system memory to have *appropriately* bigger cache size and thereby queue more update messages vpnv4 ibgp update groups have larger cache size Old cache sizing scheme could not take advantage of expanded memory available on new platforms Results in faster convergence IOS 38

39 Route-Refresh Update Group: When is a Route Refresh Request Sent? For Your Reference A route refresh request is sent, when: a user types clear ip bgp [AF] {* peer} in a user types clear ip bgp [AF] {* peer} soft in adding or changing the inbound filtering on the BGP neighbor via route-map configuring allowas-in for the BGP neighbor configuring soft-configuration inbound on the BGP neighbor in MPLS VPN (for AFI/SAFI 1/128) a user adds a route-target import to a VRF in 6VPE (for AFI/SAFI 2/128) a user adds a route-target import to a VRF 39

40 Parallel Processing of Route-refresh (and New Peers) Refresh Group Re-announcements Transient Updates Version 0 Original update group handles new transient updates while refresh update group handles re-announcements IOS Serving all peers for which route-refresh is needed, at once Tracking the refreshing peers Maintains a copy of neighbor instance that needs to re-announce the table Allows original update group and its members to advertise transient updates without getting blocked Tracked with flags SPE flags to indicate refresh E flag : refresh end marker. Peer is scheduled or participating in a refresh. S flag : refresh has started. If not present: waiting for its refresh to start either because another refresh for the same group is in progress or because the net prepend is not yet complete P flag : refresh is paused waiting for other refresh members that started later to catch up IOS-XR Refresh sub-goup Table Versions of Prefixes Version X 40

41 Parallel Processing of Route-Refresh For Your Reference RR#show bgp ipv4 unicast update-group 1 summary Summary for Update-group 1, Address Family IPv4 Unicast Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd :32: :00:09 0 (SE) :33: :32: RR#show bgp ipv4 unicast update-group 1 BGP version 4 update-group 1, internal, Address Family: IPv4 Unicast BGP Update version : 1/100001, messages 1999 Topology: global, highest version: , tail marker: RR#show bgp ipv4 unicast replication 1 Current Next Index Members Leader MsgFmt MsgRepl Csize Version Version /2000 1/ Format state: Current working (OK, last no message space) Refresh blocked (no message space, last no message space) Update messages formatted 59997, replicated 89997, current 0, refresh 1999, limit 2000 Has 4 members:

42 Update Groups in IOS XR IOS-XR RP/0/6/CPU0:replanet#show bgp vpnv4 unicast update-group address family update groups sub-groups refresh sub-groups filter groups neighbors Update group for VPNv4 Unicast, index 0.2: Attributes: Internal Common admin First neighbor AS: 1 Send communities Send extended communities Route Reflector Client 4-byte AS capable Send AIGP Minimum advertisement interval: 0 secs Update group desynchronized: 0 Sub-groups merged: 5 Number of refresh subgroups: 0 Messages formatted: 36, replicated: 68 All neighbors are assigned to sub-group(s) Neighbors in sub-group: 0.2, Filter-Groups num:3 Neighbors in filter-group: 0.3(RT num: 3) Neighbors in filter-group: 0.1(RT num: 3) Neighbors in filter-group: 0.2(RT num: 3)

43 Update Groups in IOS XR RP/0/6/CPU0:replanet#show bgp vpnv4 unicast update out update-group 0.2 VRF "default", Address-family "VPNv4 Unicast" IOS-XR address family update groups sub-groups refresh sub-groups filter groups neighbors show bgp... replication in IOS equivalent Update-group 0.2 Flags: 0x b Sub-groups: 1 (0 throttled) Refresh sub-groups: 0 (0 throttled) Filter-groups: 3 Neighbors: 3 (0 leaving) Update OutQ: 0 bytes (0 messages) Update generation recovery pending? [No] Last update timer start: Apr 3 08:44: Last update timer stop: --- Last update timer expiry: Apr 3 08:44: (1w4d ago) Update timer running? [No] (0.000 sec remaining; last started for sec) History: Update OutQ Hi: 3600 bytes (5 messages) Update OutQ Cumulative: bytes (54 messages) Update OutQ Discarded: 0 bytes (0 messages) Update OutQ Cleared: 0 bytes (0 messages) Last discarded from OutQ: --- (never) Last cleared from OutQ: --- (never) Update generation throttled 0 times, last event --- (never) Update generation recovered 0 times, last event --- (never) Update generation mem alloc failed 0 times, last event --- (never) 43

44 Update Groups in IOS XR RP/0/6/CPU0:replanet#show bgp vpnv4 unicast update-group 0.2 performance-statistics IOS-XR address family update groups sub-groups refresh sub-groups filter groups neighbors Update group for VPNv4 Unicast, index 0.2: Attributes: Internal Common admin First neighbor AS: 1 Send communities Send extended communities Route Reflector Client 4-byte AS capable Send AIGP Minimum advertisement interval: 0 secs Update group desynchronized: 0 Sub-groups merged: 5 Number of refresh subgroups: 0 Messages formatted: 36, replicated: 68 All neighbors are assigned to sub-group(s) Neighbors in sub-group: 0.2, Filter-Groups num:3 Neighbors in filter-group: 0.3(RT num: 3) Neighbors in filter-group: 0.1(RT num: 3) Neighbors in filter-group: 0.2(RT num: 3) Updates generated for 0 prefixes in 26 calls(best-external:0) (time spent: secs) Update timer last started: Apr 3 08:44: Update timer last stopped: not set Update timer last processed: Apr 3 08:44:

45 Slow Peer

46 Slow Peer protection phase recovery phase convergence speed of update goup slow update group is no longer slow OK slow update group update group 1 RR detection phase Track peer queue & TCP slow peer = cannot keep up with the rate at which we are generating update messages over a prolonged period of time (order of minutes) filled up cache: blocking all peers Possible causes High CPU Transport issues (packet loss/loaded links/tcp) %BGP-5-SLOWPEER_DETECT: Neighbor IPv4 Unicast has been detected as a slow peer %BGP-5-SLOWPEER_RECOVER: Slow peer IPv4 Unicast has recovered Allows for fast and slow peers to proceed at the their own speed 46

47 Slow Peer - Configuration For Your Reference Phase Where? Command detection per AF/per VRF bgp slow-peer detection [threshold <seconds>] default is 5 min per peer neighbor {<nbr-addr>/<peer-grp-name>} slow-peer detection [threshold < seconds >] neighbor {<nbr-addr>/<peer-grp-name>} disable per peer policy template slow-peer detection [threshold < seconds >] protection static per neighbor/ per peer-group static via peer policy template automatic per AF/ per VRF automatic per neighor automatic per peer policy template neighbor {<nbr-addr>/<peer-grp-name>} slow-peer split-update-group static slow-peer split-update-group static bgp slow-peer split-update-group dynamic [permanent] neighbor {<nbr-addr>/<peer-grp-name>} slow-peer split-update-group dynamic [permanent] slow-peer split-update-group dynamic [permanent] 47 permanent = peer is not moved back automatically to the update group

48 Old Slow Peer Solution Solution before this feature: manual movement Create a different outbound policy for the slow peer Policy must be different than any other You do not want the slow peer to move to another already existing update group Use something that does not affect the actual policy For example: change minimum advertisement interval (MRAI) of the peer (under AF) 48

49 Slow Peers: Displaying/Clearing For Your Reference CLI to display slow peers Applicable to all address families show bgp all summary slow show bgp ipv4 unicast neighbors slow show bgp ipv4 unicast update-group summary slow This is a forced clear of the slow-peer status; the peer is moved to the original update group Needed when the permanent keyword is configured clear bgp AF {unicast multicast} * slow clear bgp AF {unicast multicast} <AS number> slow clear bgp AF {unicast multicast} peer-group <group-name> slow clear bgp AF {unicast multicast} <neighbor-address> slow 49

50 Slow Peer Mechanism Details Clearing For Your Reference CLI to clear This is a forced clear of the slow-peer status; the peer is moved to the original update group Needed when the permanent keyword is configured clear bgp AF {unicast multicast} * slow clear bgp AF {unicast multicast} <AS number> slow clear bgp AF {unicast multicast} peer-group <group-name> slow clear bgp AF {unicast multicast} <neighbor-address> slow 50

51 Slow Peer Mechanism Details Identifying Slow Peer For Your Reference RR#show bgp ipv4 unicast update-group 1 summary Summary for Update-group 1, Address Family IPv4 Unicast BGP router identifier , local AS number 1 BGP table version is , main routing table version network entries using bytes of memory BGP using total bytes of memory BGP activity /15574 prefixes, / paths, scan interval 60 secs convergence is achieved if all peers are at the table version Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd :56: :23: :56: :01: output queue is not empty, persistently? 51

52 Slow Peer Mechanism Details Identifying Slow Peer For Your Reference RR#show bgp ipv4 unicast replication 1 Current Next Index Members Leader MsgFmt MsgRepl Csize Version Version / / RR#show bgp ipv4 unicast update-group 1 BGP version 4 update-group 1, internal, Address Family: IPv4 Unicast BGP Update version : /500001, messages 2000 Topology: global, highest version: , tail marker: (pending) Format state: Current blocked (no message space, last no message space) Refresh blocked (not in list, last not in list) Update messages formatted 78115, replicated , current 2000, refresh 0, limit 2000 Minimum time between advertisement runs is 0 seconds Has 4 members:

53 RR Problems & Solutions

54 BGP Best Path Selection For Your Reference Path selection mechanism Weight Details This is a Cisco-defined attribute that is assigned locally to your router and does not get carried through to the router updates. If there are multiple paths to a particular IP address (which is very common), then BGP looks for the path with the highest weight. There are several ways to set the weight parameter, such as the neighbor command, the as-path access list, or route maps. Local Preference This is an indicator to the AS as to which path has local preference, with the highest preference being preferred. The default is 100. Network or Aggregate This criterion prefers the path that was locally originated via a network or aggregate. The aggregation of specific routes into one route is very efficient and saves space on your network. Shortest AS_PATH Lowest origin type Lowest multi-exit discriminator (MED) ebgp over ibgp Lowest IGP metric Multiple paths External paths Lowest router ID Minimum cluster list Lowest neighbor address BGP uses this one only when there is a tie comparing weight, local preference, and locally originated vs. aggregate addresses. This deals with protocols such as Interior Gateway Protocol (IGP) being a lower preference than Exterior Gateway Protocol (EGP). This is also known as the external metric of a route. A lower MED value is preferred over a higher value Similar to lowest origin type, BGP AS Path prefers ebgp over ibgp This criterion prefers the path with the lowest IGP metric to the BGP next hop. This determines if multiple paths require installation in the routing table. When both paths are external, it prefers the path that was received first (the oldest one). This prefers the route that comes from the BGP router with the lowest router ID. If the originator or router ID is the same for multiple paths, it prefers the path with the minimum cluster list length. This prefers the path that comes from the lowest neighbor address 54

55 Best Path Selection Route Advertisement on RR ingress PE does not learn 2nd path NH: PE1, P: Z P: Z Path 1: NH: PE1, best Path 2: NH: PE2 P: Z Path 1: NH: PE1, best P:Z CE1 PE1 NH: PE1, P: Z RR PE3 CE2 PE2 NH: PE2, P: Z The BGP4 protocol specifies the selection and propagation of a single best path for each prefix If RRs are used, only the best path will be propagated from RRs to ingress BGP speakers This behavior results in number of disadvantages for new applications and services The next slides explain features to distribute redundant paths to get around the filtering by the BGP Bestpath Calculation & Advertisement 55

56 Why Having Multiple Paths? Convergence BGP Fast Convergence (2+ paths in local BGP DB) BGP PIC Edge (2+ paths ready in forwarding plane) Multipath load balancing ECMP Allow hot potato routing = use optimal route The optimal route is not always known on the border routers Prevent oscillation The additional info on backup paths leads to local recovery as opposed to relying on ibgp Stop persistent route oscillations caused by comparison of paths based on MED in topologies where route reflectors or the confederation structure hide some paths 56

57 Diverse BGP Path Distribution Overview VPN unique RD BGP Best External BGP shadow RR / session BGP Add-Path 57

58 Diverse BGP Path Distribution Unique RD for MPLS VPN P:Z CE1 RD1 VRF RD2 VRF PE1 PE2 NH: PE1, P: Z/RD1 NH: PE2, P: Z/RD2 RR1 NH: PE1, P: Z/RD1 NH: PE2, P: Z/RD2 PE3 VRF P: Z Path 1: NH: PE1 Path 2: NH: PE2 CE2 Unique RD per VRF per PE One IPv4 prefix in one VRF becomes unique vpnv4 prefix per VPN per PE RR advertises everything Available since the beginning of MPLS VPN, but only for MPLS VPN 58

59 Diverse BGP Path Distribution Shadow Route Reflector (aka RR Topologies) NH: PE1, P: Z P: Z Path 1: NH: PE1, best Path 2: NH: PE2 NH: PE1, P: Z P: Z Path 1: NH: PE1 Path 2: NH: PE2 PE1 P:Z CE1 RR1 PE3 CE2 PE2 RR2 shadow RR NH: PE2, P: Z NH: PE2, P: Z P: Z Path 1: NH: PE1, best Path 2: NH: PE2, 2nd best router bgp 1 address-family ipv4 bgp additional-paths select backup neighbor advertise diverse-path backup Easy deployment One additional shadow RR per cluster RR2 does announce the 2nd best path, which is different from the primary best path on RR1 by next hop 59

60 Diverse BGP Path Distribution Shadow Route Reflector Note: primary RRs do not need diverse path code equal distance P: Z Path 1: NH: PE1, best Path 2: NH: PE2 RR and shadow RR are co-located. They re on same vlan with same IGP metric towards prefix. P:Z PE1 PE2 P RR1 RR2 P: Z Path 1: NH: PE1, best Path 2: NH: PE2, 2nd best shadow RR Note: primary and shadow RRs do not need to turn off IGP metric check P: Z Path 1: NH: PE1, best Path 2: NH: PE2 RR and shadow RR are not co-located. P:Z PE1 PE2 RR1 P RR2 all links have the same IGP cost P: Z Path 1: NH: PE1, 2nd best Path 2: NH: PE2, best shadow RR Note: primary and shadow RRs need to turn IGP metric check off. All RRs to calculate the same best path so that primary and shadow RRs do not advertise the same path solution RR(config-router-af)#bgp bestpath igp-metric ignore 60 RR2 advertises same path as RR1!

61 Diverse BGP Path Distribution Shadow Route Reflector For Your Reference The shadow RR needs to (1) calculate and (2) advertise the backup path Calculation: triggered by any of the following per address family CLI Backup path calculation bgp additional-paths select backup Multipath calculation (existing CLI) maximum-paths (e)ibgp <n> Backup path calculation AND the installation can be triggered by the following CLI if bgp additionalpaths select backup is NOT configured already bgp additional-path install Advertisement of the backup path Backup path advertisement advertise diverse-path backup shadow RR router bgp 1 address-family vpnv4 bgp additional-paths select backup neighbor activate neighbor send-community both neighbor route-reflector-client neighbor advertise diverse-path backup example config 61

62 Diverse BGP Path Distribution Shadow Route Reflector - Advertised-Routes For Your Reference on the shadow RR RR2#show bgp ipv4 unicast neighbors advertised-routes BGP table version is 10, local router ID is Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, Filter, r RIB-failure, S Stale, m multipath, b backup-path, f RT- x best-external, a additional-path, c RIB-compressed, Origin codes: i - IGP, e - EGP,? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path *bia / i *bia / i *>i / i RR2#show bgp ipv4 unicast /24 BGP routing table entry for /24, version 7 Paths: (3 available, best #3, table default) Advertised to update-groups: Refresh Epoch , (Received from a RR-client) (metric 3) from ( ) Origin IGP, metric 0, localpref 100, valid, internal, backup/repair rx pathid: 0, tx pathid: 0 Refresh Epoch , (Received from a RR-client) (metric 3) from ( ) Origin IGP, metric 0, localpref 100, valid, internal, best rx pathid: 0, tx pathid: 0x0 this path is advertised *>i / i * valid no > not the best path B backup-path I internal A additional-path 62

63 Diverse BGP Path Distribution Shadow Session Note: second session from RR to RR-client (PE3) has diversepath command in order to advertise 2nd best path NH: PE1, P: Z P: Z Path 1: NH: PE1, best Path 2: NH: PE2, 2nd best NH: PE1, P: Z P: Z Path 1: NH: PE1 Path 2: NH: PE2 PE1 P:Z CE1 CE1 RR PE3 CE2 PE2 NH: PE2, P: Z NH: PE2, P: Z Easy deployment only RR needs diverse path code and new ibgp session per each extra path (CLI knob in RR1) Shadow ibgp session does announce the 2nd best path 2 nd session between a pair of routers is no issue (use different loopback interfaces) 63

64 BGP PIC (Prefix Independent Convergence) Edge Problem Convergence in flat FIB is prefix dependent More prefixes -> more convergence time Classical convergence (flat FIB) Routing protocols react - update RIB - update CEF table (for affected prefixes) Time is proportional to # of prefixes Result Improved convergence Reduce packet loss Have the same convergence time for all BGP prefixes (PIC) Solution The idea of PIC: In both SW and HW: Pre-install a backup path in RIB Pre-install a backup path in FIB Pre-install a backup path in LFIB 64

65 MPLS VPN Dual Homed CE No PIC Edge NH: PE1, P: Z P: Z Path 1: NH: PE1, best Path 2: NH: PE2 P:Z CE1 PE1 PE3 CE2 PE2 NH: PE2, P: Z Steps in convergence on ingress PE 1. Egress PE goes down 2. IGP notifies ingress PE in sub-second 3. Ingress PE recomputes BGP bestpath 4. Ingress PE installs new BGP bestpath in RIB 5. Ingress PE installs new BGP bestpath in FIB 6. Ingress PE reprograms hardware 65

66 MPLS VPN Dual-Homed CE PIC Edge NH: PE1, P: Z P: Z Path 1: NH: PE1, best Path 2: NH: PE2, backup/repair router bgp 1 address-family vpnv4 bgp additional-paths install P:Z CE1 PE1 PE3 CE2 PE2 NH: PE2, P: Z We eliminate convergence dependance on: Scanning of the BGP table Bestpath calculation (because there is a pre-computed backup/repair path) Time to generate and propagate updates (PE and RR) Updating the FIB (with PIC the FIB update is prefix independent) Steps in convergence on ingress PE 1. Egress PE goes down 2. IGP notifies ingress PE in sub-second 3. Switch to repair path with new NH 4. Ingress PE reprograms hardware this scales to the number of prefixes 66

67 No BGP Best External Default BGP Policy P: Z Path 1: NH: CE1, localpref 100, external, best P: Z Path 1: NH: PE1, internal, localpref 100, best Path 2: NH: PE2, internal, localpref 100, backup/repair NH: PE1, localpref: 100, P: Z P:Z CE1 NH: PE1, localpref: 100, P: Z PE1 NH: PE2, localpref: 100, P: Z PE3 CE2 PE2 NH: PE2, localpref: 100, P: Z P: Z Path 1: NH: CE1, localpref 100, external, best 67

68 No BGP Best External Changed BGP Policy If default policy is changed, one egress PE could have ibgp path to other egress PE as best path and not its own external BGP path P: Z Path 1: NH: CE1, external, best P: Z Path 1: NH: PE1, internal, localpref 200, best local preference 200 NH: PE1, localpref: 200, P: Z P:Z CE1 NH: PE1, localpref: 200, P: Z PE1 NH: PE2, localpref: 100, P: Z PE3 CE2 no backup/repair path PE2 P: Z Path 1: NH: CE1, localpref 100, external, best Path 2: NH: PE1, localpref: 200, internal, best Even with full mesh in ibgp, policy can prevent egress PE from learning all paths 68

69 BGP Best External Changed BGP Policy P: Z Path 1: NH: CE1, external, best Path 2: NH: PE2, localpref 100, internal, backup/repair P: Z Path 1: NH: PE1, internal, localpref 200, best Path 2: NH: PE2, localpref 100, internal, backup/repair local preference 200 NH: PE1, localpref: 200, P: Z P:Z CE1 NH: PE1, localpref: 200, P: Z PE1 NH: PE2, localpref: 100, P: Z PE3 CE2 PE2 NH: PE2, localpref: 100, P: Z router bgp 1 address-family vpnv4 bgp additional-paths install bgp additional-paths select best-external neighbor x.x.x.x advertise best-external P: Z backup/repair, advertise-best-external Path 1: NH: CE1, external, best Path 1: NH: PE1, localpref: 200, internal, best With Best External, the backup PE (PE2) still propagates its own best external path to the RRs or ibgp peers PE1 and PE3 have 2 paths 69

70 ADD Path P: Z Path 1: NH: PE1, best Path 2: NH: PE2, best2 router bgp 1 address-family ipv4 bgp additional-paths select best 2 bgp additional-paths send neighbor PE3 advertise additional-paths best 2 P:Z CE1 PE1 NH: PE1, P: Z NH: PE1, P: Z P: Z Path 1: NH: PE1, best Path 2: NH: PE2, backup/repair RR PE3 CE2 CE2 PE2 NH: PE2, P: Z Add-Path will signal diverse paths from 2 to X paths NH: PE2, P: Z RRs and edge BGP speakers need to have Add-Path support Add-Path capability (send or receive or both) means that the NRLI encoding is extended to carry the Path Identifier router bgp 1 address-family ipv4 bgp additional-paths receive bgp additional-paths install PE routers need to run newer code in order to understand second path 70

71 BGP Add Path draft-ietf-idr-add-paths-guidelines defines 4 flavours of Add-x-Path 2 options are implemented by Cisco add-n-path RR will do best path computation for up to n paths and send n paths to the border routers This is the only mandatory selection mode Pros less storage used for paths less BGP info exchanged Cons more best path computation Usecase: Primary + n-1 backup scenario (n is limited to 3 (IOS) or 2 (IOS-XR), to preserve CPU power) = fast convergence bgp additional-paths select best<n> add-all-path RR will do the first best path computation and then send all paths to the border routers Pros all paths are available on border routers Cons all paths stored more BGP info is exchanged Usecase: ECMP, hot potato routing bgp additional-paths select all 71

72 Add-Path IOS-XR Path selection is configured in a routepolicy Global command, per address family, to turn on add-path in BGP Configuration in VPNv4 mode applies to all VRF IPv4-Unicast AF modes unless overridden at individual VRFs example config router bgp 1 address-family vpnv4 additional-paths install backup (deprecated) additional-paths advertise additional-paths receive additional-paths selection route-policy apx example RPL config route-policy ap1 if community matches-any (1:1) then set path-selection backup 1 install elseif destination in ( /16, /16) then set path-selection backup 1 advertise install endif end-policy route-policy ap2 set path-selection all advertise end-policy route-policy ap3 set path-selection multipath advertise end-policy 72

73 Cost of ADD-Path Memory overhead Additional memory overhead on the receiving PE due to additional paths Additional memory overhead of maintaining per path Adj-Rib-Out information CPU cycle increase for update processing Update reception at edge routers increases proportional to #additional paths Update generation at aggregators also increases proportional to #additional paths CPU cycle increase for other internal processing as well e.g. Next-hop trigger 73

74 Hot-Potato Routing NO RR Hot-potato routing = packets are passed on (to next AS) as soon as received Shortest path though own AS must be used In transit AS: same prefix could be announced many times from many ebgp peers ebgp: P: Z NH: PE1, P: Z ebgp: P: Z PE3 NH: PE3, P: Z P: Z Path 1: NH: PE1 Path 2: NH: PE2 Path 3: NH: PE3, best PE1 ebgp: P: Z PE4 PE2 NH: PE2, P: Z 74

75 Hot-Potato Routing With RR Introducing RRs break hot potato routing Solutions: Unique RD for MPLS VPN or Add Path P: Z Path 1: NH: PE1, best Path 2: NH: PE2 Path 3: NH: PE3 ebgp: P: Z ebgp: P: Z NH: PE1, P: Z PE3 P: Z Path 1: NH: PE1, best PE1 NH: PE3, P: Z ebgp: P: Z PE2 RR NH: PE2, P: Z NH: PE1, P: Z PE4 75

76 Hot-Potato Routing in Large Transit SP No ADD- Path PE PE PE RR RR PE PE RR RR PE PE RR RR PE PE Often large transit ISPs do have full mesh ibgp between regional RRs and hub/spoke between local BR and RR Full mesh and global hot potato routing 76

77 Hot-Potato Routing in Large Transit SP With ADD- Path PE PE PE RR RR PE PE RR RR RR PE PE RR RR PE PE With introduction of add-all-path we now could have centralized RR and do hot-potato routing Initially, add-all-path could be deployed between centralized and regional RR s Also possible: remove the need for regional RR if all PE routers support add-path 77

78 Don t Forget For Your Reference If you want primary and backup path installed in the RIB of the ingress PE (ECMP), you must have BGP multipath enabled But multipath does not solve the issue of RR only sending best path With full mesh ibgp (and default policy), you can avoid all of the above, possible only needing/wanting BGP multipath on ingress PE With different RD per PE for one VRF, you can avoid all of the above, but only in MPLS VPN networks Inter-AS ASBRs also perform Bestpath calculation and advertise only the bestpath across the ASborder / into the local AS Consequently, the same requirements apply to Inter-AS as Intra-AS Unique RD, Add-Path, Best-external and/or RR-topologies 78

79 Deployment

80 Path MTU Discovery (PMTUD) MSS (Max Segment Size) Limit on the largest segment that can traverse a TCP session Anything larger must be fragmented & re-assembled at the TCP layer MSS is 536 bytes by default for client BGP without PMTUD Enable PMTU for BGP with Older command ip tcp path-mtu-discovery Newer command bgp transport path-mtu-discovery (PMTUD now on by default) 536 bytes is inefficient for Ethernet (MTU of 1500) or POS (MTU of 4470) networks TCP is forced to break large segments into 536 byte chunks Adds overheads Slows BGP convergence and reduces scalability Another helpful command show ip bgp neighbors include max data 80

81 Session/Timers Timers = keepalive and holdtime Default is ok Smallest is 3/9 for keepalive/holdtime Scaling <> small timers Use BFD Built for speed When failure occurs, BFD notifies BFD client (in 10s of msecs) Do not use Fast Session Deactivation (FSD) Tracks the route to the BGP peer A temporary loss of IGP route, will kill off the ibgp sessions Very dangerous for ibgp peers IGP may not have a route to a peer for a split second FSD would tear down the BGP session It is off by default neighbor x.x.x.x fall-over Next Hop Tracking (NHT), enabed by default, does the job fine BFD Clients BFD Clients OSPF IS-IS EIGRP BF D BFD Control Packets BFD OSPF IS-IS EIGRP BGP BGP 81

82 Multisession

83 Multisession BGP Multisession = multiple BGP (TCP) sessions between 2 BGP speakers Even if there is only one BGP neighbor statement defined between the BGP speakers in the configuration) R1 single session carrying all AFs R2 Introduced with Multi Topology Routing (MTR) One session per topology multisession 1 topo per session R1 R2 Now: possibility to have one session per AF/group of AFs Good for incremental deployment of AFs Avoids a BGP reset But multisession needs to be enabled beforehand Good for troubleshooting Good for issues when BGP session resets For example malformed update Not so good for scalability IOS only and not enabled by default R1 multisession 1 AF per session R2 83

84 Multisession For Your Reference capability multisession for MTR multisession without MTR BGP: passive rcvd OPEN w/ optional parameter type 2 (Capability) len 3 BGP: passive OPEN has CAPABILITY code: 131, length 1 BGP: passive OPEN has MULTISESSION capability, without grouping R2#show bgp ipv4 unicast neighbors BGP neighbor is , remote AS 1, internal link BGP multisession with 3 sessions (3 established), first up for 00:05:43 Neighbor sessions: 3 active, is multisession capable Session: session 1 Topology IPv4 Unicast Session: session 2 Topology IPv4 Unicast voice Session: session 3 Topology IPv4 Unicast video R2#show ip bgp neighbors include session address family BGP multisession with 3 sessions (3 established), first up for 00:02:29 Neighbor sessions: 3 active, is multisession capable Session: session 1 Session: session 2 Session: session 3 Route refresh: advertised and received(new) on session 1, 2, 3 Multisession Capability: advertised and received For address family: IPv4 Unicast Session: session 1 session 1 member For address family: IPv6 Unicast Session: session 2 session 2 member For address family: VPNv4 Unicast Session: session 3 session 3 member 1 session per topology 1 session per address family 84

85 Multisession Conclusion Increases # of TCP sessions Not really needed Current default behavior = multisession is off Can be turned on by neighbor x.x.x.x transport multi-session Makes sense to have IPv4 and IPv6 on seperate TCP sessions IPv6 over IPv4 (or IPv4 over IPv6) can be done, but next hop mediation is needed 85

86 MPLS VPN

87 RR-groups Use one RR (set of RRs) for a subset of prefixes By carving up range of RTs Only for vpnv4/6 RR only stores and advertises the specific range of prefixes Less storage on RR, but more RRs needed + more peerings rr-group 1 RR1 vpnv4/6 RR2 vpnv4/6 vpnv4/6 PE1 rr-group 2 PE2 vpnv4/6 RR1 RR2 87

88 RR-groups Configuration Example Dividing of RTs done by simple ext community list 1-99 or ext community list with regular expression PEs still send all vpnv4/6 prefixes to RR, but RR filters them rr-group 1 RR1 address-family vpnv4 bgp rr-group 100 address-family vpnv6 bgp rr-group 100 ip extcommunity-list 100 permit RT:1:(1 3 5)... vpnv4/6 RR2 vpnv4/6 vpnv4/6 PE1 rr-group 2 PE2 vpnv4/6 Dividing RT = more work PEs are not involved, only RRs RR1 RR2 address-family vpnv4 bgp rr-group 100 address-family vpnv6 bgp rr-group 100 ip extcommunity-list 100 permit RT:1:(2 4 6)... BGP(4): rcvd UPDATE w/ attr: nexthop , origin?, localpref 100, metric 0, extended community RT:1:10001 BGP(4): rcvd 1:10001: /32, label DENIED due to: extended community not supported; 88

89 Route Target Constraint (RTC) Current behavior: RR sends all vpnv4/6 routes to PE PE routers drops vpnv4/6 for which there is no importing VRF RTC behavior: RR sends only wanted vpnv4/6 routes to PE wanted : PE has VRF importing the Route Targets for the specific routes RFC 4684 New AF rtfilter Received RT filters from neighbors are translated into oubound filtering policies for vpnv4/6 prefixes The RT Filtering information is obtained from the VPN RT import list Result: RR do not send vpnv4/6 unnecessarily prefixes to the PE routers 89

90 Route Target Constraint (RTC) CE1 CE2 PE1 RR PE2 CE3 CE4 BGP capability exchange OPEN message capability 1/132 (RTFilter) for vpnv4 & vpnv6 AF RTFilter exchange MP_REACH_NLRI AF RTFilter origin AS Route Target 1:2 origin AS Route Target 1:1 origin AS Route Target 0:0 PE1 installs Default RT filter for RR RR installs RT filter RT:1:1 & RT 1:2 for PE1 (implicitly denying all else) AF vpnv4/6 prefixes exchange PE sends all its vpnv4/6 prefixes to RR 90 RR sends only RED and green (not blue) vpnv4/6 prefixes to PE1

91 Route Target Constraint (RTC) Results Eliminates the waste of processing power on the PE and the waste of bandwidth Number of vpnv4 formatted message is reduced by 75% BGP Convergence time is reduced by 20-50% The more sparse the VPNs (few common VPNs on PEs), the more performance gain Note: PE and RR need the support for RTC Incremental deployment is possible (per PE) Behavior towards non-rt Constraint peers is not changed Notes RTC clients of RR with different set of importing RTs will be in the same update group on the RR In IOS-XR, different filter group under same subgroup 91

92 Legacy PE RT Filtering Problem: If one PE does not support RTC (legacy prefix), then all RRs in one cluster must store and advertise all vpn prefixes to the PE Solution: Legacy PE sends special prefixes to mimic RTC behavior, without RTC code Legacy PE Collect import RTs Create route-filter VRF (same RD for all these VRFs across all PEs) Originate special route-filter route(s) with the import RTs attached one of 4 route-filter communties NO-ADVERTISE community RR The presence of the community triggers the RR to extract the RTs and build RT membership information RR only advertises wanted vpn prefixes towards legacy PE 4 route-filter communties 0xFFFF0002 ROUTE_FILTER_TRANSLATED_v4 0xFFFF0003 ROUTE_FILTER_v4 0xFFFF0004 ROUTE_FILTER_TRANSLATED_v6 0xFFFF0005 ROUTE_FILTER_v6 92

93 Import RT 1:1 CE1 Legacy PE CE2 Import RT 1:1 Import RT 1:2 CE3 PE1 RTC RR no RTC PE2 CE4 Import RT 1:3 for vpnv4 for vpnv6 Route-filter VRF Export RT 1:1 1:3 legacy PE sends route-filter VRF route(s) with unique RD, route-filter community and importing RTs 9999:9999: /32 Community: Extended Community: RT: :1 RT: :3 no-export no-advertise RD:prefix One of 4 route-filter communities All import RTs of the legacy PE NO-ADVERTISE NO-EXPORT community AF vpnv4/6 prefixes exchange PE1 sends all its vpnv4/6 prefixes to RR RR sends only RED (not green) vpnv4/6 prefixes to PE2 93

94 Legacy PE RT Filtering - Configuration Legacy PE config RR config router bgp 1 address-family vpnv4 neighbor route-reflector-client neighbor accept-route-legacy-rt ip vrf route-filter rd 9999:9999 export map SET_RT router bgp 1 address-family vpnv4 neighbor route-map legacy_pe out address-family ipv4 vrf route-filter network mask ip route vrf route-filter Null0 ip prefix-list match_rt_1 seq 5 permit /32 RR new code PE2 old code CE2 CE4 Import RT 1:1 Import RT 1:3 route-map SET_RT permit 10 match ip address prefix-list match_rt_1 set community (equals 0xFFFF0002) set extcommunity rt : :3 additive route-map legacy_pe permit 10 match ip address prefix-list match_rt_1 set community no-export no-advertise additive Route-filter VRF Export Map 1:1 1:3 94

95 VRF-Based Dampening BGP route dampening is now configurable per-vrf instead of for whole VPN table Allows service provider to configure dampening parameters on an individual customer basis Gives operators more flexible control of unstable customer routes in service provider network router bgp <asn> address-family ipv4 [unicast multicast ] vrf vrf-name bgp dampening <parameters> 95

96 Full Internet in a VRF? Why? Because design dictates it Unique RD, so that RR can advertise 2 paths? PRO Remove Internet routing table from P routers Security: move Internet into VPN, out of global Added flexibility More flexible DDOS mitigation CON Increased memory and bandwidth consumption Platform must support enough MPLS labels Label allocation is per-prefix by default Perhaps per-ce or per-vrf label allocation is wanted here Now also per-ce and per-vrf label allocation for 6PE (in IOS-XR) 96

97 Per-CE Label Per-CE : one MPLS label per next-hop (so per connected CE router) No IP lookup needed after label lookup Caveats Doesn t work together with ebgp, ibgp Multipath and CsC because they make usage of the label diversity BGP PIC is not supported (it makes usage of label diversity) Only single hop ebgp supported, no multihop NH: PE1, P: Z1, label L1 2 CEs = 2 labels Number of prefixes (n) is much larger than number of CE routers (x) per VPN Number of MPLS labels used is very low CE1 P:Z 1-n CEx PE1 RR PE2 CE3 CE2 NH: PE1, P: Zx, label Lx 97

98 Per-VRF Label Per-VRF : one MPLS label per VRF (all CE routers in the VRF) - Con: IP lookup needed after label lookup Number of MPLS labels used per VRF is 1! NH: PE1, P: Z1, label L1 CE1 P:Z 1-n CEx PE1 RR PE2 CE3 CE2 NH: PE1, P: Zx, label L1 98

99 Full Internet in a VRF? Considerations Two Internet gateways for redundancy RRs are present: unique RDs needed Then double # vpn prefixes ADD-PATHS increases paths too RD 1:1 Internet Peerings PE1 PE3 RR RD 1:2 PE2 PE4 99

100 Selective VRF Download (SVD) Applies to IOS-XR Multiple linecards The basic idea is to download to a line card only those prefixes and labels that are actively required to forward traffic through that line card 100

101 OS Enhancements

102 ASR9K: Scaling Enhancement BGP RIB Scale enhancement in Only for RSP440-SE Reload is needed Get more virtual address space for BGP process From 2 GB to 2.5 GB RP/0/RSP1/CPU0:router(admin-config)#hw-module profile scale? default Default scale profile l3 L3 scale profile l3xl L3 XL scale profile Profile Layer 3 (Prefixes) Layer 2 (MAC Table) default Small (512k) Large (512k) l3 Large (1,000k) Small (128k) l3xl Extra large (1,300k) Minimal l3xl (5.1.1 RSP3) Extra large (2,500k) Minimal 102

103 BGP Scaling Enhancements Overview - IOS For Your Reference BGP PE-CE Scale Enhancements Modified internal data structures and optimized internal algorithms for VRF based update generation Result = faster convergence / greater VRF and PE-CE session scaling BGP PE Scale Enhancements Modified internal data structures for VRFs Result = considerable memory savings / greater prefix scalability BGP Generic Scale Enhancements Parceling of BGP processes Created new BGP task IOS process: BGP Task Result = optimized update generation / faster convergence BGP Keepalive Enhancements Priority queues for reading/writing Keepalive/Update messages Results = avoid neighbor flaps / ability to support small keepalive values in a scaled setup 103

104 Multi-Instance BGP A new IOS-XR BGP architecture to support multiple BGP instances Each BGP instance is a separate process running on the same or a different RP/DRP node Different prefix tables Multiple ASNs are possible * Solves the 32-bit OS virtual memory limit Different BGP routers: isolate services/afs on common infrastructure Achieve higher prefix scale (especially on a RR) by having different instances carrying different BGP tables Achieve higher session scale by distributing the overall peering sessions between instances 104 multi-instance BGP PE1 BGP vpnv4 BGP IPv different peerings * restrictions apply multi-instance BGP RR1 BGP vpnv4 BGP IPv4 single-instance BGP RR2 BGP

105 When is the Boat Not Big Enough? Convergence IOS IOS-XR NX-OS show bgp convergence show bgp convergence detail Measure Prefix instability Traffic drops Table Versions Timestamps show bgp all summary show bgp table show bgp process performance-statistics detail IOS IOS-XR NX-OS Memory show bgp all summary show bgp table show bgp internal mem-stats detail - Look for Grand total, Private memory, Shared memory show processes memory sorted show process memory <job-id> location <> show system resource show watchdog memory-state IOS IOS-XR NX-OS CPU show processes cpu history show processes cpu include BGP show processes cpu show processes bgp show processes cpu include bgp show processes cpu history show processes cpu include bgp show process cpu detailed <bgp pid> 105

106 Participate in the My Favorite Speaker Contest Promote Your Favorite Speaker and You Could be a Winner Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress) Send a tweet and include Your favorite speaker s Twitter handle Two hashtags: #CLUS #MyFavoriteSpeaker You can submit an entry for more than one of your favorite speakers Don t forget to View the official rules at 106

107 Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center. Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online 107

108 Continue Your Education Demos in the Cisco Campus Walk-in Self-Paced Labs Table Topics Meet the Engineer 1:1 meetings 108

109