BGP Border Gateway Protocol A short introduction Karst Koymans Informatics Institute University of Amsterdam (version 18.3, 2018/12/03 13:53:22) Tuesday, December 4, 2018 Karst Koymans (UvA) BGP Tuesday, December 4, 2018 1 / 76
1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 2 / 76
General ideas behind BGP Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 3 / 76
General ideas behind BGP Background Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 4 / 76
General ideas behind BGP Background BGP version 4 Border Gateway Protocol version 4 (BGP4) Specified in RFC 4271 The inter-as routing protocol Monopolises the Internet Based on path vector routing which is in-between distance vector and link state routing Uses (often non-coordinated) routing policies which can be problematic for convergence Karst Koymans (UvA) BGP Tuesday, December 4, 2018 5 / 76
General ideas behind BGP Background Autonomous system (AS) Definition (AS Autonomous System) A connected group of networks and routers representing some assigned set of IP prefixes having a single, consistent routing policy both internally and externally Karst Koymans (UvA) BGP Tuesday, December 4, 2018 6 / 76
Autonomous system illustration
General ideas behind BGP Providers, Customers and Peers Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 8 / 76
Providers and Customers Internet IP IP Provider $$ IP Customer Internet The IP means IP data packet flow The $$ means money transfer
Peers Provider 1 $$ Provider 2 $$ Provider 3 IP IP $$ $$ $$ Customer 1 Customer 2 No packets Customer 3 No packets here means that packet flow between Customer 1 and Customer 3 usually will not flow through Provider 2.
Providers, Customers and Peers G1 IP $$ $$ G2 $$ $$ R1 $$ C1 C2 $$ $$ P1 P2 IP $$ $$ C3 $$ C4 What route will traffic between C1 and C2 take?
The AS abstraction
Different AS types Source: RIPE NCC BGP course
Providers, Customers and Peers routing preferences The order of preference for incoming route advertisements is Customers have highest preference Peers have the next highest preference Providers have the lowest preference Transit relationships are enforced by export filtering Outgoing route advertisements attract incoming traffic Do not advertise provider or peer routes to other providers or peers Do advertise all routes to customers Do advertise customer routes to providers and peers
Providers, Customers and Peers: Export filtering
General ideas behind BGP External and Internal BGP Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 16 / 76
General ideas behind BGP External and Internal BGP External and Internal BGP (1) EBGP (External BGP) Used for BGP neighbors between different ASs Exchanging prefixes Implementing policies IBGP (Internal BGP) Used for BGP neighbors within one and the same AS Distributing Internet prefixes across the backbone in order to create a consistent view among all entry/exit points and in the whole routing domain Inserting locally originated prefixes, for instance for customers that do not speak BGP Karst Koymans (UvA) BGP Tuesday, December 4, 2018 17 / 76
General ideas behind BGP External and Internal BGP External and Internal BGP (2) Routes imported from one IBGP peer are not distributed to another IBGP peer This prevents possible routing loops Loop detection is based on duplicates in AS paths EBGP detects this between different ASs IBGP cannot detect this inside one and the same AS Requires IBGP peers to be configured as a full mesh 1 1 I have seen authors write this as full mess Karst Koymans (UvA) BGP Tuesday, December 4, 2018 18 / 76
General ideas behind BGP BGP information bases Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 19 / 76
General ideas behind BGP BGP information bases Routing Information Bases (RIBs) Adj 2 -RIB-In (one per peer) Unprocessed 3 routes received from peer Loc-RIB (only one globally) Routes after input filtering and best path selection Every AS needs an input policy Path selection is a fixed and specified algorithm Adj-RIB-Out (one per peer) Routes after output filtering Every AS needs an output policy 2 Adj stands for Adjacent, although the RFC doesn t say so 3 According to the RFC, but implementations may tweak? Karst Koymans (UvA) BGP Tuesday, December 4, 2018 20 / 76
BGP route processing Receive BGP update Install route in Adj-RIB-In Apply import policy: filter routes and tweak attributes Apply best route selection algorithm Install selected route in Loc-RIB Influence IP forwarding table Apply export policy: filter routes and tweak attributes Possibly apply route aggregation Install exported routes in Adj-RIB-Out Transmit BGP update Send Adj-RIBs-Out to peers
The BGP protocol Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 22 / 76
The BGP protocol BGP protocol Uses TCP over port 179 EBGP usually connects to a directly connected neighbor on layer 2, except when using multi-hop EBGP (not common) IBGP uses routes from an IGP to reach its peers Exchanges Network Layer Reachability Information (NLRI) Prefixes that can or can no longer be reached through the router Accompanied by BGP attributes used by the best route selection algorithm Karst Koymans (UvA) BGP Tuesday, December 4, 2018 23 / 76
The BGP protocol BGP attributes Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 24 / 76
The BGP protocol BGP attributes Some important BGP attributes In order of path selection importance LOCAL_PREF (Local Preference) AS_PATH ORIGIN (Historical) MULTI_EXIT_DISC (MED; Multi-exit discriminator) Unrelated to path selection NEXT_HOP Must be reachable directly or via IGP, except in the case of multi-hop BGP, otherwise route is not used in path selection For instance used by route servers in IXPs Karst Koymans (UvA) BGP Tuesday, December 4, 2018 25 / 76
Next Hop in EBGP and IBGP Unless router BC uses next-hop-self, in which case the NEXT_HOP in the second BGP UPDATE would contain C instead of A.
The BGP protocol BGP attributes BGP attribute types Well-known, mandatory AS_PATH, NEXT_HOP, ORIGIN Well-known, discretionary LOCAL_PREF, ATOMIC_AGGREGATE Optional, transitive COMMUNITIES, AGGREGATOR Optional, non-transitive MULTI_EXIT_DISC Karst Koymans (UvA) BGP Tuesday, December 4, 2018 27 / 76
The BGP protocol BGP attributes LOCAL_PREF (Local Preference) Advertised within a single AS (via IBGP) Used to implement local policies Can depend on any locally available information This might be learned outside of BGP Default value is 100 Highest value wins Karst Koymans (UvA) BGP Tuesday, December 4, 2018 28 / 76
The BGP protocol BGP attributes AS_PATH Sequence of ASs An AS can also be generalized to a set of ASs Used for loop detection The sequence length defines the metric (distance) Shortest path wins Prepend your own AS in EBGP updates Possibly multiple times, enabling traffic engineering Leave unchanged in IBGP updates Karst Koymans (UvA) BGP Tuesday, December 4, 2018 29 / 76
AS_PATH example (origin AS 1)
Another AS_PATH example (origin D) Source: RIPE NCC BGP course
AS_PATH length can be deceptive
Traffic often follows AS_PATH (in reverse)
Sometimes traffic does not follow AS_PATH
The BGP protocol BGP attributes ORIGIN (Historic attribute) The ORIGIN attribute tells where the route (NLRI) originated Interior to the originating AS: ORIGIN = 0 Via the EGP protocol (historic): ORIGIN = 1 Via some other means (incomplete): ORIGIN = 2 A lower ORIGIN wins In practice the ORIGIN is always 0 Karst Koymans (UvA) BGP Tuesday, December 4, 2018 35 / 76
The BGP protocol BGP attributes MULTI_EXIT_DISC (Multi-Exit Discriminator or MED) The MED (or metric, formerly INTER_AS_METRIC) is meant to be advertised between neighboring ASs (via EBGP) Some implementations carry MED on by IBGP Hot potato versus cold potato The MED is non-transitive (is not transferred into a third AS) A lower MED wins The default MED is 0 (lowest possible value) Some implementations choose the highest possible value Karst Koymans (UvA) BGP Tuesday, December 4, 2018 36 / 76
Best route selection criteria Definition (Route preference order) 1 (Drop if own AS in AS_PATH) 2 (Weight; Cisco specific) 3 Highest Local Preference 4 Shortest AS Path 5 (Lowest Origin; hardly used; historic) 6 Lowest MED (to the same neighbor) 7 Prefer ebgp over ibgp
Best route selection tiebreakers Definition (Route preference tiebreakers) 1 Lowest IGP cost to BGP egress (Next Hop) 2 Already existing received path 3 Lowest Router ID 4 Lowest neighbor IP address
The BGP protocol BGP messages Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 39 / 76
The BGP protocol BGP messages BGP message header 0 15 16 23 24 31 Marker Length Type We use the term message and not packet, because BGP packets are in fact part of one single TCP-stream. Karst Koymans (UvA) BGP Tuesday, December 4, 2018 40 / 76
The BGP protocol BGP messages a No superfluous bytes are allowed inside the TCP stream Karst Koymans (UvA) BGP Tuesday, December 4, 2018 41 / 76 BGP header fields BGP header fields Marker 128 bits of 1 (compatibility) Length Total length (min 19, max 4096) No padding a, Including header Type 1: OPEN 2: UPDATE 3: NOTIFICATION 4: KEEPALIVE 5: Route-REFRESH
The BGP protocol BGP messages BGP OPEN message 0 7 8 15 16 31 Version My Autonomous System Opt Parm Len Hold Time These bits are absent BGP Identifier as well as these Optional Parameters and these and these (variable) Karst Koymans (UvA) BGP Tuesday, December 4, 2018 42 / 76
The BGP protocol BGP messages OPEN message fields OPEN message fields Version 4 My Autonomous System Hold Time BGP Identifier Opt Parm Length Optional Parameters Sender s AS Liveness detection Sender s identifying IP address Length of parameter field TLV-encoded options One interesting parameter is the Capabilities Optional Parameter, which defines (among others) the Route Refresh, 4-Octet AS number, Multiprotocol and BGPsec capabilities. Karst Koymans (UvA) BGP Tuesday, December 4, 2018 43 / 76
The BGP protocol BGP messages BGP KEEPALIVE message Karst Koymans (UvA) BGP Tuesday, December 4, 2018 44 / 76
The BGP protocol BGP messages BGP KEEPALIVE message This page intentionally left blank. Karst Koymans (UvA) BGP Tuesday, December 4, 2018 44 / 76
The BGP protocol BGP messages BGP KEEPALIVE message This page intentionally left blank. http://www.this-page-intentionally-left-blank.org/ 4 4 One of those Internet memes like http://endoftheinternet.com/ Karst Koymans (UvA) BGP Tuesday, December 4, 2018 44 / 76
The BGP protocol BGP messages KEEPALIVE message fields KEEPALIVE message fields :) Usually the keepalive timer is something like 60 seconds and the hold time three times as long. Karst Koymans (UvA) BGP Tuesday, December 4, 2018 45 / 76
The BGP protocol BGP messages BGP NOTIFICATION message 0 7 8 15 16 31 Error code Error subcode Data (variable) Karst Koymans (UvA) BGP Tuesday, December 4, 2018 46 / 76
The BGP protocol BGP messages NOTIFICATION message fields NOTIFICATION message fields Error code Error subcode Data 1: Message Header Error 2: OPEN Error 3: UPDATE Error 4: Hold Timer Expired Depends on error code Depends on error code and subcode Karst Koymans (UvA) BGP Tuesday, December 4, 2018 47 / 76
The BGP protocol BGP messages BGP Route-REFRESH message 0 15 16 23 24 31 AFI Reserved SAFI The AFI (Address Family Identifier) is used for Multiprotocol extensions to BGP (MBGP). This AFI is not compatible with the one used in RIP. A SAFI (Subsequent Address Family Identifier) is also defined for a further subdivision, for instance to distinguish unicast from multicast. Route refresh is a better solution than the soft-reconfiguration solution, which kept all received updates unmodified in memory (Adj-RIB-In). Karst Koymans (UvA) BGP Tuesday, December 4, 2018 48 / 76
The BGP protocol BGP messages Route-REFRESH message fields Route-REFRESH message fields AFI Address Family Identifier a Reserved 0 SAFI Subsequent Address Family Identifier b a https://www.iana.org/assignments/address-family-numbers/address-family-numbers.xhtml b https://www.iana.org/assignments/safi-namespace/safi-namespace.xhtml Karst Koymans (UvA) BGP Tuesday, December 4, 2018 49 / 76
The BGP protocol BGP messages BGP UPDATE message 0 15 16 31 Unfeasible Routes Length Withdrawn Routes (variable length) Absent bits Total Path Attribute Length Path Attributes (variable length) Absent bits Network Layer Reachability Information (variable length) Karst Koymans (UvA) BGP Tuesday, December 4, 2018 50 / 76
The BGP protocol BGP messages UPDATE message fields UPDATE message fields Unfeasible Routes Length Withdrawn Routes Total Path Attribute Length Path Attributes Network Layer Reachability Information Length of Withdrawn Routes List of prefixes a Length of Path Attributes TLV-encoded attributes List of NLRI prefixes a A prefix is specified by its length in bits and just enough bytes of the network IP address to cover this length Karst Koymans (UvA) BGP Tuesday, December 4, 2018 51 / 76
Traffic Engineering Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 52 / 76
Tweaking your policies Outbound traffic Influenced by inbound routes and filters Tweak attributes to influence best route selection You are in control yourself Inbound traffic Influenced by outbound routes and filters Tweak attributes trying to influence your peers best route selection You are dependent on your peers policies
Traffic Engineering Outbound Traffic Engineering Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 54 / 76
Traffic Engineering Outbound Traffic Engineering Outbound Traffic Engineering Outbound TE works by manipulating incoming routes Changing local preference Extending inbound AS paths Manipulating the metric (MED), for instance by using inbound communities It is relatively simple Based on your own policy You are in control yourself Karst Koymans (UvA) BGP Tuesday, December 4, 2018 55 / 76
Choice between provider, peer or customer
Manipulating local preference Prefer customer over peer over provider
Multihomed setup
Singlehomed primary and backup links
Traffic Engineering Inbound Traffic Engineering Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 60 / 76
Traffic Engineering Inbound Traffic Engineering Inbound Traffic Engineering Inbound TE works by manipulating outgoing routes Extending outbound AS_PATHs is a traditional hack Manipulating the metric (MED) is the official way Setting outbound communities is a more modern approach Agreements with your neighbors are necessary (common policy) Inbound is more complex than outbound Inbound depends (also) on neighbor s policy You are not in control all by yourself Announcing more specific routes Method of last resort, but often a bad idea Karst Koymans (UvA) BGP Tuesday, December 4, 2018 61 / 76
Advertising a longer AS_PATH
Your providers may overrule your effort
But you can make an agreement by using a community
Hot potato routing
Burnt by the hot potato
Cold potato routing by honoring MEDs
Traffic Engineering Inbound Traffic Engineering Communities An optional transitive attribute A community can be used to communicate preferred treatment of a route Communities can be used with both inbound as well as outbound Communities can also be used with IBGP, for instance tagging customer, peer and provider routes Some communities have a well-known semantics NO_EXPORT: don t export beyond current AS (or confederation) NO_ADVERTISE: don t export at all Karst Koymans (UvA) BGP Tuesday, December 4, 2018 68 / 76
Traffic Engineering Inbound Traffic Engineering Use of communities Inbound from your upstream Learn where your upstream imported this route You can base policy decisions on that Outbound to your upstream Request specific upstream treatment Setting of local preference Announcements or not to specific ASs AS_PATH prepending for certain peerings Your upstream promises to implement the requested policy Karst Koymans (UvA) BGP Tuesday, December 4, 2018 69 / 76
IBGP scaling Outline 1 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases 2 The BGP protocol BGP attributes BGP messages 3 Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering 4 IBGP scaling Karst Koymans (UvA) BGP Tuesday, December 4, 2018 70 / 76
IBGP scaling Route reflectors Specified in RFC 4456 A route reflector is a kind of super IBGP peer A route reflector has clients with which it peers via IBGP and for which it reflects (transitively) routes A route reflector is part of a full mesh of other route reflectors and non-clients Karst Koymans (UvA) BGP Tuesday, December 4, 2018 71 / 76
Full mesh IBGP
Route reflector mesh
IBGP scaling Confederations Specified in RFC 5065 Use multiple private ASs inside your main AS Talk to the outside world with your main AS This hides the private ASs Talk to the inside world as if using EBGP and IBGP Using the different private ASs This needs special AS_PATH segment types Karst Koymans (UvA) BGP Tuesday, December 4, 2018 74 / 76
Confederation with SubAS s
IBGP scaling BGP in data centers Modern (Clos) fabrics in data centers can Eliminate layer 2 Use IP only on point-to-point links Use EBGP internally with private ASs as if it were an IGP Karst Koymans (UvA) BGP Tuesday, December 4, 2018 76 / 76