BGP Border Gateway Protocol (an introduction) Karst Koymans Informatics Institute University of Amsterdam (version 310, 2014/03/11 10:50:06) Monday, March 10, 2014 General ideas behind BGP Background Providers, Customers and Peers External and Internal BGP BGP information bases The BGP protocol BGP attributes BGP messages Traffic Engineering Outbound Traffic Engineering Inbound Traffic Engineering IBGP scaling BGP version 4 Autonomous system (AS) Border Gateway Protocol version 4 (BGP4) Specified in RFC 4271 The inter-as routing protocol Monopolises the Internet Based on path vector routing which is inbetween distance vector and link state Uses (often non-coordinated) routing policies which can be problematic for convergence Definition (AS Autonomous System) A connected group of networks and routers Representing some assigned set of IP prefixes Having a single, consistent routing policy Both internally and externally
Autonomous system illustration Providers and Customers Autonomous Systems AS2503 AS192 Internet IP IP Provider IP Customer Internet AS29077 3 Slide courtesy Iljitsch van Beijnum Peers Providers, Customers and Peers Provider 1 Provider 2 Provider 3 IP IP Customer 1 Customer 2 Customer 3 G1 IP R1 G2 No packets C1 C2 P1 IP P2 C3 C4
The AS abstraction Providers, Customers and Peers routing preferences AS Graph!= Internet Topology BGP was designed to throw away information! The order of preference for a route is Customers have highest preference Peers have the next highest preference Providers have the lowest preference Transit relationships are enforced by export filtering Do not advertise or peer routes to other s or peers Do advertise all routes to customers Do advertise customer routes to s and peers The AS graph may look like this Reality may be closer to this Providers, Customers and Peers: Import Import Routes Providers, Customers and Peers: Export Export Routes route peer route customer route ISP route route peer route customer route ISP route From From To From From peer From peer To peer To peer From customer From customer To customer To customer filters block
External and Internal BGP (1) External and Internal BGP (2) EBGP (External BGP) Used for BGP neighbors between different ASs Exchanging prefixes Implementing policies IBGP (Internal BGP) Used for BGP neighbors within one and the same AS Distributing Internet prefixes across the backbone in order to create a consistent view among all entry/exit points Inserting locally originated prefixes for instance for customers that do not speak BGP Routes imported from one IBGP peer are not distributed to another IBGP peer This prevents possible routing loops Loop detection is based on duplicates in AS paths EBGP detects this between different ASs IBGP cannot detect this inside one and the same AS Requires IBGP peers to be configured as a full mesh Routing Information Bases (RIBs) Adj-RIB-In (one per peer) Routes after input filtering Every AS needs an input policy Loc-RIB (only one globally) Routes after best path selection Path selection is a fixed and specified algorithm Adj-RIB-Out (one per peer) Routes after output filtering Every AS needs an output policy BGP route processing Receive BGP Updates BGP Route Processing Apply Import Policies Open ended programming Constrained only by vendor configuration language Apply Policy = filter routes & tweak attributes Based on Attribute Values Best Route Selection Best Routes Best Route Table Install forwarding Entries for best Routes IP Forwarding Table Apply Policy = filter routes & tweak attributes Apply Export Policies Transmit BGP Updates 52
BGP protocol Some important BGP attributes Uses TCP over port 179 Usually with a directly connected neighbor on layer 2 Exchanges Network Layer Reachability Information (NLRI) Prefixes that can or can no longer be reached through the router Accompanied by BGP attributes used by the best route selection algorithm In order of path selection importance LOCAL PREF (Local Preference) AS PATH ORIGIN (Historical) MULTI EXIT DISC (MED; Multi-exit discriminator) And unrelated to path selection NEXT HOP Must be reachable (directly or via IGP) except in the case of multi-hop BGP Next Hop in EBGP and IBGP Interaction between BGP and IGP BGP Next Hop Attribute Join EGP with IGP For Connectivity 1212513390 AS 6431 AT&T Research AS 7018 AT&T 13520700/16 Next Hop = 1212513390 121270121 2654 RIPE NCC RIS project 13520700/16 Next Hop = 121270121 Every time a route announcement crosses an AS boundary, the Next Hop attribute is changed to the IP address of the border router that announced the route 53 10101010 Forwarding Table destination next hop 192020/30 + EGP destination 13520700/16 10101010 next hop 192021 13520700/16 Next Hop = 192021 192021 192020/30 Forwarding Table destination next hop 13520700/16 10101010 192020/30 10101010 13520700/16
BGP attribute types LOCAL PREF (Local Preference) Well-known mandatory ORIGIN, AS PATH, NEXT HOP Well-known discretionary LOCAL PREF, ATOMIC AGGREGATE Optional transitive COMMUNITIES, AGGREGATOR Optional non-transitive MULTI EXIT DISC Advertised within a single AS (via IBGP) Used to implement local policies Can depend on any locally available information This might be learned outside of BGP Default value is 100 Highest value wins AS PATH Sequence of ASs An AS can also be generalized to a set of ASs Used for loop detection The sequence length defines the metric (distance) Shortest path wins Prepend your own AS in EBGP updates Possibly multiple times, enabling traffic engineering Leave unchanged in IBGP updates AS PATH example 13520700/16 AS Path = 1239 7018 6341 239 Sprint 13520700/16 AS Path = 6341 AS 6341 AT&T Research 13520700/16 Prefix Originated ASPATH Attribute 13520700/16 AS Path = 1755 1239 7018 6341 755 Ebone 13520700/16 AS Path = 7018 6341 AS7018 AT&T 13520700/16 AS Path = 7018 6341 129 Global Access 13520700/16 AS Path = 1129 1755 1239 7018 6341 2654 RIPE NCC RIS project 13520700/16 AS Path = 3549 7018 6341 AS 3549 Global Crossing 64
AS PATH length can be deceptive Shorter Doesn t Always Mean Shorter AS PATH for loop prevention Interdomain Loop Prevention In fairness: could you do this right and still scale? Exporting internal state would dramatically increase global instability and amount of routing state AS 3 Mr BGP says that path 4 1 is better than path 3 2 1 Duh! AS 4 BGP at AS YYY will never accept a route with ASPATH containing YYY AS 7018 Don t Accept! 122200/16 ASPATH = 1 333 7018 877 66 Traffic often follows AS PATH Sometimes traffic does not follow AS PATH Traffic Often Follows ASPATH But It Might Not 13520700/16 AS 3 13520700/16 ASPATH = 3 2 1 AS 4 13520700/16 13520700/16 ASPATH = 1 135207440/25 ASPATH = 5 filters all subnets with masks longer than /24 AS 3 13520700/16 ASPATH = 3 2 1 AS 4 IP Packet Dest = 1352074466 AS 5 135207440/25 IP Packet Dest = 1352074466 From AS 4, it may look like this packet will take path 3 2 1, but it actually takes path 3 2 5
ORIGIN MULTI EXIT DISC (Multi-Exit Discriminator or MED) The ORIGIN attribute tells where the route (NLRI) originated Interior to the originating AS: ORIGIN = 0 Via the EGP protocol (historic): ORIGIN = 1 Via some other means: ORIGIN = 2 A lower ORIGIN wins The MED (or metric, formerly INTER AS METRIC) is meant to be advertised between neighboring ASs (via EBGP) Some implementations carry MED on by IBGP Hot potato versus cold potato The MED is non-transitive (is not transferred into a third AS) A lower MED wins The default MED is 0 (lowest possible value) Some implementations choose the highest possible value Best route selection BGP message header Definition (Route selection preference) 1 (Weight; Cisco specific) 2 Highest Local Preference 3 Shortest AS Path 4 (Lowest Origin; hardly used; historic) 5 Lowest MED 6 Prefer EBGP over IBGP 7 Lowest IGP cost to BGP egress 8 Lowest Router ID 0 15 16 23 24 31 Length Marker Type We use the term message and not packet, because BGP packets are in fact part of one single TCP-stream
BGP header fields BGP OPEN message BGP header fields Marker 128 bits of 1 (compatibility) Length Total length (min 19, max 4096) No padding 1, Including header Type 1: OPEN 2: UPDATE 3: NOTIFICATION 4: KEEPALIVE 5: Route-REFRESH 0 7 8 15 16 31 Version My Autonomous System Opt Parm Len Hold Time BGP Identifier Optional Parameters (variable) 1 No superfluous bytes are allowed inside the TCP stream OPEN message fields BGP KEEPALIVE message OPEN message fields Version 4 My Autonomous System Sender s AS Hold Time Liveness detection BGP Identifier Sender s identifying IP address Opt Parm Length Length of parameter field Optional Parameters TLV-encoded options This page intentionally left blank http://wwwthis-page-intentionally-left-blankorg/ One interesting parameter is the Capabilities Optional Parameter, which defines (among others) the Route Refresh Capability
KEEPALIVE message fields BGP NOTIFICATION message KEEPALIVE message fields :) 0 7 8 15 16 31 Error code Error subcode Data (variable) NOTIFICATION message fields BGP Route-REFRESH message NOTIFICATION message fields Error code Error subcode Data 1: Message Header Error 2: OPEN Error 3: UPDATE Error 4: Hold Timer Expired Depends on error code Depends on error code and subcode 0 15 16 23 24 31 AFI Reserved SAFI
Route-REFRESH message fields BGP UPDATE message Route-REFRESH message fields AFI Address Family Identifier Reserved 0 SAFI Subsequent Address Family Identifier 0 15 16 31 Unfeasible Routes Length Total Path Attribute Length Withdrawn Routes (variable length) Path Attributes (variable length) Network Layer Reachability Information (variable length) UPDATE message fields Tweaking your policies UPDATE message fields Unfeasible Routes Length Length of Withdrawn Routes Withdrawn Routes List of prefixes 2 Total Path Attribute Length Length of Path Attributes Path Attributes TLV-encoded attributes Network Layer Reachability Information List of NLRI prefixes Tweak Tweak Tweak For inbound traffic Filter outbound routes Tweak attributes on outbound routes in the hope of influencing your neighbor s best route selection For outbound traffic Filter inbound routes Tweak attributes on inbound routes to influence best route selection inbound traffic outbound traffic outbound routes inbound routes In general, an AS has more control over outbound traffic 2 A prefix is specified by its length and just enough bytes of the network IP address to cover this length
Outbound Traffic Engineering Choice between, peer or customer So Many Choices peer peer This works by manipulating incoming routes Changing local preference Extending inbound AS paths Manipulating the metric (MED), for instance by using inbound communities It is relatively simple Based on your own policy You are in control yourself customer Frank s Internet Barn AS 4 AS 3 Which route should Frank pick to 131300/16? 131300/16 60 Manipulating local preference Prefer customer over peer over LOCAL PREFERENCE Primary and backup links Implementing Backup Links with Local Preference (Outbound Traffic) Local preference used ONLY in ibgp AS 4 local pref = 80 local pref = 90 AS 3 primary link backup link local pref = 100 Set Local Pref = 100 for all routes from AS 65000 Set Local Pref = 50 for all routes from Higher Local preference values are more preferred 131300/16 61 Forces outbound traffic to take primary link, unless link is down We ll talk about inbound traffic soon 70
Multihomed primary and backup links Multihomed Backups (Outbound Traffic) primary link Set Local Pref = 100 for all routes from AS 3 backup link Set Local Pref = 50 for all routes from AS 3 Forces outbound traffic to take primary link, unless link is down 71 Inbound Traffic Engineering This works by manipulating outgoing routes Extending outbound AS PATHs is a traditional hack Manipulating the metric (MED) is the official way Setting outbound communities is a more modern approach Agreements with your neighbors are necessary (common policy) Inbound is more complex than outbound Inbound depends (also) on neighbor s policy You are not in control by yourself Announcing more specific routes Method of last resort, but often a bad idea Traffic engineering a longer AS PATH Shedding Inbound Traffic with ASPATH Padding Yes, this is a Glorious Hack Your might overrule your effort But Padding Does Not Always Work 192020/24 ASPATH = 2 AS 3 192020/24 ASPATH = 2 2 2 2 2 2 2 2 2 2 2 2 2 2 192020/24 ASPATH = 2 primary customer backup 192020/24 192020/24 ASPATH = 2 2 2 Padding will (usually) force inbound traffic from to take primary link 72 primary customer backup 192020/24 AS 3 will send traffic on backup link because it prefers customer routes and local preference is considered before ASPATH length! Padding in this way is often used as a form of load 73 balancing
But you can make an agreement by using a community COMMUNITY Attribute to the Rescue! Hot potato routing Hot Potato Routing: Go for the Closest Egress Point AS 3 AS 3: normal customer local pref is 100, peer local pref is 90 19244780/24 192020/24 ASPATH = 2 primary backup 192020/24 ASPATH = 2 COMMUNITY = 3:70 egress 1 egress 2 15 56 IGP distances customer 192020/24 Customer import policy at AS 3: If 3:90 in COMMUNITY then set local preference to 90 If 3:80 in COMMUNITY then set local preference to 80 If 3:70 in COMMUNITY then set local preference to 70 74 This Router has two BGP routes to 19244780/24 Hot potato: get traffic off of your network as Soon as possible Go for egress 1! 75 Burnt by the hot potato Cold potato routing by honoring MEDs Getting Burned by the Hot Potato Cold Potato Routing with MEDs (Multi-Exit Discriminator Attribute) High bandwidth Provider backbone 2865 17 Heavy Content Web Farm Prefer lower MED values 2865 17 Heavy Content Web Farm SFF NYC 19244780/24 MED = 15 19244780/24 MED = 56 Low bandwidth customer backbone 15 56 15 56 San Diego Many customers want their to carry the bits! tiny http request huge http reply 76 19244780/24 This means that MEDs must be considered BEFORE IGP distance! Note1 : some s will not listen to MEDs Note2 : MEDs need not be tied to IGP distance 77
Communities Use of communities An optional transitive attribute A community can be used to communicate preferred treatment of a route Communities can be used with both inbound as well as outbound Some communities have a well-known semantics NO EXPORT: don t export beyond current AS (or confederation) NO ADVERTISE: don t export at all Inbound from your upstream Learn where your upstream imported this route You can base policy decisions on that Outbound to your upstream Request specific upstream treatment Setting of local preference Announcements or not to specific ASs AS PATH prepending for certain peerings Your upstream promises to implement the requested policy Structure and semantics of communities Route Reflectors How Can Routes be Colored? BGP Communities! A community value is 32 bits By convention, first 16 bits is ASN indicating who is giving it an interpretation community number Used for signally within and between ASes Very powerful BECAUSE it has no (predefined) meaning Community Attribute = a list of community values (So one route can belong to multiple communities) Specified in RFC 4456 A route reflector is a kind of super IBGP peer A route reflector has clients with which it peers via IBGP and for which it reflects (transitively) routes A route reflector is part of a full mesh of other route reflectors and non-clients RFC 1997 (August 1996) Two reserved communities no_export = 0xFFFFFF01: don t export out of AS no_advertise 0xFFFFFF02: don t pass to BGP neighbors 58
Route reflectors illustration Route reflectors illustration Full Mesh Route Reflection 39 40 Slide courtesy Iljitsch van Beijnum Slide courtesy Iljitsch van Beijnum Confederations Confederations illustration Confederations Specified in RFC 5065 Use multiple private ASs inside your main AS Talk to the outside world with your main AS This hides the private ASs Talk to the inside world as if using EBGP and IBGP Using the different private ASs This needs special AS PATH segment types 41 Slide courtesy Iljitsch van Beijnum