Interdomain Routing BGP. Benjamin Baron

Transcription

1 Interdomain Routing BGP Benjamin Baron

2 Goals of Today s Lecture Challenges of interdomain routing Scale, privacy, and policy Limitations of link-state and distance-vector routing Path-vector routing Faster loop detection than distance-vector routing More flexibility than shortest-path routing Border Gateway Protocol (BGP) Incremental, prefix-based, path-vector protocol Programmable import and export policies Multi-step decision process for selecting best route Multiple routers within an AS BGP convergence delay

3 Background IP addressing Address allocation blocks Packet forwarding Routing protocols Autonomous Systems Interdomain routing Intradomain routing

4 IP Addressing 32-bit number in dotted-quad notation ( ) Divided into network & host portions (left and right) /24 is a 24-bit prefix with 2 8 addresses Network (24 bits) Host (8 bits)

5 Some History: Why Dotted-Quad Notation? In the older days Class A: 0* Very large /8 blocks (e.g., MIT has /8) Class B: 10* Large /16 blocks (e.g,. Princeton has /16) Class C: 110* Small /24 blocks (e.g., AT&T Labs has /24) Class D: 1110* Multicast groups Class E: 11110* Reserved for future use (sounds a bit scary ) And then, address space became scarce

6 Classless Inter-Domain Routing (CIDR) Use two 32-bit numbers to represent a network. Network number = IP address + Mask IP Address : IP Mask: Address Mask Network Prefix for hosts Usually written as /15

7 CIDR = Hierarchy in Address Allocation Prefixes are key to Internet scalability Address allocation by ARIN/RIPE/APNIC and by ISPs Routing protocols and packet forwarding based on prefixes Today, routing tables contain ~500,000 prefixes / / / / / / / / / / / / / / / / /19...

8 IP Address!= Host Machine Dynamic IP address assignment (DHCP) Single client may have multiple addresses over time Address may correspond to multiple clients over time Shared machines Multiple users on a shared compute server Transfers traveling through proxies and firewalls Multiple Web sites hosted on a single machine Replicated sites Multiple machines hosting a single (popular) Web site Addresses do not correspond to geographic location Similar prefix does not necessarily imply nearby hosts Single prefix may span hosts in large geographic region Source IP address may be spoofed (e.g., DoS attack)

9 Forwarding vs. Routing Forwarding: data plane Directing a data packet to an outgoing link Individual router using a forwarding table Routing: control plane Computing the paths the packets will follow Routers talking amongst themselves Individual router creating a forwarding table

10 Routing vs. Forwarding Default to upstream router A R R1 R4 R R2 B R5 R3 R Net A B C D E default C Nxt Hop R1 Direct R3 R1 R3 R1 Forwarding always works Routing can be badly broken Net A B C D E default Nxt Hop R2 R2 Direct R5 R5 R2 D E Forwarding: determine next hop Routing: establish end-to-end paths Net A B C D E default Nxt Hop R4 R3 R3 R4 Direct R4

11 Longest Prefix Match Forwarding Forwarding tables in IP routers Maps each IP prefix to next-hop link(s) Destination-based forwarding Packet has a destination address Router identifies longest-matching prefix Nice algorithmic problem: very fast lookups forwarding table destination / / / / /24 outgoing link Serial0/0.1

12 Where do Forwarding Tables Come From? Routers have forwarding tables Map prefix to outgoing link(s) Entries can be statically configured E.g., map /24 to Serial0/0.1 But, this doesn t adapt To failures To new equipment To the need to balance load That is where routing (protocols) come in

13 How Are Forwarding Tables Populated to implement Routing? Statically Administrator manually configures forwarding table entries + More control + Not restricted to destination-based forwarding - Doesn t scale - Slow to adapt to network failures Dynamically Routers exchange network reachability information using ROUTING PROTOCOLS. Routers use this to compute best routes + Can rapidly adapt to changes in network topology + Can be made to scale well - Complex distributed algorithms - Consume CPU, Bandwidth, Memory - Debugging can be difficult - Current protocols are destination-based In practice : a mix of these. Static routing mostly at the edge

14 What is Routing? A famous quotation from RFC 791 A name indicates what we seek. An address indicates where it is. A route indicates how we get there. -- Jon Postel

15 Data, Control, and Management Planes Data Plane Control Plane Management Plane Timescale Packet (nsec) Event (10 msec to sec) Human (min to hours) Tasks Forwarding, buffering, filtering, scheduling Routing, signaling Analysis, configuration Location Line-card hardware Router software Humans or scripts

16 Architecture of Dynamic Routing OSPF AS 1 BGP IGP = Interior Gateway Protocol Metric based: OSPF, IS-IS, RIP, EIGRP (cisco) EGP = Exterior Gateway Protocol Policy based: BGP AS 2 EIGRP The Routing Domain of BGP is the entire Internet

17 Distributed Routing Technology Link State Topology information is flooded within the routing domain Best end-to-end paths are computed locally at each router Best end-to-end paths determine next-hops Based on minimizing some notion of distance Works only if policy is shared and uniform Examples: OSPF, IS-IS Vectoring Each router knows little about network topology Only best next-hops are chosen by each router for each destination network Best end-to-end paths result from composition of all nexthop choices Does not require any notion of distance Does not require uniform policies at all routers Examples: RIP, BGP

18 The Gang of Four Link State Vectoring IGP OSPF IS-IS RIP EGP BGP

19 Routers Talking to Routers Routing info Routing info Routing computation is distributed among routers within a routing domain Computation of best next hop based on routing information is the most CPU/memory intensive task on a router Routing messages are usually not routed, but exchanged via layer 2 between physically adjacent routers (internal BGP and multi-hop external BGP are exceptions)

20 Shortest-Path Routing Path-selection model Destination-based Minimum hop count or sum of link weights Dynamic vs. static link weights

21 Distance Vector Routing: Bellman-Ford Define distances at each node x d x (y) = cost of least-cost path from x to y Update distances based on neighbors d x (y) = min {c(x,v) + d v (y)} over all neighbors v u 3 2 v w x 5 s 3 4 y t 1 z d u (z) = min{c(u,v) + d v (z), c(u,w) + d w (z)} e.g., RIP and EIGRP

22 Link-State Routing: Dijsktra s Algorithm Each router keeps track of its incident links Link cost, and whether the link is up or down Each router broadcasts the link state To give every router a complete view of the graph Each router runs Dijkstra s algorithm To compute shortest paths and forwarding table e.g., OSPF and IS-IS

23 Intradomain Routing Today Link-state routing with static link weights Static weights: avoid stability problems Link state: faster reaction to topology changes Most common protocols in backbones OSPF: Open Shortest Path First IS-IS: Intermediate System Intermediate System Some use of distance vector in enterprises RIP: Routing Information Protocol EIGRP: Enhanced Interior Gateway Routing Protocol Growing use of Multi-Protocol Label Switching

24 Link-State Routing is Problematic Topology information is flooded High bandwidth and storage overhead Forces nodes to divulge sensitive information Entire path computed locally per node High processing overhead in a large network Minimizes some notion of total distance Works only if policy is shared and uniform Typically used only inside an AS E.g., OSPF and IS-IS

25 Challenges for Interdomain Routing Scale Prefixes: 150, ,000, and growing ASes: 20,000 visible ones, and growing AS paths and routers: at least in the millions Privacy ASes don t want to divulge internal topologies or their business relationships with neighbors Policy No Internet-wide notion of a link cost metric Need control over where you send traffic and who can send traffic through you

26 Shortest-Path Routing is Restrictive All traffic must travel on shortest paths All nodes need common notion of link costs Incompatible with commercial relationships National ISP1 National ISP2 YES NO Regional ISP3 Regional ISP2 Regional ISP1 Cust3 Cust2 Cust1

27 Distance Vector is on the Right Track Advantages Hides details of the network topology Nodes determine only next hop toward the dest Disadvantages Minimizes some notion of total distance, which is difficult in an interdomain setting Slow convergence due to the counting-to-infinity problem ( bad news travels slowly ) Idea: extend the notion of a distance vector

28 Path-Vector Routing Extension of distance-vector routing Support flexible routing policies Avoid count-to-infinity problem Key idea: advertise the entire path Distance vector: send distance metric per dest d Path vector: send the entire path for each dest d 3 d: path (2,1) d: path (1) 2 1 data traffic data traffic d

29 Faster Loop Detection Node can easily detect a loop Look for its own node identifier in the path E.g., node 1 sees itself in the path 3, 2, 1 Node can simply discard paths with loops E.g., node 1 simply discards the advertisement 3 d: path (2,1) d: path (1) 2 1 d: path (3,2,1)

30 Flexible Policies Each node can apply local policies Path selection: Which path to use? Path export: Which paths to advertise? Examples Node 2 may prefer the path 2, 3, 1 over 2, 1 Node 1 may not let node 3 hear the path 1,

31 Routing Protocols Dissemination Link State Distance Vector Path Vector Algorithm Converge Protocols

32 Routing Protocols Link State Distance Vector Path Vector Dissemination Flood link state advertisements to all routers Update distances from neighbors distances Update paths based on neighbors paths Algorithm Dijsktra s shortest path Bellman-Ford shortest path Local policy to rank paths Converge Fast due to flooding Slow, due to count-toinfinity Slow, due to path exploration Protocols OSPF, IS-IS RIP, EIGRP BGP

33 Two-Tiered Internet Routing Architecture Goal: distributed management of resources Internetworking of multiple networks Networks under separate administrative control Solution: two-tiered routing architecture Intradomain: inside a region of control Okay for routers to share topology information Routers configured to achieve a common goal Interdomain: between regions of control Not okay to share complete information Networks may have different/conflicting goals Led to the use of different protocols

34 Connections Between Networks dial-in access interdomain protocols private peering ISP 2 intradomain protocols ISP 1 gateway router IXP destination access router ISP 3 commercial customer destination

35 Internet Structure Federated network of Autonomous Systems (AS) Routers and links controlled by a single entity Routing between ASes, and within an AS Web client Web server

36 Internet Routing Architecture Divided into Autonomous Systems (AS) Distinct regions of administrative control Routers/links managed by a single institution Service provider, company, university, Hierarchy of Autonomous Systems Large, tier-1 provider with a nationwide backbone Medium-sized regional provider with smaller backbone Small network run by a single company or university Interaction between Autonomous Systems Internal topology is not shared between ASes but, neighboring ASes interact to coordinate routing

37 Autonomous Systems (ASes) An autonomous system is an autonomous routing domain that has been assigned an Autonomous System Number (ASN) the administration of an AS appears to other ASes to have a single coherent interior routing plan and presents a consistent picture of what networks are reachable through it. RFC 1930: Guidelines for creation, selection, and registration of an Autonomous System

38 AS Numbers (ASNs) Level 3: 1 MIT: 3 Harvard: 11 Yale: 29 ASNs are 16 bit values through are private Princeton: 88 AT&T: 7018, 6341, 5074, Sprint: 1239, 1240, 6211, 6242, UPMC: 1307 Currently around 20,000 in use ASNs represent units of routing policy

39 Characterizations of AS Topology Tier-1: small number of tier-1 ASes A near-clique of ~15 ASes with no providers AT&T, Sprint, UUNET, Transit core: peer with tier-1s and each other Around large ASes UUNET Europe, KDDI, and Singapore Telecom Regional ISPs: non-stubs near the edge Around 2000 medium-sized ASes Minnesota Regional Network, US West Stub ASes: no peer or customer neighbors Princeton, Rutgers, MIT, AT&T Research,

40 AS Institution Not equivalent to an AS Many institutions span multiple autonomous systems Some institutions do not have their own AS number Ownership of an AS may be hard to pinpoint (whois) Not equivalent to a block of IP addresses (prefix) Many institutions have multiple (non-contiguous) prefixes Some institutions are a small part of a larger address block Ownership of a prefix may be hard to pinpoint (whois) Not equivalent to a domain name (att.com) Some sites may be hosted by other institutions Some institutions have multiple domain names (att.net)

41 Interdomain Routing (Between ASes) Path: 6, 5, 4, 3, 2, Web server Client

42 Routing is Not Symmetric client Web request and TCP ACKs Web response server

43 Interdomain Routing: Border Gateway Protocol ASes exchange info about who they can reach IP prefix: block of destination IP addresses AS path: sequence of ASes along the path Policies configured by the AS s operator Path selection: which of the paths to use? Path export: which neighbors to tell? /24: path (2,1) /24: path (1) data traffic data traffic

44 Zooming in to AS 3 Border router Learns BGP route from neighbor AS Creates forwarding-table entry for prefix But, how do the other routers get there? Border router: /24

45 How do Other Routers Learn the BGP Route? Internal BGP ibgp sessions between the routers Allows other routers to get the big picture Simplest case: full mesh of ibgp sessions /24 through red router ibgp session /24

46 How To Get to the Egress Router? Interior Gateway Protocol (OSPF/IS-IS) Routers flood information to learn topology Routers determine next hop to other routers Compute shortest paths based on the link weights Link weights configured by the operator Use Serial0/0.1 to get to the red router

47 Constructing the Forwarding Table Three protocols External BGP: learn the external route Internal BGP: propagate inside the AS IGP: learn outgoing link on path to other router Router joins the data Prefix /24 reached through red router Red router reached via link Serial0/0.1 Forwarding entry: /24 à Serial0/0.1 Router forwards packets Lookup destination in table Forward packet out link Serial0/0.1

48 Border Gateway Protocol Interdomain routing protocol for the Internet Prefix-based path-vector protocol Policy-based routing based on AS Paths Evolved during the past 15 years 1989 : BGP-1 [RFC 1105] Replacement for EGP (1984, RFC 904) 1990 : BGP-2 [RFC 1163] 1991 : BGP-3 [RFC 1267] 1995 : BGP-4 [RFC 1771] Support for Classless Interdomain Routing (CIDR)

49 Components of BGP BGP protocol Definition of how two BGP neighbors communicate Message formats, state machine, route attributes, etc. Standardized by the IETF Policy specification Flexible language for filtering and manipulating routes Indirectly affects the selection of the best route Varies across vendors, though constructs are similar BGP decision process Complex sequence of rules for selecting the best route De facto standard applied by router vendors Being codified in a new RFC for BGP coming soon

50 BGP Operations Establish session on TCP port 179 AS1 Exchange all active routes BGP session AS2 Exchange incremental updates While connection is ALIVE exchange route UPDATE messages

51 Two Types of BGP Neighbor Relationships AS1 External Neighbor (ebgp) in a different Autonomous Systems Internal Neighbor (ibgp) in the same Autonomous System ebgp ibgp is routed (using IGP!) ibgp AS2

52 ibgp Mesh Does Not Scale ibgp updates ebgp update N border routers means N(N-1)/2 peering sessions Each router must have N-1 ibgp sessions configured The addition a single ibgp speaker requires configuration changes to all other ibgp speakers Size of ibgp routing table can be order N larger than number of best routes (remember alternate routes!) Each router has to listen to update noise from each neighbor Currently four solutions: (0) Buy bigger routers! (1) Break AS into smaller ASes (2) BGP Route reflectors (3) BGP confederations

53 Route Reflectors RR RR RR Route reflectors can pass on ibgp updates to clients Each RR passes along ONLY best routes ORIGINATOR_ID and CLUSTER_LIST attributes are needed to avoid loops

54 BGP Confederations AS AS AS AS AS AS 1 From the outside, this looks like AS 1 Confederation ebgp (between member ASes) preserves LOCAL_PREF, MED, and BGP NEXTHOP.

55 Four Basic Messages Open: Establishes BGP session (uses TCP port #179) Notification: Report unusual conditions Update: Inform neighbor of new routes that become active Inform neighbor of old routes that become inactive Keepalive: Inform neighbor that connection is still viable

56 Incremental Protocol A node learns multiple paths to destination Stores all of the routes in a routing table Applies policy to select a single active route and may advertise the route to its neighbors Incremental updates Announcement Upon selecting a new active route, add node id to path and (optionally) advertise to each neighbor Withdrawal If the active route is no longer available send a withdrawal message to the neighbors

57 BGP Protocol: Update Messages Update messages Advertisement New route for the prefix (e.g., /24) Attributes such as the AS path (e.g., 2 1 ) Withdrawal Announcing that the route is no longer available Numerous BGP attributes AS path Next-hop IP address Local preference Multiple-Exit Discriminator

58 Advertising a prefix When a router advertises a prefix to one of its BGP neighbors: information is valid until first router explicitly advertises that the information is no longer valid BGP does not require routing information to be refreshed if node A advertises a path for a prefix to node B, then node B can be sure node A is using that path itself to reach the destination

59 BGP Route Destination prefix (e.g, /16) Route attributes, including AS path (e.g., ) Next-hop IP address (e.g., ) AS 7018 AT&T AS 88 Princeton AS RIPE NCC RIS project /16 AS path = 88 Next Hop = /16 AS path = Next Hop =

60 Joining BGP and IGP Information /16 Next Hop = / Forwarding Table destination next hop / BGP destination next hop / AS 7018 AS Forwarding Table destination next hop / /

61 BGP Path Selection Simplest case Shortest AS path Arbitrary tie break Example Four-hop AS path preferred over a three-hop AS path AS prefers path through Global Crossing But, BGP is not limited to shortestpath routing Policy-based routing AS 1129 Global Access /16 AS Path = AS RIPE NCC RIS project /16 AS Path = AS 3549 Global Crossing

62 BGP Policy: Applying Policy to Routes Import policy Filter unwanted routes from neighbor E.g. prefix that your customer doesn t own Manipulate attributes to influence path selection E.g., assign local preference to favored routes Export policy Filter routes you don t want to tell your neighbor E.g., don t tell a peer a route learned from other peer Manipulate attributes to control what they see E.g., make a path look artificially longer than it is

63 BGP Policy: Influencing Decisions Open ended programming. Constrained only by vendor configuration language Receive BGP Updates Apply Policy = filter routes & tweak attributes Based on Attribute Values Best Routes Apply Policy = filter routes & tweak attributes Transmit BGP Updates Apply Import Policies Best Route Selection Best Route Table Apply Export Policies Install forwarding Entries for best Routes. IP Forwarding Table

64 BGP Decision Process: Path Selection on a Router Routing Information Base Store all BGP routes for each destination prefix Withdrawal message: remove the route entry Advertisement message: update the route entry Selecting the best route Consider all BGP routes for the prefix Apply rules for comparing the routes Select the one best route Use this route in the forwarding table Send this route to neighbors

65 Recall: Four Types of BGP Messages Open: Establish a peering session Keep Alive: Handshake at regular intervals Notification: Shuts down a peering session Update: Announcing new routes or withdrawing previously announced routes Announcement = prefix + attributes values

66 BGP Attributes Value Code Reference ORIGIN RFC1771] 2 AS_PATH [RFC1771] 3 NEXT_HOP [RFC1771] 4 MULTI_EXIT_DISC [RFC1771] 5 LOCAL_PREF [RFC1771] 6 ATOMIC_AGGREGATE [RFC1771] 7 AGGREGATOR [RFC1771] 8 COMMUNITY [RFC1997] 9 ORIGINATOR_ID [RFC2796] 10 CLUSTER_LIST [RFC2796] 11 DPA [Chen] 12 ADVERTISER [RFC1863] 13 RCID_PATH / CLUSTER_ID [RFC1863] 14 MP_REACH_NLRI [RFC2283] 15 MP_UNREACH_NLRI [RFC2283] 16 EXTENDED COMMUNITIES [Rosen] reserved for development Most important attributes Not all attributes need to be present in every announcement From IANA:

67 Attributes are Used to Select Best Routes /24 pick me! /24 pick me! /24 pick me! /24 pick me! Given multiple routes to the same prefix, a BGP speaker must pick at most one best route (Note: it could reject them all!)

68 Route Selection Summary Highest Local Preference Enforce relationships Shortest ASPATH Lowest MED i-bgp < e-bgp Lowest IGP cost to BGP egress Traffic engineering Lowest router ID Throw up hands and break ties

69 Implementing Customer/Provider and Peer/Peer relationships Two parts: Enforce transit relationships Outbound route filtering Enforce order of route preference provider < peer < customer

70 Import Routes provider route peer route customer route ISP route From provider From provider From peer From peer From customer From customer

71 Export Routes provider route peer route customer route ISP route To provider From provider To peer To peer To customer To customer filters block

72 How Can Routes be Colored? BGP Communities! A community value is 32 bits By convention, first 16 bits is ASN indicating who is giving it an interpretation community number Used for signally within and between ASes Very powerful BECAUSE it has no (predefined) meaning Community Attribute = a list of community values (So one route can belong to multiple communities) RFC 1997 (August 1996) Two reserved communities no_export = 0xFFFFFF01: don t export out of AS no_advertise 0xFFFFFF02: don t pass to BGP neighbors

73 Communities Example 1:100 Customer routes 1:200 Peer routes 1:300 Provider Routes Import To Customers 1:100, 1:200, 1:300 To Peers 1:100 To Providers 1:100 Export AS 1

74 So Many Choices peer provider peer customer AS 4 AS5 AS 3 AS 2 Which route should AS5 pick to /16? AS /16

75 LOCAL PREFERENCE Local preference used ONLY in ibgp AS 4 local pref = 80 local pref = 90 AS 3 local pref = 100 Higher Local preference values are more preferred AS 2 AS /16

76 Example: Customer to Provider router A B import policies route selection export policies local pref = 100 select UPMC route select A s route send to other ibgp neighbors send to other ebgp neighbors /24 DT A B FT Wanadoo UPMC FT

77 Example: Peers router A B C import policies route selection export policies local pref = 90 select DT route send to other ibgp routers select A s route don t send select A s route send to customers /16 A C FT DT B Wanadoo Suppose DT, FT, and BT are peers UPMC BT

78 Example: Customers vs. Peers router A import policies route selection export policies local pref (D)= 100 local pref (B)= 80 select DT route send to other ibgp and ebgp neighbors Suppose: DT is a customer of FT and BT FT and BT are peers /16 DT UPMC A FT BT B Wanadoo

79 Example: Multiple Egress Points router import policies route selection export policies A local pref = 80 select FT route send to other ibgp B local pref = 80 select FT route send to other ibgp C local pref = 80 select BT route send to other ibgp What will router D choose? DT UPMC FT BT B A route to UPMC Wanadoo D C

80 Tweak Tweak Tweak For inbound traffic Filter outbound routes Tweak attributes on outbound routes in the hope of influencing your neighbor s best route selection For outbound traffic Filter inbound routes Tweak attributes on inbound routes to influence best route selection inbound traffic outbound traffic outbound routes inbound routes In general, an AS has more control over outbound traffic

81 ASPATH Attribute /16 AS Path = AS 1129 Global Access /16 AS Path = Sprint AS 1239 AS 1755 Ebone /16 AS Path = /16 AS Path = AS RIPE NCC RIS project /16 AS Path = 6341 AS 6341 AT&T Research /16 AS7018 AT&T /16 AS Path = /16 AS Path = AS 3549 Global Crossing Prefix Originated

82 Shorter Doesn t Always Mean Shorter In fairness: could you do this right and still scale? AS 3 Mr. BGP says that path 4 1 is better than path Duh! AS 4 Exporting internal state would dramatically increase global instability and amount of routing state AS 2 AS 1

83 Interdomain Loop Prevention AS 7018 BGP at AS YYY will never accept a route with ASPATH containing YYY. Don t Accept! /16 ASPATH = AS 1

84 Traffic Often Follows ASPATH /16 ASPATH = AS 1 AS 2 AS 3 AS /16 IP Packet Dest =

85 But It Might Not AS /16 ASPATH = /25 ASPATH = 5 AS 2 AS 2 filters all subnets with masks longer than /24 AS /16 ASPATH = AS /16 IP Packet Dest = AS /25 From AS 4, it may look like this packet will take path 3 2 1, but it actually takes path 3 2 5

86 AS Graphs Depend on Point of View peer provider peer customer

87 Implementing Backup Links with Local Preference (Outbound Traffic) AS 1 primary link backup link Set Local Pref = 100 for all routes from AS 1 AS Set Local Pref = 50 for all routes from AS 1 Forces outbound traffic to take primary link, unless link is down.

88 Multihomed Backups (Outbound Traffic) AS 1 provider primary link AS 3 provider backup link Set Local Pref = 100 for all routes from AS 1 Set Local Pref = 50 for all routes from AS 3 AS 2 Forces outbound traffic to take primary link, unless link is down.

89 Control Inbound Traffic with ASPATH Padding. Yes, this is a Glorious Hack AS 1 provider /24 ASPATH = /24 ASPATH = primary customer AS 2 backup /24 Padding will (usually) force inbound traffic from AS 1 to take primary link

90 But Padding Does Not Always Work AS 1 provider AS 3 provider /24 ASPATH = /24 ASPATH = primary customer AS 2 backup /24 AS 3 will send traffic on backup link because it prefers customer routes and local preference is considered before ASPATH length! Padding in this way is often used as a form of load 99 balancing

91 COMMUNITY Attribute to the Rescue! AS 1 provider AS 3 provider AS 3: normal customer local pref is 100, peer local pref is /24 ASPATH = /24 ASPATH = 2 COMMUNITY = 3:70 primary backup customer AS /24 Customer import policy at AS 3: If 3:90 in COMMUNITY then set local preference to 90 If 3:80 in COMMUNITY then set local preference to 80 If 3:70 in COMMUNITY then set local preference to 70

92 Hot Potato Routing: Go for the Closest Egress Point /24 egress 1 egress IGP distances This Router has two BGP routes to /24. Hot potato: get traffic off of your network as Soon as possible. Go for egress 1!

93 Getting Burned by the Hot Potato High bandwidth Provider backbone Heavy Content Web Farm SFF NYC Low bandwidth customer backbone San Diego Many customers want their provider to carry the bits! tiny http request huge http reply

94 Cold Potato Routing with MEDs (Multi-Exit Discriminator Attribute) Prefer lower MED values Heavy Content Web Farm /24 MED = /24 MED = /24 This means that MEDs must be considered BEFORE IGP distance! Note1 : some providers will not listen to MEDs Note2 : MEDs need not be tied to IGP distance

95 Customer-Provider Relationship Customer pays provider for access to Internet Customer needs to be reachable from everyone Provider exports customer s routes to everybody Customer exports provider s routes to customers Customer does not want to provide transit service Customer does not export from one provider to another Traffic to the customer Traffic from the customer advertisements AT&T d AT&T traffic d Princeton Princeton

96 Peer-Peer Relationship Peers exchange traffic between customers AS exports only customer routes to a peer AS exports a peer s routes only to its customers Traffic to/from the peer and its customers AT&T advertisements traffic Sprint Princeton d UBC

97 How Peering Decisions are Made? Peer Don t Peer Reduces upstream transit costs Can increase end-to-end performance May be the only way to connect your customers to some part of the Internet ( Tier 1 ) You would rather have customers Peers are usually your competition Peering relationships may require periodic renegotiation

98 Peering Disputes and Outcomes Peering disputes are not new, though a number of peering disputes in the U.S. and France have been increasingly publicized in recent years A peering relationship represents an agreement between two networks to exchange traffic in some agreed upon manner If one or both of the networks determine that the peering relationship no longer meets the agreed upon terms or is no longer mutually beneficial, several things can happen.

99 Congestion: Comcast ISP Interconnection and Its Impact on Consumer Internet Performance. Measurement Lab Report. October 2014.

100 Congestion: Netflix and Free Mbps

101 Paths You Should Never See ( Invalid ) Customer-provider Peer-peer two peer edges transit through a customer

102 Valid and Invalid Paths AS relationships limit the kinds of valid paths Uphill portion: customer-provider relationships Plateau: zero or one peer-peer edge Downhill portion: provider-customer relationships Valid Invalid Invalid

103 Other Kinds of Relationships Siblings A E Same company Mutual transit service Like one bigger AS B C H D F G Mergers, acquisitions, Backup Used only when failure primary Second provider Backup peering Geography-specific Customer in U.S. backup Peer in Europe

104 Backup Relationship Backup provider Only used if the primary link fails Routes through other paths AT&T USLEC Princeton /16

105 Sibling Relationship Two ASes owned by the same institution E.g., two ASes that have merged E.g., two ASes simply for scaling reasons Essentially act as a single AS AT&T CerfNet

106 An AS is Not a Single Node Multiple routers in an AS Need to distribute BGP information within the AS Internal BGP (ibgp) sessions between routers AS1 ebgp ibgp AS2

107 Internal BGP (I-BGP) Used to distribute routes learned via ebgp to all the routers within an AS ibgp and ebgp are same protocol in that same message types used same attributes used same state machine BUT use different rules for readvertising prefixes Rule #1: prefixes learned from an ebgp neighbor can be readvertised to an ibgp neighbor, and vice versa Rule #2: prefixes learned from an ibgp neighbor cannot be readvertised to another ibgp neighbor

108 Internal BGP and Local Preference Example Both routers prefer the path through AS 100 on the left even though the right router learns an external path AS 200 AS 100 AS 300 Local Pref = 100 Local Pref = 90 AS 256 I-BGP

109 An AS is Not a Single Node Multiple connections to neighboring ASes Multiple border routers may learn good routes with the same local-pref and AS path length Multiple links

110 Hot-Potato (Early-Exit) Routing Hot-potato routing Each router selects the closest egress point based on the path cost in intradomain protocol BGP decision process Highest local preference Shortest AS path Closest egress point Arbitrary tie break dst A 9 D F 5 8 E C 8 10 B 4 G hot potato

111 Some Routers Don t Need BGP Customer that connects to a single upstream ISP The ISP can introduce the prefixes into BGP and the customer can simply default-route to the ISP Qwest Nail up routes /16 pointing to Yale Nail up default routes /0 pointing to Qwest Yale University /16

112 Some Routers Don t Need BGP Routers inside a stub network Border router may speak BGP to upstream ISPs But, internal routers can simply default route AT&T BGP USLEC AS 88 Princeton University /16

113 Multihoming Get address block from one ISP example: customer gets address from ISP 1 ISP 1 s aggregation is not broken customer s prefix not aggregatable at ISP 2 longer prefix becomes a traffic magnet /16 ISP /16 ISP / /24 ISP /16 How good is load sharing? If all ISP s generate same amount of traffic for customer, then ISP2-customer link twice as loaded as ISP1-customer link /24 customer / /24

114 Multihoming Get address block from both ISPs announcement policy: announce prefix only to its parent advantage: both ISP s can aggregate the prefix they receive disadvantage: lose reliability load balancing good? depends upon how much traffic sent to each prefix ISP 3 ISP 1 ISP / / /24 customer / / /24

115 Aggegration and Multihoming Most common scenario: customer breaks its address block in 2 for load sharing purposes. YET, also advertises whole block for reliability. ISP 3 AS 1 AS / / / /23 customer /23

116 Black Holes and Cardinal Sins The cardinal sin of BGP routing is advertising routes that you don't know how to get to; called "black-holing" someone If you announce part of IP space owned by someone else, using a more-specific prefix, all their traffic flows to you. Makes that address block disconnected from the Internet. Example: black holes can happen inadvertently by non-careful aggregation /16 ISP / /21 [ ] Net A NAP /21 [ ] Net C ISP / /20 wrong!! / /16 [ ] Net B

117 Limitations to Aggregation Lack of hierarchical allocation of address space prior to CIDR (before 1995) A single AS can have noncontiguous address blocks Customer AS and provider AS can have non-contiguous address blocks Reluctance of customers to renumber their address space when they switch providers Multi-homing multi-homed prefixes require global visibility may choose not to: load sharing

118 Rules of Thumb for Aggregation To avoid black holes: an ISP is not allowed to aggregate routes unless it is a supernet of the address block it wants to aggregate In other words, an ISP has to specifically announce each IP address range of its downstream customers that are not part of its own address space.

119 Route Stability Routing instability: rapid fluctuation of network reachability information route flapping: when a route is withdrawn and re-announced repeatedly in a short period of time happens via UPDATE messages because messages propagate to global Internet, route flapping behavior can cascade and deteriorate routing performance in many places Effects: increased packet loss, increased network latency, CPU overhead, loss of connectivity

120 Causes of BGP Routing Changes Topology changes Equipment going up or down Deployment of new routers or sessions BGP session failures Due to equipment failures, maintenance, etc. Or, due to congestion on the physical path Changes in routing policy Reconfiguration of preferences Reconfiguration of route filters Persistent protocol oscillation Conflicts between policies in different ASes

121 BGP Session Failure BGP runs over TCP BGP only sends updates when changes occur TCP doesn t detect lost connectivity on its own Detecting a failure Keep-alive: 60 seconds Hold timer: 180 seconds Reacting to a failure Discard all routes learned from the neighbor Send new updates for any routes that change AS1 AS2

122 Routing Change: Before and After 0 0 (1,0) (2,0) 1 2 (1,2,0) 1 2 (2,0) (3,1,0) 3 3 (3,2,0)

123 Routing Change: Path Exploration AS 1 Delete the route (1,0) Switch to next route (1,2,0) Send route (1,2,0) to AS 3 AS 3 Sees (1,2,0) replace (1,0) Compares to route (2,0) Switches to using AS 2 0 (2,0) (1,2,0) 1 2 (3,2,0) 3

124 Routing Change: Path Exploration Initial situation Destination 0 is alive All ASes use direct path When destination dies All ASes lose direct path All switch to longer paths Eventually withdrawn E.g., AS 2 (2,0) à (2,1,0) (2,1,0) à (2,3,0) (2,3,0) à (2,1,3,0) (2,1,3,0) à null (1,0) (1,2,0) (1,3,0) (2,0) (2,1,0) (2,3,0) (2,1,3,0) (3,0) (3,1,0) (3,2,0)

125 Time Between Steps in Path Exploration Minimum route advertisement interval (MRAI) Minimum spacing between announcements For a particular (prefix, peer) pair Advantages Provides a rate limit on BGP updates Allows grouping of updates within the interval Disadvantages Adds delay to the convergence process E.g., 30 seconds for each step

126 BGP Converges Slowly, if at All Path vector avoids count-to-infinity But, ASes still must explore many alternate paths to find the highest-ranked path that is still available Fortunately, in practice Most popular destinations have very stable BGP routes And most instability lies in a few unpopular destinations Still, lower BGP convergence delay is a goal Can be tens of seconds to tens of minutes High for important interactive applications or even conventional application, like Web browsing

127 Example daily update count

128 Types of Routing Updates Forwarding instability reflects legitimate topology changes e.g., changes in Prefix, NEXT_HOP and/or ASPATH affects forwarding paths used Policy fluctuation reflects changes in policy e.g., changes in MED, LOCAL_PREF, etc. may not necessarily affect forwarding paths used Pathological redundant messages reflect neither topology nor policy changes

129 General Statistics 1996: 3-5 million updates per day in Internet core 1998: 300K-700K updates per day in Internet core 1996: average number of announcements per day was ~275K 1998: average number of announcements per day was ~400K Correlation of instability and usage instability highest during business hours instability lowest during nights, on weekends and in summer

130 Route Flap When a router/link goes down and up, it leads BGP session down BGP route withdrawal by neighbor BGP routers BGP session reset Route exchanges When it repeats, it is considered route flap BGP storms Route flap damping allows a BGP speaker to take into account the past stability of a route in deciding whether to use or re-advertise that route.

131 Who s Responsible? AS s No single AS dominates instability statistics No correlation between the size of an AS and its share of updates generated. Prefixes Instability is evenly distributed across routes. Example of measurements: 75% of AADiff events come from prefixes change less than 10 times a day % of instability comes from prefixes that are announced less than 50 times/day.

132 What is the Sound of One Route Flapping?

133 A few bad apples Typically, 80% of the updates are for less than 5% Of the prefixes.

134 Sources of Instabilities in General router configuration errors transient physical and data link problems software bugs problems with leased lines (electrical timing issues that cause false alarms of disconnect) router failures network upgrades and maintenance

135 Controlling route instability: Route Dampening track number of times a route has flapped over a period of time routes that exhibit a high level of instability in a short period of time should be suppressed (not advertised) penalize ill behaved routes proportionally to their expected future stability if a suppressed route stops flapping for a long enough period of time, unsuppress it (readvertise)

136 Interaction with Route Flap Damping Motivation for route-flap damping Flaky equipment goes up and down repeatedly Leading to excessive BGP update messages Eventually, want to suppress those updates Route Flap Damping Accumulate a penalty with each routing change for each (prefix, peer) pair Add a fixed penalty for each update message and decay the penalty exponentially with time Apply thresholds to suppress or reuse the route

137 Operation of Route Flap Damping Suppress limit Penalty Reuse limit Time Network Announced Network Not Announced Network Re-announced

138 Interaction with Route Flap Damping What are the right thresholds? Set too high Route flaps consume a lot of resources Users experience more transient disruptions Set too low Regular path exploration triggers suppression Users experience loss of connectivity Not easy to set the parameters correctly Route-flap damping disabled in many ASes

139 Conclusions BGP is solving a hard problem Routing protocol operating at a global scale With tens of thousands of independent networks That each have their own policy goals And all want fast convergence Key features of BGP Prefix-based path-vector protocol Incremental updates (announcements and withdrawals) Policies applied at import and export of routes Internal BGP to distribute information within an AS Interaction with the IGP to compute forwarding tables

140 Conclusions BGP protocol vs. policy Protocol is simple Policy is complicated BGP policy is a black art Indirect way of specifying policy Manipulating attributes to influence decisions Filtering routes to scope the routing information Common examples of policy today Business relationships Traffic engineering Security