Wresting Control from BGP: Scalable Fine-grained Route Control

Transcription

1 Wresting Control from BGP: Scalable Fine-grained Route Control Patrick Verkaik, Dan Pei, Tom Scholl, Aman Shaikh, Alex C. Snoeren, and Jacobus E. van der Merwe University of California, San Diego AT&T Labs Abstract Today s Internet users and applications are placing increased demands on Internet service providers to deliver fine-grained, flexible route control. To assist operators in addressing this challenge we present an Intelligent Route Service Control Point (), a route control architecture that allows a operator to flexibly control routing between the traffic ingresses and egresses of an ISP s without modifying existing routers. In essence, subsumes the control plane of an ISP s by replacing the distributed BGP decision process of each router in the with a more flexible, logically centralized route computation. extends the traditional BGP decision process to allow routecontrol applications to provide a per-destination, peringress ranking of traffic egresses. We describe a straightforward set of correctness requirements that prevents routing anomalies, and demonstrate through emulation that our implementation is both scalable and fault-tolerant. To illustrate the potential of application-controlled route selection, we use our prototype to implement a simple form of dynamic customer-traffic load balancing. 1 Introduction Given the best-effort communication model of the Internet, routing in the Internet has historically been purely concerned with connectivity; that is, concerned with finding a loop free shortest path between Internet endpoints. Deviations from this default behavior normally involved policy changes at fairly slow time-scales to effect business and management objectives [6]. Further, within a particular (or autonomous system), routing was realized by a fixed, fairly simple decision process, designed to ensure consistent decision making between the routers in the. As ed applications and traffic engineering techniques have evolved, however, they have placed increasingly sophisticated demands on the routing infrastructure. For example, applications such as VoIP and online gaming can be very sensitive to the characteristics of the actual chosen data path [8, 9, 18]. Numerous studies have shown non-default Internet paths can often provide improved performance characteristics [1, 2, 20], suggesting the potential benefit of making routing aware of conditions [11]. Additionally, today s operators are often required to restrict the any-to-any connectivity model of the Internet to deal with unwanted traffic in the form of DDoS attacks. Responses can take the form of black-holing traffic, redirecting it to scrubbing complexes, or even more sophisticated differentiation of unwanted traffic based on intelligence [24]. Finally, in some cases the default BGP decision process is simply at odds with provider and/or customer goals. For example, using IGP cost as a tie breaker in the decision process can lead to unbalanced egress links for customers that are multi-homed to a provider [23]. All these demands have in common the need for route control that is (i) fine-grained, (ii) informed by external information, and (iii) applied at time-scales much shorter than normal routing configuration changes. Unfortunately, BGP does not provide adequate means for performing fine-grained route control. The tuning parameters BGP does provide are both arcane and indirect. Operators are forced to tweak BGP attributes in cumbersome, vendor-specific router configuration languages at a low level of abstraction, frequently leading to ineffective or, worse, incorrect route selections. We address these requirements by presenting a logically centralized but physically distributed Intelligent Route Service Control Point (), which subsumes the BGP decision process in a platform separate from the routers in a. The concept of a logically centralized Route Control Platform (RCP) that is separate from and backwards compatible with existing routers was introduced previously [12]. The original RCP work envisioned different phases of evolution that would eventually allow the replacement of BGP as the inter-domain routing protocol. We initially presented a realization of the simplest of these phases [7]. Specifically, we demonstrated the feasibility of a centralized ibgp-speaking RCP that performed per-router route selection and thereby implemented a correct route-reflector replacement. Because of the modest scaling requirements in that scenario, each RCP instance dealt with the complete and 1

2 the system simply relied on replicated RCP instances to deal with redundancy. By integrating external information into the RCP decision process, however, we can enable a number of sophisticated connectivity management applications [23]. In particular, we can utilize intelligence to influence the route selection process. To reflect this thinking we changed the name of our architecture to the Intelligent Route Service Control Platform (). The work presented in this paper builds on this earlier work and has three important new contributions in the evolutionary path towards a new routing infrastructure: Phase-two : Following our previous taxonomy [12], the architecture presented in this paper represents a phase-two in that it enables the to communicate directly with routers in neighboring s via ebgp in addition to speaking ibgp with the routers in the. This has a number of desirable properties: First, the now has complete visibility of all routes available to the, as opposed to the ibgp-speaking (phase-one) case, where the routers only pass routes they have themselves selected to the. Complete visibility is useful for a variety of reasons. For one, as we demonstrate in this paper, complete visibility is an important ingredient for preventing route oscillations within the [3, 15, 17]. Second, given that the routers in the no longer maintain ebgp sessions with routers in neighboring s, is now (effectively) the sole controller of BGP route selection. This means that all of the s routing policy can be handled in the, as opposed to placing some policy configuration on the routers themselves. Physical distribution: An ebgp-speaking (phase-two) has two important consequences. First, because the forms BGP peering sessions with all remote routers connected to the, the scalability requirements are significantly higher than that of a phaseone : each provider edge () router at the edge of a provider typically peers with tens to hundreds of ebgp speaking routers (i.e., customer edge () routers or routers in neighboring s). Second, because routers in the now completely rely on the for routes, the redundancy requirement of the infrastructure is much more critical than with a phase-one. For these reasons, the architecture presented in this paper is physically distributed while maintaining the ability to reason about the architecture in a logically centralized manner and ensuring consistent decision making across different replicas. Application-directed route selection: Finally, since a phase-two provides all routes to routers in the there is no reason why it should be constrained to the standard BGP decision process. We exploit this possibility to ease the realization of dynamic connectivity management applications. Specifically, we allow route control applications to impact the route selection process by directly introducing a ranking of egress routes on a per-ingress and per-destination basis through a well-defined interface. We demonstrate the effectiveness of s route control interface by evaluating an example application that uses the s interface to load-balance customer traffic [23]. The key challenge to extending the BGP decision process, however, is to ensure that the resulting protocol retains BGP s robustness, scalability, and consistency properties. We prove that a simple set of constraints on the application-provided route ranking ensure that installs only safe routing configurations, even in the face of router failures or dramatic changes in IGP topology. We further show through experimentation that our prototype implementation is capable of managing the routing load of a large tier-i ISP. 2 Background We begin by providing a brief background about how routing and forwarding happens in typical ISP s. We specifically consider the role played by BGP, IGP and MPLS and describe BGP s default route selection process. Figure 1-a shows a simplified view of the physical infrastructure of an MPLS-enabled ISP backbone. The routers at the periphery of the ISP connect to customers and other ISPs (called peers). These routers are termed as Provider Edge () routers, and the routers that interconnect the routers are called Provider Core (P) routers. The customer routers connecting to s are called Customer Edge () routers. BGP allows an ISP to learn about destinations reachable through its customers and through its peers. Typically every runs BGP sessions with its attached s, and also with other s in the ISP ; the former are known as ebgp sessions while the latter are termed as ibgp sessions as shown in Figure 1-b. When a router learns a route over its ebgp session, it propagates the route to other s over the ibgp sessions which allows every to learn how to reach every customer. When a data packet arrives at a for a given customer, this ingress uses the BGP routing table to determine the that is connected to the destination customer, and forwards the packet to this. This is depicted in Figure 1-c. Since the packet leaves ISP at the directly connected to the customer, we call it an egress, and its link to the an egress link. The path between ingress and egress router is determined by another routing protocol known as an interior gateway protocol IGP. OSPF and IS-IS are two of the 2

3 provider edge router in peer ISP Peering ISP Peering ISP loaded egress link Peering ISP Customer P P ISP P provider core router Customer ISP Customer ingress routers egress routers ISP customer edge router provider edge router (a) physical link (b) MPLS tunnel and ibgp ebgp unloaded egress link (c) traffic Figure 1: ISP routing infrastructure. (a) Physical connectivity. (b) BGP and MPLS. (c) Traffic ingresses and egresses. widely used IGPs. IGPs determine a path between every pair. Thus, when a packet traverses from ingress to egress, the set of P routers it goes through is determined by the IGP running in the ISP. In an MPLS (or indeed any that utilizes tunneling technologies), when the packet goes through P routers, the P routers are not aware of the destination of the packet. They only know that the packet is going to the egress. This is achieved by setting up tunnels, or more correctly multipoint-to-point trees, between all s in ISP s (see Figure 1-b), and by prepending information about the end-point (egress ) to the packet. This obviates the need of running BGP on the P routers since they are only forwarding packets to the egress encoded in the packet and is not concerned with the ultimate destination of the packet. A usually receives more than one egress route for a given destination and must run a route selection algorithm called the BGP decision process to select the best route to use for data forwarding. The decision process (shown in the first column of Table 1) consists of a series of steps. Starting with the set of routes available to the, each step compares a set of routes and passes the most preferred routes to the next step while discarding the remaining routes. Steps 1 4 compare routes in terms of the BGP attributes attached to the routes, while steps 0 and 5 consider the IGP information associated with the egress of the route. Steps 5 and 6 are responsible for what is called hot-potato routing, i.e. forwarding traffic to the nearest (in terms of IGP distance) egress. Step 7 is a tie-breaker that ensures that the always ends up with a single best route. The uses the best route to forward traffic and if the route was learned via an ebgp session also sends this route to other s and s. 2.1 Hot-potato routing We will now use Figure 1 to explain a specific problem introduced by hot-potato routing as an exemplary illustration of the lack of fine-grained route control of the default BGP decision process. Assume for our example that the dual connectivity of the customer to the provider is for redundancy reasons so that the same destinations are reachable via both these links. All s in the provider will therefore have two possible routes to reach the customer. Assume further that most of the traffic destined to the customer enter the provider from the peering ISP. Assuming unit IGP costs for each internal provider link, the two s at the top of the figure will both prefer the route via the top connected with the customer. This in turn will lead to a complete imbalance in the traffic load on the two egress links with the top egress link carrying all (or most) of the traffic. Clearly a more desirable solution, enabled by fine-grained route control, would be to balance the load on the two egress links, by basing the route selection process for the ingress routers on a load balancing algorithm that takes into account both the offered ingress load as well as the load on the egress links where load balancing is desired. Our solution to this problem entails the allowing intelligent route control applications, for example, the load balancing application described above, to directly influence the route selection process on a per-destination, per- basis. Specifically, the allows applications to supply a ranking of the possible egress links which is taken into account during the route selection process. This partitioning of functionality allows the route control application to be arbitrarily complex, while the itself remains fairly simple. The ranking function effectively replaces the hot-potato steps (i.e., steps 5 and 6) in the route selection process. This has the desirable property that our modified decision process still honors other route attributes and their impact on the decision process. These changes to the decision process is explained in more detail in the next section. While it is possible in principle to overcome this specific problem using existing BGP mechanisms, the required configuration changes are both complicated and fragile. For example, a system could be devised to provide the appropriate ingress policy rules on all ingress edge routers so that routes from the appropriate egress link gets assigned with a higher localpref value so that (in that router) it will be preferred over other routes. However, 3

4 since localpref is an attribute with wide scope, the localpref would have to be reset before the route is advertised to other routers to prevent interference with their selection process. provider edge router ISP App 2.2 Related work Earlier work on route s [16] proposed changes to the way routes were distributed between routers, but specifically did not envision any route selection to be performed in these s. Later ebgp-speaking route s [13] similarly addressed the full-meshed connectivity problem between ebgp speakers (typically in Internet exchange points). A similar approach to ours has also been proposed to the IETF in the form of more sophisticated route reflectors [4]. The authors described a number of potential applications but restricted their proposal to changes to the ibgp infrastructure. More recently a complete refactoring of the architecture in the 4D project also proposed a logically centralized control plane that is separated from the forwarding elements [14]. Another IETF proposal on changes to the BGP route selection process is similar to our egress ranking functionality [10, 19]. The proposal defines a new extended BGP community, the cost community, which can be assigned to routes and then be used to break ties at a certain points of insertion in the BGP decision process. Their proposal does not indicate under what conditions the cost community would be safe to use; we show how our rankings should be constrained to ensure correctness (i.e., no deflections/looping). The load balancing application we are using as an example of fine grained route control is a specific instance of traffic engineering, a well-studied problem. For example, the inadequacies of hot-potato routing are also addressed in [21]. The authors propose a TIE ranking metric, which allows operators to trade off reacting to changes (like hot-potato routing does) versus a more static ranking, which might be designed to favor specific applications or services. The authors of [5] considers the optimal assignment of routes to routers to satisfy both traffic engineering and capacity constraints. Neither of these works fully deal with realization options, and indeed one can think of these solutions as potential route-control applications for our platform. 3 Fine-grained route control In this section we describe the architecture and then how our modified BGP decision process, the ranking decision process, is able to combine the directives provided by the the route control application with runtime routing information. We formulate a consistency require- customer edge router application control protocol BGP updates physical link Figure 2: Overview of the architecture. ment for the ranking decision process that prevents forwarding anomalies and show that enforcing simple constraints on the application input is sufficient to satisfy the requirement. For completeness, we discuss the role of IGP in the ranking decision process and address issues of scalability and fault-tolerance. 3.1 architecture The architecture (Figure 2) consists of a route control application, and a distributed set of s, which collectively perform route selection for provider edge routers, and customer edge routers. The function of the is to compute a routing solution in which egress routes received from routers are assigned to routers, so that a router will forward traffic it receives towards the route. carries out its function by receiving BGP routes from routers, executing a decision process to determine what routes each should use, and sending the resulting BGP routes to the s. In addition updates the s attached to a with the decision it has made for the. is a distributed system, consisting of multiple s. The motivation for this lies in requirements of fault-tolerance and scalability. If we designed as a single centralized, failure of that would leave every in the unable to forward traffic. In a distributed we can tolerate the failure of an by letting s peer with multiple s. Further, a distributed architecture allows the distribution of different instances according to the redundency requirements of the. The main scalability concern is the number of BGP peering sessions a single would have to maintain. A large Tier-1 ISP will have thousands of BGP sessions with routers in neighboring s, something that no current BGP implementation is able to support. A distributed allows us to partition the BGP sessions among different s. A second potential scal- 4

5 ability issue is the number of BGP routes that a single must store and the ensuing number of BGP updates it must process. We show in Section 5 that the number of routes does not pose a problem. The defines two types of decision processes: the unmodified BGP decision process and the ranking decision process. Both perform route selection for individual s and so are defined on a per- basis. The BGP decision process is used for the subset of destinations, unranked prefixes, for which the customer, ISP or route control application has determined that conventional hotpotato routing can be used. For the remaining, ranked prefixes the route control application determines a desirable assignment of egress routes to ingress routers (for example based on traffic load measurement) to be implemented by the (Figure 3-b). Our previous work describes the architecture for unranked prefixes [7], and so in this section we mostly consider ranked prefixes. 3.2 Ranking decision process In our architecture the route-control application is responsible for instructing the as to what ingress routers should use what egress routes on the basis of routing information, traffic load measurements, etc. Ideally, the application is able to continuously update the such that the is able to act in accordance with the current state of the. In practice, however, the, being an active participant in BGP, is in a better position to respond instantly to changes in routing state such as BGP route attributes, IGP distances, and the availability of BGP routes. We therefore require that the combine the input from the application (based on historic routing information) with current routing information. First, the application specifies a per-destination, peringress-router explicit ranking of egress links: i.e., egress links ranked by desirability. We can use egress links rather than egress routes, since each egress route corresponds to exactly one egress link. (In the following discussion we use egress link and egress route interchangeably.) Using a ranking rather than a fixed assignment of egress routes to ingress routers accommodates unavailability of egress routes. For example, if the top-ranked egress route is unavailable, the next-ranked egress route may be selected, etc. The ranking is specified per destination and per ingress router since in the architecture, runs a decision process per destination and router. We refer to an egress link using an egress link identifier, a (,) pair of the routers incident on the egress link. Next, we base our decision process for ranked prefixes on the BGP decision process: we adopt the first five steps of the BGP decision process (Table 1) and then apply the explicit ranking instead of performing hot-potato routing. This ensures that the ranking decision process respects BGP Decision Process Ranking Decision Process 0. Ignore if egress router unreachable 1. Highest local preference 2. Lowest AS path length (same) 3. Lowest origin type 4. Lowest MED (with same next-hop AS) 5. ebgp-learned over ibgp-learned 5. Highest explicit rank 6. Lowest IGP distance to egress router 6. Lowest egress link ID 7. Lowest router ID of BGP speaker Table 1: Left: the steps of the BGP decision process. Right: the steps of the ranking decision process. Steps 0-4 are identical and produce the egress set. Subsequent steps select a route from the egress set. The BGP decision process uses hot-potato routing (Steps 5 and 6), whereas the ranking decision process uses the applicationprovided explicit ranking (Step 5). Both decision processes end with a tie-breaker. BGP attributes such as AS path length and that it takes reachability of egress routers into account. It is possible to apply the explicit ranking function at any point in the decision process. e.g., to override earlier steps of the decision process, however, we leave exploring the extent to which we may safely override or replace BGP attributes as future work. As we explain in Section 3.4 we specifically do not include a step based on IGP distance (Step 6 in the BGP decision process). The resulting ranking decision process is shown in Table 1. We illustrate the ranking decision process by considering the scenarios shown in Figure 3. The runs the decision process for every in the ISP s. We examine the execution of the decision process for in Figure 3-a. First the receives all routes for the given prefix:, and. (We identify each route using its egress link identifier and assume that the destination is reachable via each of the three egress links.) Next the ranking decision process for executes steps 0 4 of the decision process and in step 2 eliminates egress route based on the longer AS path length. (We assume the routes have identical local preference, origin type and MED attributes.) The result is the egress set. In step 5 the ranking decision process applies the explicit ranking for to the egress set. Since the top-ranked egress link is present in the egress set the decision process selects this route for. Similarly, the ranking decision process selects route for, and route for s and, resulting in the forwarding behavior shown in Figure 3-b. An important observation is that steps 0-4 are identical for all s. Therefore the decision process for any computes the same egress set. Between the time that the application sends the rankings to the and the time that the ranking decision process runs, new egress routes may be announced and old routes may be withdrawn. Until the application updates its rankings, the must accommodate discrepancies 5

6 denotes egress link between E and C Ranking A 1. E C 2. F C 3. G D A ISP s B Ranking B 1. G D 2. E C 3. F C A ISP s B Ranking C 1. E C 2. F C 3. G D C D Ranking D 1. G D 2. E C 3. F C C D Ranking A 1. E C 2. F C 3. G D E F ASPlen=1 ASPlen=2 A (a) ISP s F C is eliminated G based on AS path length ASPlen=1 Ranking B 1. G D B 2. E C 3. F C BGP session Egress link Traffic E F ASPlen=1 ASPlen=2 (b) A ISP s G ASPlen=1 B Ranking C 1. E C 2. F C 3. G D C D Ranking D 1. G D 2. E C 3. F C C D E F ASPlen=1 ASPlen=2 (c) G ASPlen=1 E F ASPlen=1 ASPlen=2 (d) G ASPlen=1 Figure 3: Example of ranking and its effect on forwarding. The application (not shown) has provided the with the explicit rankings indicated. Each ranking is a list of egress link identifiers. s, and announce routes for the same prefix (with different AS path lengths) through their ebgp sessions with the. between the available routes assumed when the application creates the rankings and the actual available routes. The case that an egress route is withdrawn is illustrated in Figures 3-c and d, in which withdraws egress route, and the egress set changes to. As a result, the decision process changes its selection for s and to! and all traffic egresses through. In other words, a ranking specifies not only the desired routing for the in the absence of failure, but also the desired failover behavior that the should adopt. When new egress routes are advertised, simply appends them to the end of the explicit ranking (in order of egress link identifier) until the application is able to provide a revised ranking (implemented by steps 5 and 6 of the ranking decision process). Alternatively, the application may elect not to implicitly append routes in this manner. For example, the application may wish to restrict the set of egress routes of a particular customer to a fixed set, thereby preventing some forms of prefix hijacking. We define a virtual black-hole egress route that is part is part of every egress set and (conceptually) sinks traffic directed to it. We also define a corresponding black-hole egress ID that an application can include as part of a a s ranking. If the ranking decision process for a selects the black-hole egress route, the does not send a route to the (or its attached s), thus making the destination unavailable through that. Although the ranking abstraction can express any consistent assignment of egress routes to ingress routers in the absence of route failures, it is not powerful enough to fail over from one arbitrary assignment to another. For example a given ranking set that ranks egress link "$# highest for cannot fail over in such a way that egress link "&% is assigned to unless "'# fails. Essentially the ranking abstraction captures the concept of a preferred egress link for a and a per- fail-over behavior such that traffic does not get deflected. 3.3 Consistency The concept of application-provided explicit rankings allows a route control application a great deal of flexibility. However, it also introduces the possibility of the executing the decision process in an inconsistent manner for different s, which can lead to forwarding anomalies. Figure 4 depicts the forwarding anomalies that may result: inadvertent black-holing, deflection, and forwarding loops. We wish to prevent deflection for two reasons. First, assuming the operator has configured MPLS tunnels for optimal transport between the two endpoints of each tunnel, forwarding through an intermediate BGP router is suboptimal (Figure 4-c). Second, uncontrolled deflection can lead to a forwarding loop (Figure 4-d). Similarly we 6

7 (a) correct 2 (c) deflection forwarding path 3 3 (b) black holing (d) loop Figure 4: Forwarding anomalies. In all cases a-d the decision process for 2 has selected an egress route through 3 as best route for some destination. In a the decision process for 3 has selected a local egress route as best route, and therefore a is correct. In b the decision process for 3 has not selected any route for this destination, thus traffic is black-holed. In c the decision process for 3 has selected 1 as best egress route, resulting in a deflection. The forwarding loop in d is a result of multiple deflections. wish to avoid black-holing as in Figure 4-b since clearly it is suboptimal to carry traffic through the only to have it dropped. If the intent is for the traffic to be dropped it should be dropped at in ingress (i.e., at 2). Ultimately, the correctness of the rankings is specific to the application. can, however, consider consistency to be a minimum standard of correctness for any route control application and therefore require that a set of constraints be enforced on any set of rankings provided by an application. Definition: Deflection-free: For each ( : if egress route " is selected as best egress route for (, then some route ) is selected as best egress route for *,+.-./,"$0 (the local incident on " ) such that *1+.-./2)3054*,+.-./,"$0. Instantiating 2 for ( and 3 for *1+.-./6"'0 it should be obvious that Deflection-freedom prevents the anomalies shown in Figure 4-b and c. Claim: The BGP decision process in is Deflection-Free. Proof: Suppose for router ( egress route " is selected as best egress route. If (74 *1+.-./6"'0 we are done, so assume (98 4*1+.-&/,"$0. Since " is in the egress set of ( and all routers share the same egress set, " is also in the egress set of *1+.-&/,"$0. (Recall the definition of egress set in Table 1.) Note that for every route ) : ) is an ebgp route from the perspective of *1+.-./6"'0 iff *1+.-./2)3074:*1+.-./6"$0. Therefore the BGP decision process for *,+.-./,"$0 has at least one ebgp route when it enters Step 5 and eliminates all non-ebgp routes in Step 5. Eventually it must select an ebgp route. ; 2 2 We observe that it is possible to place a set of constraints on the application-provided explicit rankings that ensure that the ranking decision process is deflection free. Definition: We define the operator <>= as: " # <?=?" % iff in the explicit ranking for router ( egress link " # is ranked higher than egress link " %. For example, in Figure 3-a, A, we A<CB D E and D E F<?BGD 9. The constraints on explicit ranking are as follows. Definition: Ranking-Consistent-1: The set of egress routes appearing in each explicit ranking is identical. Definition: Ranking-Consistent-2: For each router ( and all egress links "&#."% : if "&#C< = "% then "&#C<IHKJLMKNOQP5"%. The rankings shown in Figure 3-a clearly satisfy Ranking-Consistent-1: all rankings contain the same egress links. They also satisfy Ranking-Consistent-2. For example, checking the ranking for B we see that (1) D 9:<?RS and ET<>U E, (2) ET<>R and EV<>UD, (3) F<?RS D and W S <?XY Y. Claim: If the explicit rankings given to a ranking decision process satisfy Ranking-Consistent-1 and Ranking- Consistent-2 then the ranking decision process satisfies Deflection-Free. Proof: Suppose for router ( egress route " is selected as best egress route. We show that " is also selected as best egress route for router *,+.-./,"$0. Since " is in the egress set of ( and all routers share the same egress set, " is also in the egress set of *1+.-./6"$0. We also know that " is ranked highest among the routes in the egress set by Steps 5 and 6 of the ranking decision process for (, and that by Ranking- Consistent-1 the same egress links appear in ( and *,+.-./,"$0 s explicit rankings. There are two cases: " does or does not appear in the explicit rankings. If " does not appear in the explicit rankings, none of the routes in the egress set appear in the explicit rankings (or Step 5 would have selected a different route for ( ). Therefore both *,+.-./,"$0 and ( identically rank the egress set using Step 6, and *1+.-./6"'0 selects ". If on the other hand " does appear in the explicit rankings, then ( has selected it in Step 5. Furthermore, with " available we know that *1+.-./6"$0 must select some route " % that also appears in the explicit rankings (and in the egress set). Suppose " % 4 " 8. "<?=I" % or Step 5 would not have selected " for (. But from Ranking-Consistent-2 it follows that also "Z<IHKJLMKN[P3"% and so *,+.-./,"$0 cannot have selected "%. ; 3.4 IGP We now consider the IGP s role in. Route selection for unranked prefixes is governed by the BGP decision process, and so the role IGP plays for unranked prefixes is the same as in BGP. Note however that an runs a decision process on behalf of a but the s position in the IGP topology is different 7

8 A,B 1. B 2. D 3. C B 1 MPLS tunnel A 1 rankings for s C and D 1 C 5 (a) IGP distance 5 4 of tunnel C,D 1. D 2. C 3. B A MPLS tunnel with ballooned distance D B C D Figure 5: Example of IGP ballooning. All routers are connected by MPLS tunnels whose distance metric is computed by IGP. (a) shows a topology based on which the application has computed a set of rankings. In (b) the distance of various tunnels has increased excessively. from that of the. As we have shown in previous work the is able to take the perspective of the on whose behalf it is running the BGP decision process by employing an IGP viewer that provides a global view of the IGP topology [7]. Further, in the distributed realization presented here, each instance () only needs IGP information from the perspective of the set of s for which it will be making routing decisions rather than for the as a whole. For ranked prefixes, we assume that the application has taken IGP distances into account when it creates the ranking. Although the decision process might conceivably rerank egress links in response to IGP distances, we generally do not let it do so for several reasons. First, for applications such as load-balancing customer traffic, strict adherence to a shortest path policy appears to be of secondary importance. Indeed, tracking IGP distance changes can have adverse effects, such as causing large volumes of traffic to shift inadvertently [22]. The explicit ranking provided by an application introduces a degree of stability, by pinning routes. If it is necessary to respond to IGP changes, we require the application to do so by providing an updated ranking. Teixeira et al. s results [21] suggest that in a large ISP with sufficient path diversity in its IGP topology the latency of MPLS tunnels is not greatly affected by IGP changes. For these cases, route pinning does not sacrifice much performance in terms of latency. However, we do wish to handle the case in which IGP distances balloon excessively, effectively making some egress routes unusable. For example, this can occur when physical connectivity is disrupted and IGP diverts traffic around the disruption. Another example is router maintenance: typically the maintenance procedure involves setting the IGP distance between the router and the rest of the to a very high value in order to gracefully move the traffic away from the router before it is brought down. Coming back to the ballooning of the IGP distance, let us look at the example shown in Figure 5. The shown has three egress routes for some given destination: 1000 (b) through s \ and. The application has assigned the egress route through to s and and the egress route through to s and. In Figure 5-b several IGP distances have ballooned, making s preferred egress route through virtually unusable for, yet s rankings have not yet been updated. We define an emergency exit procedure for cases such as these, which is as follows. If an finds that the IGP distance to a s preferred egress route balloons, the ignores the rankings for that and destination and reverts to hot-potato routing (i.e., selects the nearest egress router, possibly itself). In the example, overrides its ranking and chooses. s most preferred egress route (through ) has not ballooned and therefore deflects to, at which point the traffic egresses. As this example shows, ignoring the rankings may lead to a deflection. We consider this acceptable, since (a) in a well-engineered excessive ballooning should be the exception and (b) at most one deflection and therefore no forwarding loop can occur, which we prove below. Claim: Adding emergency exit to a deflection-free decision process introduces at most one deflection. Proof: We consider traffic entering at some ingress and assume that two deflections occur, i.e., traffic is forwarded from to some s \ and, in that order. We make two observations. First, s \ and are egress routers: both a Deflection-Free decision process and Emergency Exit forward only to egress routers. Second, applying Emergency Exit at an egress router causes traffic to egress at that. Therefore these deflections can only occur if s and do not invoke Emergency Exit, but instead follow their deflection-free decision process. Now consider a different set of traffic for the same destination, ingressing at. Since a router s forwarding behavior does not depend on where the traffic originates, this traffic follows the path ] G ^. In other words the traffic is deflected without passing through a router that invokes Emergency Exit. This implies that the decision process executed by and is not deflection free. ; 3.5 Consistency in Distributed Our previous discussion covers the execution of the decision process on behalf of each router, and describes how to maintain consistency among executions for a particular destination. However we have made two implicit assumptions: (a) when the route control application sends a set of explicit rankings to the they are provided to the multiple executions of the decision process simultaneously, (b) the routing state (i.e., the set of egress routes and the IGP state) that is input to each execution of the decision process is identical. Note that these assumptions are made only on a per-destination basis, and that (a) is ap- 8

9 plicable to ranked prefixes, whereas (b) applies to ranked as well as unranked prefixes. If we construct an from a single that runs all decision process executions, these two assumptions are easily met: (a) the route control application can provide its rankings to the in a single communication, and (b) an has a single view of the routing state. However, for scalability and robustness reasons the is a distributed system consisting of multiple s. Ignoring failure, assumption (a) can be met simply by having the route control application send its rankings to every (Figure 2). Under normal circumstances the rankings arrive at the s at approximately the same time. Assumption (b) requires additional care. To maintain consistency of IGP routing state, each runs an IGP viewer and has the same global view of the IGP topology [7]. s exchange external routes with each other using an protocol (Figure 2). Each pair of s in the establishes an session. When an learns a route from a router it sends the route to all other s through the sessions, as shown in Figure 8-a. Comparing this solution with route reflection in BGP, the most important difference between route reflection and is that a route reflector selects a single route as best route for each destination (using the BGP decision process) and only makes that route available to other route reflectors. As a result different route reflectors may observe a different set of available routes, which in turn has led to non-deterministic or divergent behaviour in BGP, e.g., MED oscillation [15, 17]. (MED oscillation continues to be observed in the Internet [25].) Basu et al. [3] show that exchanging all routes selected by steps 0-4 of the decision process is sufficient to prevent non-determinisism and divergence in route reflection. 1 In we exchange a superset. 3.6 Fault tolerance We now discuss how handles a number of common failure scenarios including the loss of customer connectivity to, failure of individual sessions and IGP failures. First we examine failure in customer connectivity to, by which we mean the ability for a to announce or learn reachability of a route to or from. As is apparent from Figure 6-a, connectivity can be disrupted by failure of an, a, an ebgp session, an ibgp session or a physical link. (We consider failure of the the customer s responsibility.) Robustness to failure of any of these components can be im- 1 At least to the extent that the anomalous behaviour is contained within a single. (a) (c) ebgp ibgp physical link (b) (d) Figure 6: Replicating customer connectivity. (a) No replication. (b) Replication of - connectivity. Replication of the - ebgp session. (d) Replication of all components that make up customer connectivity. proved by having the customer connect to several s and several s, as shown in Figure 6-d. In addition, individual components can be made more robust as shown in Figure 6-b and c. 2 Since s do not re-advertise updates learned from other s, each can only learn a particular route from one (i.e., the that learned the route via an ebgp session with a router). A failure of an session can therefore cause s to learn a different set of routes, potentially leading to inconsistency. Our aim is to prevent inconsistency from leading to a situation in which (a) s send updates to a that are inconsistent with updates sent to other s, (b) several s send inconsistent updates to the same. Among many numerous potential causes underlying an session failure we focus on misconfiguration of peering and partitioning. We first discuss the case of a (IGP) partitioning. By definition, a cannot have connectivity to s in different partitions. Furthermore, there is no reachability between s in different partitions and therefore no inconsistent routing between such s. It follows that it is sufficient to ensure consistency among the s within each partition separately. Next we examine inconsistency within a par- 2 Figure 6-c introduces two egress routes for a single egress link, implying that Step 6 of the ranking decision process does not always decide. In practice we add a tie-breaker on ID at the end of the ranking decision process. The ID of an egress route identifies the that receives the route on an ebgp session. 9

10 RIB of 1 (a) (b) (c) D destination A B destination E C F destination destination egress sender link ID D A D D B E C F C 2 (b) attributes (d) (f) (e) session IGP partition (a) 1 destination A E D 2 C session B BGP session physical link 3 F destination destination (c) Figure 8: route propagation example. Figure 7: Healing the graph within a partition. (a) Complete graph. (b) Incomplete graph. (c) Partitioned graph. (d) Adding missing peerings to an incomplete graph. (e) A partitioning. (f) Partitioned graph in which all but one partition consists of a single. tition. We define an graph consisting of s (vertices) and sessions (edges) within a single partition. In the absence of failure of sessions within the partition the graph is complete and consists of a single connected component (Figure 7-a). Failures of sessions within the IGP partition can cause the graph to become incomplete (Figure 7-b) or even partition (Figure 7-c). We treat these two cases separately. To handle failures within one partition in the graph, we have s signal to each other (through the sessions) which s are in their partition. If an _ notices that it does not have an peering with some ` in its partition, it establishes the peering with ` (Figure 7-d). For failures that partition the graph, we consider two subcases. The case shown in Figure 7-c in which multiple graph partitions contain more than one we assume to be highly unlikely, since it implies that all s in every such partition are misconfigured. We do not handle this case. If on the other hand an finds itself alone in an graph partition (i.e., it cannot establish any sessions), it proceeds by checking its IGP viewer to see if any of the s it is trying to contact are present in its IGP partition. If none of them are, the graph partition is in fact a partition (Figure 7-e), and the continues to function correctly (see above). If the does find one of the other s in its partition it is highly likely that there is a problem with the itself, and the halts (Figure 7-f). Here we rely on replication of s (discussed earlier) and let another take over. Effectively, this procedure aligns each graph partition with the corresponding partition. 4 Implementation Having discussed our design for scalable, fault-tolerant and correct fine-grained route control we now turn to our prototype implementation of this design. As should be clear the interacts with BGP routers and runs a decision process very similar to the BGP decision process. As such a significant part of its functionality is identical to that of a BGP router, and our prototype system is therefore based on a BGP protocol stack implementation. In particular, we have chosen to use the code base of the open-source openbgpd router, version 3.9. Below we discuss our modifications to the code base to implement the protocol and RIB (Section 4.1) and the per- decision process (Section 4.2). We rely on the underlying BGP protocol implementation to support common filtering policies [6]. While we implemented an IGP viewer for a previous centralized control platform [7], we have not yet ported it to our prototype. 4.1 Protocol and RIB We have previously established that to guarantee consistent decision making s should distribute all available routes to each other. We now describe the protocol through which the routes are distributed. Figure 8-a shows an example in which a route is propagated from a router to an (through BGP) and from there to other s (through the protocol). We discuss subsequent propagation of routes to BGP routers later. s exchange routes using a simple extension to the BGP protocol. A pair of s maintain a TCP-based session over which they exchange incremental updates in the form of advertisements and withdrawals of routes. At session startup the s exchange a series of advertisements correspond- 10

11 ing to all known routes, similar to normal BGP sessions. When an session goes down, all routes exchanged previously on the session are implicitly withdrawn. When an receives routes from BGP routers and from other s, it stores the routes in a routing information base (RIB), so that the routes are available to the decision process and for further propagation to routers and s. For example, the RIB of 1 shown in Figure 8-b contains four entries for prefix Each entry has fields for the destination prefix, the egress link, the neighbor of the from which the route was received, and BGP attributes that belong to the route. In the example, has sent routes to 1 and 3, resulting in the first two entries. The last two entries correspond to routes sent to 2 by s E and F. The data structures used to implement the RIB are adapted from openbgpd s implementation of the BGP RIB. openbgpd provides various indexes into the BGP RIB, the most important of which is a Red-Black Tree of destination prefixes, where each entry points to a list of routes (one for each neighbor). We adopted this structure, despite the fact that the number of routes per prefix in increases to one route per egress link, thus proportionally increasing the search time for a route in the RIB. We present the performance evaluation of our implementation in Section Per- Decision Process Based on the routes stored in the RIB for a destination prefix, an is able to run the decision process for each of its neighboring s. Note, however, that the first five steps of the (BGP or ranking) decision process (which compute the egress set) are independent of the. Therefore a straightforward optimization that our implementation makes is not to execute the entire decision process on a per- basis. Rather an computes the egress set once for a given set of routes, and only executes the subsequent steps on a per- basis. Application rankings are stored in a per- Red-Black Tree of destination prefixes, where each prefix points to the ranked list of egress identifiers for that and prefix. Applications currently provide rankings to through a file, which can be transferred using any convenient file transfer protocol. When a new ranking file arrives, the updates its in-memory ranking tree and reruns the decision process for affected destinations. Following (re-)execution of the decision process for a given, the distributes its decision to the and to all s attached to the (shown in Figure 8- c). Note that the does not need to run a decision process for a : as in BGP, the route sent to the is the same as is selected for the associated. 5 Evaluation In this section we first present a functional evaluation, demonstrating that an example load-balancing application is able to effect fine-grained route control using the ranking abstraction and our prototype implementation. We then evaluate the performance of our prototype implementation, paying particular attention to memory usage, processing throughput of updates, and number of peering sessions supported. 5.1 Functional Evaluation We verified the functionality of with a small testbed as depicted in Figure 5.1. Our testbed consisted of an, a simple load balancing route control application, three ingress s (Ingress1, Ingress2, and Ingress3), two egress s (Egress1 and Egress2), a core router (P) and a number of hosts. P uses point-to-point links to connect to the s and we used GRE tunnels as the tunneling technology between the Ingress and Egress s. The load-balancing application performs an SNMP GET query to each of the s every 30 seconds to collect the offered and egress loads. Based on the offered load, the application computes a new ranking set in order to balance the load on the egress links. If the new ranking set is different from the ranking set active in the, the application generates a new configuration file with the new ranking set, and issues a configuration reload command. The hosts in the source (on the left) send constant rate UDP packets to the hosts in the destination (on the right). The offered load at the three ingress s as measured by the application is shown in Figure 5.1. The loads at the Ingress1 and Ingress2 are about 100kbps and 128kps throughout the experiments. The offered load at Ingress3 is zero at the beginning and end of the experiment, but increases to approximately 256kpbs from around minute 26 to minute 47. The load at the egress s links is shown in Figure 5.1. Before minute 27, the load at these two egresses are balanced as a result of the initial ranking set at the (i.e., Ingress1 prefer Egress2, and Ingress2 prefer Egress1). At around minute 26, the offered load at Ingress3 increases, and the application might take up to 30-second (the SNMP polling interval in our experiment) to detect this change. It then realizes that the best way to balance the load is to let load from Egress1 and Egress2 go to one egress point, and let the load from Egress3 go to another. A new ranking is generated and communicated to the. Within one second, the finishes reloading the new configuration, and sends the updates to the ingress s, and the s changes their 11