BGP Convergence in Virtual Private Networks

Transcription

1 BGP Convergence in Virtual Private Networks Dan Pei Jacobus Van der Merwe AT&T Labs Research Abstract Multi-protocol label switching (MPLS) virtual private networks (VPNs) has had significant and growing commercial deployments. VPNs often carry applications, such as VoIP, data replication, and financial transactions that do not react well to even the smallest disruptions in connectivity. From previous studies performed on data from the public Internet, it is well-known that BGP can have significant traffic disruptions due to its slow convergence. However, despite the fact that BGP is a key component of MPLS VPNs, the use of BGP in MPLS VPNs has been largely been unexplored by the research community. In particular, BGP convergence properties in MPLS VPNs has not been studied. In this paper we present the first systematic study of BGP convergence in MPLS VPNs using data collected from a large tier-1 ISP. For our analysis we combine several data sources to produce a methodology to accurately estimate routing convergence delays. Despite VPN s obvious smaller scale than the public Internet, we found that BGP convergence in VPNs suffer from many of the same ill effects as BGP in the public Internet. Specifically, we discovered an ibgp version of the BGP path exploration, and showed that the route invisibility problem occur frequently and is one of the most significant contributing factors to BGP convergence delay in VPNs. We therefore propose and evaluate several configuration changes that can be employed to greatly improve the routing convergence time and minimizing the connectivity disruption in the face of network changes. 1 Introduction Multi-protocol label switching (MPLS) virtual private networks (VPNs) [22] have had significant and growing commercial deployments. VPNs often carry business applications, such as VoIP, data replication, and finanical transactions that do not react well to even the smallest disruptions in connectivity. Therefore, the timely convergence of routing protocols in the face of network events is critically important to the continued operation of these applications. Despite the importance of MPLS VPN networks, their routing behavior has not been studied by the research community, with the notable exception of [6] which formally analyzed the necessary configuration conditions to ensure the correct MPLS VPN operation. As is the case in the public Internet, Border Gateway Protocol (BGP) [20] plays a key role in MPLS VPNs. Specifically, multi-protocol extensions to BGP [4] allows routes associated with VPN sites to be distributed via the provider BGP infrastructure. It is well-known that in the public Internt BGP can have significant traffic disruptions due to its slow convergence [10, 11, 9, 16, 14, 15, 19]. However, the BGP convergence in MPLS VPNs have not been studied by the research community, no doubt because of the paucity of data. In this paper we present an analysis of the convergence of BGP in MPLS VPNs using data obtained from a large Tier-1 ISP. To the best of our knowledge this study is the first of its kind. Our study made use of several data sources from the provider network. Specifically, we used router configurations, forwarding table dumps, syslog messages as well as BGP update messages. Combining these data sources allowed us to develop a new methodology to accurately determine the BGP convergence times caused by network events. Our methodology has similar accuracy to beacon based measurement approaches [10, 11, 15], but because it can be applied to BGP data from an operational network is more representative, in much the same way as passive measurement approaches [21, 3, 13, 24, 8]. In our analysis we discovered an ibgp version of the well-known path exploration phenomena [10]. Further, there is a route invisibility problem in which certain network configurations can also cause alternative (backup) paths to remain hidden from the network, until after the route for the primary path has been completely withdrawn, thus causing periods of disconnect during convergence. Our measurement results show that, while most convergence delays we observed were relatively short (less than 20 seconds), significant convergence delays can occur. We also found that route invisibility occurs frequently and significantly contributes to the total convergence delay, while path exploration does not. Fortunately, most of the factors contributing to convergence delay can be either eliminated completely or significantly mitigated through a series of router configuration changes. Through measurement-based estimation, we show that our proposed configuration changes can greatly reduce 1

2 Figure 1: Components of an MPLS VPN the convergence delay in the MPLS VPN networks. 2 Background In order to put our work in context, we now present a brief overview of MPLS VPNs and specifically the role played by BGP. In Figure 1 we show the essential components that constitute an MPLS VPN. The figure shows two VPNs: sites A, D and E are part of VPN I and sites B, C and F are part of VPN II. A customer-edge (CE) router at each site connects to a provider-edge (PE) router in the provider network. The PE routers in turn connect to a set of provider (P) routers in the core of the provider network. The figure also shows a set of route-reflectors (RR) that takes care of route distribution within the provider network 1. Note that only the routers in the provider network (i.e., the PE, RR and P routers) are MPLS aware. The provider network provides connectivity between the different VPN sites in such a way that separation of traffic between VPNs are maintained. A key construct to achieve this separation is the virtual routing and forwarding (VRF) table that is associated with each VPN on each PE. Each VRF table contains only routing information related to the VPN it is associated with. First, the VRF contains routes associated with the directly connected CE(s) which it typically obtains via an ebgp session between the PE and the CE 2. In much the same way as regular IPv4 BGP, these ebgp learned routes need to be distributed to other PEs via ibgp to allow remote PEs to reach the prefixes in question. This is achieved through the use of a VPN specific address family (VPNv4) in multiprotocol-bgp (MP-BGP) ibgp sessions between the PEs (or between the PEs and RRs) 3. 1 In order to simplify the diagram we show route-reflectors to be coresident with provider routers. 2 Routes associated with the CE can also be statically configured. 3 A VPN specific BGP extended community value, called route-target A salient feature of the VPNv4 address family for our discussion is the eight byte route-distinguisher (RD) with which each VRF is configured. When routes associated with a VRF gets distributed via the MP-BGP session, this RD gets added to each IPv4 prefix to form a VPNv4 prefix (i.e., (RD:IPv4-Prefix)). Inside the provider network, the complete VPNv4 prefix is used for route comparison as part of the route selection process. This allows different VPNs to use the same address space (e.g., 10./8), without any conflict (assuming of course that they are configured with different RD values). Once routes are imported into the VRF tables, however, the RDs are effectively ignored and only the IPv4 prefixes are compared for route selection purposes. Note that the way RDs are assigned within a particular VPN also impacts how routes are distributed, especially with sites that are multi-homed to the provider network. For example, CE 2 in Figure 1 is connected to both P E 1 and P E 2. If the VRFs associated with this CE on P E 1 and P E 2 are respectively assigned different RD values, then the prefixes advertised by CE 2 through both P E 1 and P E 2 will be considered unique VPNv4 prefixes and be advertised to all PEs in the provider network, i.e., each PE associated with this VPN will therefore have two possible routes to reach site B. On the other hand, if the VRFs in P E 1 and P E 2 are assigned the same RD for CE 2, then the VPNv4 prefixes advertised by P E 1 and P E 2 on behalf of CE 2, will look the same and each route-reflector will therefore have to select only one of these routes to advertise to other PEs in the network. 2.1 Previous BGP Convergence Studies The convergence properties of BGP in the public Internet have been well studied over the last several years. It was first studied by [10], which showed that on average, it can take 3 minutes for the whole Internet to switch from failed routes to valid ones for a given destination. In some cases, it may even take up to 15 minutes for the routing tables in all routers to stabilize. BGP slow convergence was later simulated in [9], shown in [14] to be exacerbated by the BGP route flap damping mechanism [23], revisited by [15], and analyzed in [11, 16, 19]. As a path vector routing protocol, BGP s routing update messages include the entire AS path to each destination. After a topology change (e.g., link or node failure) or policy change that invalidates a current best path, the router will select a new best path. The router, however, may mistakenly choose and propagate a path that has been obsoleted by the very same topology (or policy) change. This obsolete path may, in turn, be chosen by other nodes as their new best path, resulting in an invalid path being propagated throughout the network. This process is called path governs what routes are actually imported into a specific VRF table. 2

3 exploration in the literature. Furthermore, BGP uses a Minimum Route Advertisement Interval timer (MRAI timer) to space out consecutive updates, however the MRAI timer can also delay the propagation of valid reachability information [11, 9]. Many solutions to improve BGP convergences have been proposed [10, 18, 12, 5, 17, 7], but none of them has been deployed. The measurement works on the BGP convergence delays have been focused on IPv4 BGP. These works can be classified into two types: beacon-based active measurements [10, 11, 15], in which controlled events were injected into the Internet from a small number of beacon sites, and time window-based passive measurement [21, 3, 13, 24, 8]. Both types measures the convergence time at some vantage points (RouteViews, RIPE, or some private monitors) by grouping the BGP updates into events. Each event is a collection of BGP updates with timestamps close to each other. The difference between the active and passive measurements relates to the way updates are grouped into events. The passive measurements use a time-window (e.g., 15s, 30s, 45s, or 70s, typically derived based on the update interarrival time), and two updates separated by more than the time-window value belong to two different events. Some works, such as [3, 13, 24, 8], also correlate the updates across different prefixes to help cluster the updates into events. Active measurement determine that a new event happens around the scheduled time when the event is injected, and determine the event finishing time using a timewindow too. Clearly, the starting time of the active measurement is more accurate, but active measurements are usually small-scale in that the prefixes are from a limited set of origin ASes. Passive measurement can measure the real events for all prefixes happening in much diversified topologies and locations, but suffers from lower accuracy due to the lack of root cause information of the events. 3 BGP Convergence in MPLS VPNs With reference to Figure 2, there are three factors that contribute to the BGP convergence time that is triggered by a link failure. First, it takes time for BGP to detect the failure, because the route will not actually be withdrawn until this failure is detected. Second, BGP messages need to be distributed through the ibgp topology, which in practice normally consist of a route-reflector hierarchy. Third, there is an ebgp component, both in terms of routes advertised from CEs to the provider network, as well as routes being advertised from the provider network to CEs. In Section 3.1 we present definitions of BGP convergence events in the context of MPLS VPNs. Next, in Section 3.2 we consider ibgp specific factors that contribute to convergence. In Section 3.3 we derive an upper bound on BGP convergence for the different types of BGP convergence events, while taking into account the three contributing factors described above. 3.1 BGP Convergence Event Definitions The key metric for BGP convergence is the time it takes for the network to converge on a stable set of routing tables after a routing change that was triggered by some network event. This basic BGP convergence metric is the same in both the public Internet and MPLS VPN environments, and we borrow some terminology from earlier work on Internet BGP convergence [10]. Specifically, we use the same four types of convergence events identified in [10], namely T down, T up, T long, and T 4 short. We summarize the meaning of these events types in Table 1 and describe them below in MPLS VPN context with the aid of Figure 2. The figure shows a simple network topology with two VPN sites that are part of the same VPN. (Figure 2 has a fairly elaborate route-reflector setup, which will be used to explain VPN specific phenomena later in this section). The most basic BGP convergence event would be T down, the withdrawal of a route because of a link failure. For example, if the link between CE 2 and P E 3 in Figure 2 were to fail, any routes previously advertised by CE 2 would be withdrawn, and the convergence delay due to this T down event would be the time taken for this information to propagate through the network to CE 1. If the link between CE 2 and P E 3 were subsequently to recover, i.e., a T up event, the convergence delay would be the time taken for the routes advertised by CE 2 to propagate through the network to CE 1. Note that after a T down event, reachability between the two VPN sites are lost and it is restored again after a T up event. Therefore, while these two events relate to the protocol convergence time, a user of the VPN might be more interested in the time between a T down and a T up pair of events. In this work, however, we do not study this time and focus only on the protocol specific events. As shown in our example network in Figure 2, CE 1 is multi-homed to the provider network and is configured such that routes advertised via P E 2 will be more preferred (because it has a shorter AS path), than the same routes being advertised via P E 1. In a stable environment the path via P E 2 and CE 1 will therefore be the primary path to reach Site1, while P E 1 to CE 1 will be the backup path. In this case, if the link between CE 1 and P E 2 were to fail, BGP will converge to using the backup path, i.e., a T long event. If the link between P E 2 and CE 1 were to be restored, BGP 4 Note that, however, these event definitions cover only a subset of all the BGP events, namely, those caused by the PE-CE link failure/recovery, for which we have root cause information via syslog messages. Therefore, these event definitions do not cover other events such as router crashes or reboot, OSPF changes within the provider network, or the ebgp events propagated from a VPN site by the CE to PE into the provider networks. 3

4 Event T down T up T long T short Description Prefix unreachable after link failure Prefix reachable again after link recovery Prefix reachable via backup after primary path fails Prefix reachable via primary after primary path recovery Table 1: BGP Convergence Event Types time RR 4 s updates RR 4 s table before (RR 4,RR 3,P E 2 ) (RR 3, P E 2 ), (RR 2, P E 2 ) the failure T=0 failure happens (RR 3, P E 2 ), (RR 2, P E 2 ) T=0.7s (RR 4,RR 2,P E 2 ) (RR 2, P E 2 ) T=4.7s withdrawal none T=9.7s (RR 4,RR 1,P E 1 ) (RR 1, P E 1 ) Table 2: ibgp signal path of RR 4 s update and routing table. Figure 2: BGP Convergence an MPLS VPN would switch back to the primary path, i.e., a T short convergence event. Note that from a user perspective, during a T short event a valid path (the backup path) exists and is operational. Therefore the time taken to switch back to the primary path would normally not be that critical. On the other hand, during a T long event, the primary path is not operational, and therefore connectivity is not restored until after the T long convergence event. From a user perspective, reducing the convergence time related to a T long event is therefore crucial, especially because the backup link might have been installed precisely to provide redundancy in the face of these types of failures. We therefore emphasize factors related to T long in subsequent sections. 3.2 ibgp Convergence in MPLS VPNs To explain details of the ibgp convergence process we again make use of Figure 2. In this case we focus on the case where the link between P E 2 and CE 1 fails, and we consider the BGP convergence from the point of view of P E 3 and the BGP messages it receives from RR 4. Table 2 shows a summary of the update messages sent from RR 4 to P E 3 as well as the state of RR 4 s routing table. The event times shown in Table 2, are representative measured times from a testbed setup corresponding to Figure In the testbed RR 1 through RR 4 and P E 1 and P E 2 were Cisco routers, while CE 1 and P E 3 were software routers running the Quagga open source routing software. ibgp signaling path. We can construct the ibgp signaling path in an update, based on two attributes (originator and the cluster-list) in the updates. The originator attribute indicates the PE who originally injects the route into the AS. In our case, the originator of the primary path is P E 2, and originator of the backup path is P E 1. The cluster-list indicates the route reflectors traversed by the updates. For example, the route learned by P E 3 and RR 4 before the link failure, has the the ibgp signaling path of (RR 4,RR 3,P E 2 ), meaning that the path was first injected by P E 2 into the AS, then propagated to RR 3 s cluster, and then propagated to RR 4 s cluster. Table 2 shows that, before the failure, RR 4 has two ibgp paths, (RR 3, P E 2 ) and (RR 2, P E 2 ). One might wonder why RR 4 does not have the ibgp path (RR 1, P E 1 )? This is due to the route invisibility problem in ibgp: RR 1 prefers the primary path over the backup path, thus selects the primary path and sends it to P E 1. P E 1 compares the primary path and its own path, and selects the primary path as its best path because it is shorter than P E 1 s own path (the backup path). Because a router can only announce its own best path, P E 1 has to send a withdrawal to RR 1 to withdraw the backup path. That is, the backup path is visible only to P E 1, and is invisible to the rest of AS The invisibility of the backup path could be by design (e.g., the VPN customer wants all the traffic to go through the primary path when the primary path is available), or it can be unintentional. Nevertheless, the backup path is invisible until the primary path is withdrawn, and as we will show below, this causes significant convergence delay after the failure of the primary path Path Exploration in ibgp At T = 0 second, we tear down the ebgp session between CE 1 and P E 2, and P E 2 loses the primary path. Because P E 2 does not have any other path in its routing table, it sends a withdrawal message to RR 2 and RR 3 to withdraw the primary path. One might expect that RR 4 will quickly send a withdrawal to P E 3. But as shown in Table 2, at T = 0.7 second, RR 4 announces a path (RR 4,RR 2,P E 2 ) to P E 3. Note that at this point in time this path is in fact invalid since P E 2 does not have a route to the destination at T = 0.7 second. 4

5 What happens is the following. Due to background BGP load at RR 2, it processes the withdrawal from P E 2 much slower than RR 3 does. (In the testbed background BGP traffic was provided by an Agilent router emulator connected to RR 2.) Eventually RR 2 will process the withdrawal from P E 2 and send a withdrawal to the rest of the reflectors. However, its withdrawal arrives at RR 4 somewhat later than RR 2 s. As a result, RR 4 receives and processes the withdrawal from RR 3 first and computes the new (invalid) best path, (RR 2, P E 2 ), which is not yet withdrawn by RR 2. This new route is sent to P E 3 at T = 0.7 second, as shown in Table 2. Note that in the testbed environment, the above update propagation (P E 2 RR 3 RR 4 P E 3 ) does not involve any MRAI delay because the updates are the first to be exchanged on the sessions involved. However, after that the MRAI timers are turned on for these sessions, including the one from RR 4 to P E 3. This means that no other update can be sent over this session until M seconds later (M is by default set to 4 to 5 seconds). Therefore, when RR 2 finishes processing the withdrawal from P E 1 and sends its own withdrawal to RR 4, RR 4 realizes that there is no path in its table, and it sends a withdrawal to P E 3 at T = 4.7 second. In this process, RR 4 in effect explores a path (RR 2,P E 2 ) that is already invalidated by the failure (CE 1 /P E 2 session failure) that has triggered the convergence. This process is similar to the well-known ebgp path exploration problem [10], but, to the best of our knowledge, path exploration has never been reported in the context of ibgp Route Invisibility Problem As shown in Table 2, at T = 4.7 second, RR 4 withdraws the primary path, but the backup path is not sent until T = 9.7 second. This additional delay is caused by the invisibility problem. In order for P E 1 to announce the backup path to RR 1 and then to the rest of the network, P E 1 needs to first learn from RR 1 that the primary path is no longer valid. But, similar to RR 4, P E 1 experiences the path exploration process. RR 1 sends (RR 2,P E 2 ) to P E 1 at around T = 1 second, and then a withdrawal at around T = 5 second. P E 1 then selects the new best path, i.e., the backup path ( ). The route is propagated to RR 1, then RR 4 and P E 3. Because there has been no update on session P E 1 RR 1 and session RR 1 RR 4, the backup route is propagated over these two sessions with little delay. However, the MRAI timer on session RR 4 P E 3 is on due to the update at T = 4.7 second in Table 2, therefore, the backup path cannot be propagated to P E 3 until T = 9.7 second. The route invisibility problem impacts the convergence time in that routing updates needs to go through several ibgp hops to make the backup path available to the net- Timers Keepalive/Hold-down bgpscan (nexthop validation) fast external fail-over ibgp MRAI VRF import ebgp MRAI Default Value 60/180 seconds 60 seconds turned off 5 seconds 15 seconds 30 seconds Table 3: Timers and Cisco defaults values work. First, the withdrawal of the primary path needs to propagate through the reflector hierarchy to reach the PE which has the backup path. Second, the backup path is propagated through the reflector hierarchy to reach the rest of the PEs. Note that in the testbed scenario described here, there is no background BGP updates except on the session between the router emulator and RR 2. When there are background BGP updates on each session, as will be the case in any operational network, the per-neighbor MRAI timer is constantly on when a new update arrives at a router. Then each ibgp hop can cause a delay up to M = 5 seconds. In the following section, we provide worst case analysis of the convergence delays in a VPN network. 3.3 Worst Case Analysis The actual delay of a convergence event depends on a lot of factors such as network topology, timer setting, and type of events. In this section, we will present the analysis details T long events, but will briefly mention the results for other types of events. In our analysis, we assume a one-level route reflector topology in the provider network, and the event is a PE-CE link failure or recovery. Each PE is connected to up to n reflectors (e.g., n = 2 in Figure 2). Table 3 shows the timers relevant to convergence delay and the default values on Cisco routers. Suppose the PE-CE link fails, as described above, the fail-over convergence delay consists of three parts: failure detection, ibgp route exchanges, ebgp route propagation. Failure detection: The time for the PE to detect the failure of primary path (learned from the CE) can be 0 to 180 seconds. In the simplest setting, when the PE has received no updates or KeepAlive messages from the PE neighbor in the last 180 seconds, the hold-down timer at the PE will expire and the PE will tear down the PE-CE session. And the primary path is withdrawn. Therefore, depending on the residual value of the hold-down timer, the primary path is withdrawn by the PE with a delay of 0 to 180 seconds. In a more complex setting, if the failure of the link PE-CE causes the nexthop to reach the prefix to become unreachable at the PE 6, the bgpscan process (nexthop validation) at the PE will detect it, and the PE will withdraw the primary 6 Note that the nexthop might still be reachable if there are super-net prefixes covering the prefix to be withdrawn. 5

6 path. Because the bgpscan runs every 60 seconds by default, the primary path is withdrawn with a delay up of 0 to 60 seconds, depending on the smaller value of the bgpscan timer and the hold-down timer. If the fast external fail-over feature is turned on (the default setting is off), the session is torn down immediately after the failure is detected at the layer 1. In this case, there is very little delay for the PE to withdraw the primary path. The fast external turnover is typically turned off in the Internet because this might help damp the transient link failures and reduce session flaps. ibgp route exchange: The ibgp route exchange time is the time between when the PE withdraws the primary path and when the remote PEs learn the backup path. For the purpose of our analysis, we call P E 2 in Figure 2 primary PE, P E 1 backup PE, RR 3 (RR 2 ) primary reflector, and RR 1 backup reflector. It can take 2 ibgp hops for the withdrawal to be propagated from the PE to the reflector with (primaryp E primaryref lector backupreflector), and there can be up to 5 seconds delay at each ibgp hop. Then at the backup PE, it can experience the path exploration for up to (n 1) 5 seconds, where n is the number of reflectors that the primary PE is connected to (thus, the number of failed alternative ibgp paths). Then it takes 1 ibgp hop for the withdrawal to propagate to the backup PE, and then 3 ibgp hops for the backup path to be propagated to the remote PE s. (backupp E backupref lector remoteref lector RemoteP E), each of which can contribute to one MRAI delay of up to 5 seconds. In total, the ibgp route exchange can contribute (2 + (n 1) ) 5 = (5 + n) 5 seconds delay. ebgp route propagation: The ebgp route propagation time is the time from when a remote PE receives the backup path to when the remote PE sends the backup to the remote CE. The routes learned via BGP is imported into the PE s VRF every 15 seconds (controlled by the vrf import timer), thus there can be a delay of 0 to 15 seconds caused by the vrf import timer. Then the remote PE sends the route to the CE, via ebgp, and this can contributes a delay of 0 to 30 seconds. Summarizing the three parts results based on the Cisco default value, the total convergence delay for T long is 180+ (5 + n) = n 5 seconds. T down results can be analyzed similarly, and it is (235+n 5). In T up and T short, the ibgp route exchanges and ebgp route propagation in total contribute 55 seconds. However, there seems to be no upper bound for the ebgp session establishing time, or for the the subsequent table transfer time. Thus we will leave the further investigation as our future work. 4 Measurement Methodology In this section, we present a methodology to accurately measure the BGP convergence time in a VPN network through correlating data from several sources that are readily available or collect-able from operational networks. Our goal is to measure the time it takes for a BGP VPN network to convergence after a PE-CE link/session status change. The two main data sources used in our study are syslog and BGP update messages. Syslog message from the routers in the provider network provide timing and location information of PE-CE link/session status changes (i.e., the starting time of an event). BGP update information provide an estimate of when the BGP finishes converging to the new routes as the result of link/session status change. In order to correlate the events from these two data sources, we also need two auxiliary data sources, namely router configuration files and forwarding table dumps. We describe the data correlation process in detail in the next section. In Section 4.2 we show how we used the correlated data to perform event clustering and in Section 4.3 we describe how we detected the different BGP convergence events introduced in Section Correlating Data Sources The provider network we studied collects both VPN BGP updates and syslog messages and has one level of route reflectors that form a full-mesh. One BGP collector is set up as the client of two route reflectors to collect the BGP updates from them. Among the many types of syslog messages, the layer 1, layer 2, and layer 3 messages are relevant to our study. The available information in these syslog messages, as well as in BGP updates, are shown in Table 4. As shown in Table 4, there is no common field in BGP update and a syslog message. The BGP updates have prefix and RD information, but the syslog mesages have only the interface/session information. Therefore, to correlate the syslog messages with the BGP messages, we want to build a table in which, for each PE-CE session in the provider network, there is a list of RD:prefixes announced via the session. More specifically, we want to build the following two mappings: 1. (router, interface) RD : prefixes. This tells us the RD : prefixes affected by first two syslog messages (data sources 3 and 4) in Table 4 2. (router, neighbor ip, vrf) RD : prefixes. This tells us the RD : prefixes affected by the third syslog message (data source 5) in Table 4. Building these tables turns out to be challenging in that RD:prefixes are typically not statically configured, but are 6

7 Data Sources types available information 1. BGP updates announcement timestamp, prefix, rd, aspath, cluster-list, originator BGP updates withdrawal timestamp, prefix, rd 3. syslog messages layer-1: LINK-UPDOWN timestamp, router, interface, status 4. syslog messages layer-2: LINEPROTOL-UPDOWN timestamp, router, interface, status 5. syslog messages layer-3: BGP-ADJCHANGE timestamp, router, neighbor ip, vrf, status 6. router configurations vrf configurations router, vrf, rd 7. router configurations ebgp session/interface configurations router, interface, neighbor ip 8. PE Forwarding table daily dump router, prefix, vrf, nexthop ip, nexthop interface Table 4: Available data from the provider network sent via BGP from CE to PE. Looking at historical BGP updates won t help much because updates only show the prefixes that experienced change. Even the table dumps from the BGP collector does not help much, because the monitor has only the best paths selected by the two monitored reflectors. More importantly, table dumps or updates have only PE information (in the fields of BGP nexthop and originator), but we need the PE-CE session information and one PE can have sessions with multiple CE s via multiple interfaces. We therefore utilize router configurations and PE forwarding tables available from the provider network. The relevant information available in these two data sources is shown in Table 4. Router configurations can give us the mapping from an interface/session/vrf to RD, based on data source 6 in Table 4, but do not give us information about the prefixes because most prefixes are not statically configured. The daily PE forwarding tables, on the other hand, give us a table of (router, vrf, prefix, nexthop ip, nexthop interface), as shown in data source 8 in Table 4. We combine these data sources as follows. First, we correlate the data sources 6 and 8 to filter out those forwarding table entries whose nexthop are not the router s CE s, i.e., the best path is not learned from CE, but from ibgp. Data source 6 provides the mapping of (router, vrf) (router, RD). By plugging this mapping into data source 8, and we got a table of (router, interf ace, neighbor ip, vrf) RD : prefixes. From this table, we then project out the two needed mappings: (router, interface) RD : prefixes and (router, neighborip, vrf) RD : prefixes. Based on these two mappings, we then convert each syslog messages into m messages, where m is the number of affected RD : prefixes: syslog message 1, affected RD1 : prefix 1,... syslog message 1, affected RD1 : prefix m We are now ready to correlate the converted syslog messages with the BGP updates. 4.2 Event Clustering To correlate the syslog messages and BGP updates, we first merge the syslog messages and the BGP updates stream, and sort the combined stream based on timestamp. The clocks at the PEs and the BGP collector are NTPsynchronized, thus the timestamp indicates the relative timing of each message accurately enough for our purposes. 7 Note that the BGP path computation and propagation for different RD : pref ixes are done separately, thus we conduct the measurement for different RD : pref ix separately. Given a stream of syslogs and BGP updates for a RD : prefix, our next step is to cluster messages into events. Our clustering algorithms have two major differences from existing measurement work. First, we determine the beginning of an event using syslog as the beacon whenever possible, because the syslog messages indicate the timing of the root cause of a event. Therefore, its accuracy is very close to that of the the scheduled event beginning time in the active measurement. On the other hand, similar to the traditional passive measurement, it is more representative than the active measurement because it can be used to measure all the actual events triggered by the PE-CE link/session changes. The second difference relates to how the calculation of the time window for determining the end of an event. Existing measurement work typically derives the time window based on the distribution of the measured update interarrival time. In this work, we calculate the value of the time window based on the various timer settings and the ibgp topology in the studied network. The analytical results in Section 3.3 shows that the route exchange can take at most (5 + n) 5 = 35 seconds delay in (n = 2 in the studied network). We thus use a time window of 35 seconds. More specifically, we have the following rules for determining the starting time and ending time of an event: 1. An event starts with an update message or with one of the following syslog messages: LINK-UPDOWN, LINEPROTOL-UPDOWN, BGP-ADJCHANGE 7 Also, note that layer1 and layer 2 messages are timestamped with the time when the corresponding layer detects the failure, and the actual failure might happen some slightly earlier than the detection time. 7

8 2. If the events starts with a syslog event, there could be any combination of LINK-UPDOWN, LINEPROTOL-UPDOWN, BGP-ADJCHANGE messages in one event, but they should be in the same direction: all up or all down. Note that sometimes there are BGP-ADJCHANGE without preceding LINK-UPDOWN or LINEPROTOL-UPDOWN, because the session is torn down due to other reasons such as mal-formatted packet. Sometimes, a syslog message is lost since it is sent over UDP, although we observed few lost syslog messages in our study. 3. For an event starting with a syslog message, an event finishes when the next message is: (i) a syslog message with different (router, interface), (ii) a syslog message with the same (router, interface) but its direction is different from the one in the starting syslog message of the event, (iii) an update message but it arrives more than 35 seconds later than the last update message of the event. 4. For an event starting with an update message, the event ends when the next message is a syslog message or the next update message arrives more than 35 seconds later than the current update message. An event starting with a BGP update and ending with an update is called an update-only event, which means there is no syslog message that could be correlated with the updates observed. Update-only events can be caused by OSPF changes, session failures in ibgp, lost syslog messages, configuration at the PE, or by the BGP updates received by CEs who then propagates the updates to the PE s. An event with only syslog messages is called an syslog-only event, which means that there was a syslog message pertaining to interfaces in this VPN, but there were no corresponding BGP updates that could be correlated with the syslog message. Syslog-only events can be caused by transient layer 1 /layer 2 events which don t cause any BGP session or route changes, or a failure which affects only the backup path which is visible only at the backup PE anyway. In the rest of the paper, we consider only the events starting with a syslog message and ending with BGP update message. We call these events convergence events. 4.3 BGP Convergence Event Detection As mentioned in Section 3.1, we borrow the following four BGP convergence event types from earlier work: T down, T up, T long, and T short. Different from the designed experiment in [10], we don t know the exact event type since the same type of failure can result in different types of event. For example a link failure can result in the unreachability of a destination or can simply cause the destination to become reachable via an alternative path. Therefore, we have link event last update type of type status of the event previous event T down down withdrawal does not matter T up up announcement T down T long down announcement does not matter T short up announcement not T down Table 5: Convergence event types to decide the event type via measurement, as summarized in Table 5. Below we repeat the event definitions, and describe how we decide the event type via measurement and how we measure the convergence delay. T down : is the event where the destination becomes unreachable because of a link failure. We determine an event is T down if the syslog messages of the event indicates the link/session goes down, and the last message of the event is a withdrawal. The convergence delay for T down is measured as the time it takes from the start of the syslog message to the last withdrawal in the event. T up : is the event where the destination becomes reachable because a previously-withdrawn route is reannounced as the result of the link recovery. We determine an event is T up if the syslog messages of the event indicates the link/session goes up, and type of the previous event is T down. The convergence delay for T up is measured as the time it takes from the start of a syslog message to the last announcement in the event. T long : is the event where the primary path goes down because of the link failure, but the destination is still reachable via a less preferred path. We determine an event is T long if the syslog messages of the event indicates the link/session goes down, and the last message of the event is an announcement. The convergence delay for T long is measured as the time it takes from the start of a syslog message to the last announcement in the event. It is illustrated in Figure 3. T short : is the event where the less preferred path is replaced by a more preferred path which is re-announced as the result of the link recovery. We determine an event is T short if the syslog messages of the event indicates the link/session goes up, and type of the previous event is not T down (i.e., one of T long, T up, ort short. The convergence delay for T up is measured as the time it takes from the starting of syslog message to the last announcement in the event. Note that in the above methodology, the classifications of T down and T long are more accurate than T up and T short. This is because T down and T long can be determined solely 8

9 Figure 3: Measuring convergence delay based on the messages in an event, while T up and T short classification depends on the previous (measured) event type. In some cases, a measured event could start with a syslog down event but ends with a withdrawal update, or start with a syslog up event but ends with an announcement update. Such an unclassified convergence event, could result from two overlapping events or the loss of the syslog data. We observe few unclassified convergence events in our study. Figure 3 also illustrates how we estimate different factors contribution to T long convergence. We measure the session failure detection time as the time from the beginning of the event to the time the first BGP update is received, i.e., T3 - T0 in Figure 3. Note that this might over-estimate the detection time by up to 5 seconds, because there could be a MRAI delay of 0 to 5 seconds from the primary PE to its reflector if there is background BGP traffic from the CE to PE, which we believe do not frequently happen. To measure the path exploration time, we first found the time for the first update with an aspath equal to the primary aspath, and the originator/bgp nexthop equal to the primary PE. In the case of Figure 3, it is T3. Then we find the time for the withdrawal message (T4). Then the path exploration time is defined as T4 - T3. The invisibility s contribution is simply T5-T4, where T5 is the end of the convergence. Section 3 has shown that during T long convergence, PEs could lose the reachability to the destination for some time. Thus we also measure the route failure time during the T long convergence. We first find the last update which is either a withdrawal or has an originator equal to the primary PE (i.e., going through the failed PE-CE link), and the update U right after it. The route failure time is from when the event starts to the timestamp of the update U. 5 Measurement Results In this section we present the measurement results based on the data obtained from the provider network over a three month period in Several questions about MPLS VPN convergence delay are very interesting. How long are the delays of T long, T down, T up, and T short in MPLS VPN? Are they different from the ebgp convergence delays? These results by themselves are interesting since since there has been no prior study. Further, Section 3 has explained how different factors (failure detection, path exploration, and route invisibility) could delay the convergence, but what are these factors actual contributions to the convergence delay? Given the VPN customers higher expectation on T long events, is the reachability lost during T long convergence? If so, for how long? Are convergence delays different for different prefixes, VPNs (and their sizes), PEs, or PE interfaces? Finally, it has been observed in IPv4 BGP that a small number of prefixes contribute most of the BGP events. Is this true in MPLS VPNs? Or more generally, do different VPNs, prefixes, PEs, or PE interfaces contribute to total event count differently? The rest of this section is aimed at answering these questions. 5.1 Results for All Types of Events We observed around 300 thousands events during our study period. Among all the events, 38.3% are updateonly, and 41.7% are syslog-only, and 20% are convergence events. This observation means that only about 1/3 BGP update changes (20%/(38.3% + 20%)) are caused by syslog changes and that only about 1/3 syslog changes (20%/(41.7% + 20%)) cause convergence events. Detailed study of this phenomena would be very interesting, but is beyond the scope of this paper. Among the convergence events, 7.3% are T long, 7.4% are T short, 43.4% are T down and 39.9% are T up, and 2% are unclassified. The fact that there are more T down and T up events than T long and T short are not surprising, because only multi-homed prefixes can possibly have T long and T short events, and typically, there are much fewer multihomed prefixes than single-homed prefixes. The fact that there are more T down events than T up events can be due to the fact that there are more real T up events that are classified as syslog-only, update-only, or unclassified convergence events by our methodology as the result of loss of syslog messages or event overlapping. Again, we leave it as our future work to investigate these details. Figure 4 shows the distributions of the convergence delays for T down and T long events (two upper curves), and T up and T short events (two lower curves). Despite the potential worst-case delay of more than 250 seconds according to our analysis in Section 3, most delays (83% in T long and T down, 68% percent in T up and T short ) are actually shorter than 20 seconds, There are two differences in our results compared with the IPv4 BGP convergence delays [10, 15]. First, the con- 9

10 CDF Tlong Tdown Tup Tshort Convergence time (seconds) Figure 4: Cumulative distributions of convergence delays for T down and T long (two upper curves), T up and T short (two lower curves). vergence delays in MPLS VPNs are much shorter than those in IPv4 BGP, in which convergence delays on average are longer than 100 seconds for T down and T long and more than 30 seconds in T up and T short [10, 15]. This is due to several reasons. First of all, IPv4 BGP has a much bigger scale (thousands of ASes) than MPLS VPN networks. Thus the IPv4 BGP update propagation time along the ASes, amplified by MRAI delay, is longer than in MPLS VPN networks. Also due to smaller scale, and that the simpler route reflector-based topology limits the number of alternative paths, there are much less path exploration in MPLS VPN, as will be shown later in this section. Third, the default MRAI time value for ibgp is 5 seconds, while in ebgp it is 30 seconds, i.e., an amplifying factor of 6. Second, Figure 4 shows that the delays of T up and T short are longer than those of T down and T long. This observation is different from the results in [10, 15], but they are not conflicting. In the measurements reported in [10, 15], an event starts when the initial announcement or withdrawal are advertised, but in our methodology, an event starts when a layer 1/layer2 change happens. In T up and T short, it takes time for the PE and CE to exchange BGP protocol messages, following the Finite State Machine specification [20], to establish a BGP session. After that, the routes are exchanged between PE and CE, and depending on the size and the order of the routes in the routing table, the announcement of the prefix in question can be further delayed. Although, as we argued earlier, T up delays might not be as important and T long, it is still helpful to have a shorter T up delay in some cases such as a simple router reboot during planned maintenance. Therefore, this suggests that further research and engineering is needed to shorten the time to establish the BGP sessions and initial table exchanges after a session is setup. 5.2 Focusing on T long We now focus on T long convergence and determine the contribution of the different factors that relate to convergence delay as well as the duration of route failure during T long convergence. CDF detection + exploration + invisibility detection path exploration invisibility Convergence time (seconds) Figure 5: Cumulative distribution of delays caused by different factors in T long convergence. We break down the T long convergence delay based on the methodology shown in Figure 3. Figure 5 shows distributions of the delay contribution of the different factors contributing to T long. It shows that failure detection contributes the most, closely followed by route invisibility (both contribute around 10 seconds in 90% of events). Figure 5 shows that the path exploration contributes least in MPLS VPNs, as opposed to the Internet environment where it is the dominant factor. As we mentioned earlier, in MPLS VPNs there are much fewer alternative paths to explore due to its smaller scale and the simpler route reflector-based topology. Furthermore, as shown in the example in Section 3, only some of the message interactions cause path exploration. Figure 6 shows the distribution of the T long convergence delay and duration of route failure (prefix unreachability) in T long events. Here we emphasize the disruption caused by the T long convergence. The time for route failure (during which the prefix is unreachable) is measured based on the methodology presented at the end of Section 4. Figure 6 shows that 30% of the events have convergence time longer than 10 seconds and 30% of the events have route failure for longer than 9 seconds. Such long failure durations could cause significant disruptions to important applications such as VoIP. These results show that it is very important to improve the T long convergence delay, and that in order to improve T long convergence, shortening failure detection time and solving route invisibility have higher priorities than solving path exploration. 10

11 CDF Entity % contributing 90% events Prefixes 18.6 RDs 17 PEs 42.9 PE-CE Interfaces 6.6 Table 6: Percentage of network entities that contributes 90% of convergence events CDF Tlong Convergence Time Loss of Reachability Convergence time (seconds) Figure 6: T long route failure. Tlong Tdown Tup Tshort Convergence time (seconds) of different RDs for the median convergence delays vs the VPN size but there is no clear trend. Thus it shows that the size of a VPN does not affect the convergence delays. Actually, the factors identified in our worst-case analysis in Section 3 can all apply in the real network. That is, router timer configurations, background BGP traffic at different routers, various timer s residual value when failure happens, ibgp hops between primary PE and backup PE, can all affect the delays. Another reason is that some entities have very small number of events (as will be shown in the next subsection), thus there might not be enough data points to determine an accurate median delay for the entities. Therefore, exact reasons of varied delays with different entities, as well as why T short delays per-entity are more spreading-out than other events, remain to be discovered, and will be our future work. 5.4 Event Contribution by Network Entity Figure 7: Cumulative distributions of median delays for per- RD 5.3 Delay Distributions per Network Entity The results so far in this section haven t distinguished different entities involved in the convergence. We now try to see whether convergence delays varies with entities, i.e., different prefixes, VPNs, PEs (that triggered the events), or PE interfaces (that triggered the events). We measure the median convergence delay for each RD 8, and show its cumulative distribution in Figure 7. It shows that the convergence delays indeed vary a lot with different RDs. For example, 80% of RDs have median T long delays of 5.83 seconds or shorter, but 3% of the RDs have median T Long delays of 40 second or longer. Interestingly, the T short curves look very different from the other three in that it is the median delays are more spead-out. We generated similar plots for per-prefix, per-pe, and per-interface (not shown), and observe similar skewed-ness. Exactly what makes these differences? E.g., does a VPN s size (defined by the number of VPNv4 prefixes, PEs, or PE-CE interfaces in the VPN) matter? We generated scatter plots (not shown) 8 In the VPN networks we studied, there is basically a one-to-one mapping from an VPN network to an RD. We thus use these two terms interchangeably when there is no confusion. In IPv4 BGP, it has been observed that the majority of the events are caused by a small number of very unstable prefixes [21]. In this section, we investigate whether similar observations hold for VPNv4 prefixes, as well as for other MPLS VPN specific network entities namely, VPNs, PEs, and PE-CE interfaces. We first ranked (from high to low) the entities based on the number of events per entity. Then we determined the cumulative ranked contribution of each of the four network entities to the total convergence event count. The 90th percentile for each of these distributions are shown in Table 6. As shown in the table, only 18.6% of prefixes contributed to 90% of the convergence events. The number is approximately the same (17%) for RDs, while the PE-CE interface granularity has an even more skewed distribution with only 6.6% of interfaces contributing to 90% of the events. Considering the router level granularity, we see a somewhat more balanced distribution where 42.9% of PEs contribute to 90% of events. We also calculated the cumulative distribution of the event counts for each of the network entities and summarize the results in Table 7. From the table we see that 63.2% of prefixes were involved with at least one convergence event. The corresponding numbers for RDs, PEs and PE-CE Interfaces are 60.9%, 82% and 23.6% respectively. Considering the opposite end of the distribution, we found that the top 11