Joint Optimization of Intra- and Inter-Autonomous System Traffic Engineering

Transcription

1 Joint Optimization of Intra- and Inter-Autonomous System Traffic Engineering Kin-Hon Ho, Michae Howarth, Ning Wang, George Pavou and Styianos Georgouas Centre for Communication Systems Research, University of Surrey, Guidford, Surrey GU2 7XH, UK Emai: {K.Ho, M.Howarth, N.Wang, G.Pavou, Abstract Traffic Engineering (TE) is used to optimize IP operationa network performance. The existing iterature generay considers intra- and inter-as (Autonomous System) TE independenty. However, the overa network performance may not be truy optimized when these aspects are considered separatey. This is due to the interaction between intra- and inter-as TE, where a soution of intra-as TE may not be a good input to inter-as TE and vice versa. To remedy this situation, we propose considering intra-as aspects during inter-as TE and vice versa. We propose a oint optimization of intra- and inter-as TE to further improve the overa network performance by simutaneousy finding the best egress points for the inter-as traffic and the best routing scheme for the intra-as traffic. Three strategies are presented to attack the probem, namey sequentia, nested and integrated optimization. Our simuation study shows that, compared to sequentia and nested optimization, integrated optimization can significanty improve the overa network performance by accommodating %-6% more traffic demands. I. INTRODUCTION Traffic Engineering (TE) [1] is the set of techniques that optimize IP operationa network performance by tacticay routing traffic on a path other than the one woud have been chosen if standard routing methods had been used. The task of TE is: given a network topoogy and a coected set of traffic demands (i.e. traffic matrix), determine the best routing for the traffic so that the overa network performance is optimized. Today s Internet is a coection of more than 18, Autonomous Systems (ASes), each being an administrative region governed by its own network poicies and routing protocos. Open Shortest Path First (OSPF) and Intermediate System-Intermediate System (ISIS) are the common intra-as routing protocos, whie Border Gateway Protoco (BGP) is the de facto inter-as routing protoco. With the hierarchica characteristics of the Internet, traffic is routed within an AS or between ASes. Thus, TE can be broady divided into two types: intra-as and inter-as. In intra-as TE, the operator of an AS contros traffic routing within the network by either optimizing the ink weights of the corresponding routing protoco (mosty OSPF or IS-IS) or estabishing Labe Switched Paths (LSPs) through MutiProtoco Labe Switching (MPLS). Typica intra-as TE optimization obectives are to minimize network bandwidth consumption and to achieve oad baancing within the network. Inter-AS TE [14,15], on the other hand, aims to contro traffic entering and exiting an AS using optimization obectives such as oad baancing over inter-as inks. It is commony assumed that these inter-as inks are frequenty congestion points [6,1]. For a particuar AS, the network operator can contro traffic exiting the AS by assigning the traffic to the best egress points. This is caed Outbound inter-as TE. Likewise, the network operator can aso contro traffic entering the AS by seecting the best ingress points, which is caed Inbound inter-as TE. In current practice, the commony used method to enforce inter-as TE is by adusting BGP route attributes. Internet Service Provider (ISP) networks typicay carry both intra-as traffic that is routed ony within their networks and inter-as traffic that is routed not ony within their networks but aso across other ASes. ISPs can empoy both intra- and inter-as TE to optimize the routing of these types of traffic. However, athough some work exists on intra- and inter-as TE, much of the existing iterature deas with them separatey: prior intra-as TE work has assumed that ingress and egress points of inter-as traffic do not change whereas prior inter-as TE work has not considered route optimization within an AS. This is often inappropriate and resuts in suboptima overa network performance due to the foowing two interaction effects between intra- and inter-as TE: Inter-on-Intra-AS TE: The first interaction is the effect of inter-as TE on the performance of intra-as TE. Inter-AS TE can change the ingress and egress points of inter- AS traffic, thus causing the traffic to be routed on different ingress-to-egress paths within the network. This fundamentay changes the intra-as traffic matrix, i.e. traffic oad between each ingress and egress node pair. Such a change coud therefore significanty affect the intra-as TE soution. Intra-on-Inter-AS TE: The second interaction is the effect of intra-as TE on the performance of inter-as TE. It arises when outbound inter-as TE and OSPF/ISIS-based intra-as TE are used. One current practice is known as hot potato routing [2,]. In this approach, a BGP router wi choose the cosest exit as measured by the owest IGP (Interior Gateway Protoco) cost among mutipe equay-good egress points towards a downstream routing prefix. If the IGP costs are changed by intra-as TE, some inter-as traffic fows wi in genera be shifted to a new set of cosest egress points. This coud ead to congestion on these new egress points. If intra- and inter-as TE are not ointy optimized, a sequentia approach may therefore be regarded as the current practice in the coective use of intra- and inter-as TE. In this sequentia approach, the soution of inter-as TE becomes the input for intra-as TE or vice versa. However, since the obectives and constraints of the subsequent stage are not taken into account, the decisions made at one stage often do not provide a good input for the subsequent one, sometimes even eading to infeasibe inputs. As a resut, it is difficut to

2 caim that a truy good overa TE soution has been found when each TE is considered separatey. We therefore propose to consider inter-as TE during intra-as TE and vice versa. In this paper, we propose a oint optimization of intra- and inter- AS TE as an effective means to achieve better overa TE soutions than the one obtained by the sequentia approach. Specificay, we investigate the foowing two chaenges: How shoud intra- and inter-as TE be combined and how do we formuate their oint optimization? How can we sove the oint TE optimization probem to achieve a better overa network performance? Our contributions in this paper are as foows. First of a, we expain with exampes the two interaction effects that can ead to suboptima overa network performance. Then, for the first chaenge above, we formuate a bi-criteria oint optimization probem of intra- and inter-as TE with an aim of optimizing their obectives simutaneousy. Since the two interaction effects can generay appy to any intra- and inter-as TE approach, it is possibe to formuate the oint optimization probem for a the combinations of intra- and inter-as TE approaches. However, as the primary obective of this paper is to iustrate the point of interest on oint optimization of intraand inter-as TE in genera, we ony consider MPLS-based intra-as TE and outbound inter-as TE as an exampe in the oint optimization probem formuation. This probem formuation shoud be vaid and practica as both intra-as MPLS TE and outbound inter-as TE are nowadays widey researched in academia and empoyed in industry. Moreover, a potentia use of this probem formuation coud be to optimize BGP/MPLS VPN provisioning, a subect which is currenty attracting a great dea of industria attention. For the second chaenge above, we consider three strategies to sove the oint TE optimization probem, namey sequentia, nested and integrated optimization. These strategies aim to obtain non-dominated soutions with respect to the intra- and inter-as TE obectives. We evauate the performance of these strategies by simuation using Rocketfue [4] topoogies and synthetic traffic matrices. Our simuation resuts revea that better overa network performance can be achieved by integrated optimization, soving intra- and inter-as TE simutaneousy. The performance improvement coud aow the network to support a %-6% increase in the traffic demands. We beieve that our work provides an insight into the interactions between intra- and inter-as TE, enabing ISPs to further optimize the performances of their networks over the current practice sequentia approaches. The rest of this paper is structured as foows. In the next section, we provide background on intra- and inter-as TE. Section III expains the two interaction effects with exampes. We formuate the oint intra- and inter-as TE optimization probem in Section IV and present strategies to sove it in Section V. In Sections VI and VII, we present our evauation methodoogy and simuation resuts for these strategies. Sections VIII and IX present reated work and concusion. II. TRAFFIC ENGINEERING Traffic engineering typicay takes input eements such as traffic matrix, network topoogy, and then executes agorithms to produce a set of traffic routing pans that optimize the overa network performance. In this section, we provide background on these eements that are of importance to understand our work. For ease of presentation, Tabe 1 shows the notation used throughout this paper. Tabe 1. Notation used in this paper NOTATION DESCRIPTION K A set of downstream routing prefixes I A set of ingress points J A set of egress points (inter-as inks) E intra A set of intra-as inks t_inter(i,k) Bandwidth demand of the inter-as traffic fow at ingress point i I destined to routing prefix k K t_oc(i,) Bandwidth demand of the oca intra-as traffic fow between ingress point i I and egress point J t_intra(i,) Bandwidth demand of the intra-as traffic fow between ingress point i I and egress point J T_Inter(I,K) Inter-AS (ingress-to-prefix) traffic matrix consisting of a t_inter(i,k) T_Loc(I,J) Loca intra-as (ingress-to-egress) traffic matrix consisting of a t_oc(i,) T_Intra(I,J) Intra-AS traffic matrix consists of a t_intra(i,) Out(k) A set of egress points that has reachabiity to routing prefix k C Capacity of egress point (inter-as ink) inter bw Residua bandwidth of inter C inter C Capacity of intra-as ink intra bw Residua bandwidth of intra C intra A binary variabe indicating whether inter-as traffic fow x ik, t_inter(i,k) is assigned to egress point A binary variabe indicating whether intra-as traffic fow y i, t_intra(i,) is assigned to intra-as ink P i, A set of candidate paths reaizing intra-as traffic fow t_intra(i,) w ip,, A binary variabe indicating whether path p P i, is chosen to reaize the traffic fow t_intra(i,) s(i,k) A variabe storing the egress point that has been assigned to t_inter(i,k) A. Internet Traffic Types According to [5], Internet traffic received by an AS can be cassified into four types: interna traffic that traves from an ingress access ink to an egress access inks; transit traffic that traves from an ingress peering ink to an egress peering ink; inbound traffic that traves from an ingress peering ink to an egress access ink, and outbound traffic that traves from an ingress access ink to an egress peering ink. As mentioned in the introduction, this paper ony considers outbound inter-as TE which seects optima egress peering points for inter-as traffic (i.e transit and outbound traffic). We therefore do not manipuate the potentia ingress peering point seection in a simiar manner. Given the fact that ISPs can generay ony suggest which ingress peering points to use and the fina decisions are sti made by their customers [6], we assume in this paper that the ingress point of any traffic is fixed and known in advance. Thus, we define the foowing types of Internet traffic and refer to them throughout this paper: oca intra-as traffic: traffic that is destined to egress access inks. This corresponds to interna and inbound traffic. inter-as traffic: traffic that is destined at downstream ASes and whose egress peering points can be varied by inter-as TE. This corresponds to outbound and transit traffic.

3 intra-as traffic: a traffic that traverses the network, incuding both oca intra-as and inter-as traffic. B. Traffic Matrices t_intra(i,) = t_oc(i,) + t_inter(i,k) i t_oc(i,) t_inter(i,k) t_inter(i,k ) Figure 1. Exampe of intra-as traffic matrix A Traffic Matrix (TM) represents a matrix of traffic oad from one network point to another one over a particuar time interva. In the inter-as traffic matrix, each eement t_inter(i,k) represents the voume of inter-as traffic that enters the network at ingress point i and is destined to routing prefix k. The egress point of each inter-as traffic fow can be seected by inter-as TE. We denote s(i,k) as the egress point assigned to t_inter(i,k). On the other hand, in the intra-as traffic matrix, each eement t_intra(i,) represents the voume of traffic that enters the network at ingress point i and exits at egress point. It is the sum of the oca intra-as and inter-as traffic voume between each ingress and egress node pair: t_ intra( i, ) = t_ oc( i, ) + t_ inter( i, k) k K: s( i, k) = Figure 1 shows a network with ingress point i and egress points and. We assume both egress points can reach routing prefixes k and k. Given oca intra-as traffic fow t_oc(i,) and inter-as traffic fows t_inter(i,k) and t_inter(i,k ) with s(i,k) = and s(i,k ) =, the eements of the intra-as traffic matrix are t_intra(i,) = t_oc(i,) + t_inter(i,k) and t_intra(i, ) = t_inter(i,k ). The intra- and inter-as TM can be obtained through measurement or estimation. The intra-as TM can be measured [5] through Cisco s NetFow. It can aso be estimated from measured ink oad statistics [7,8]. In contrast to the intra-as TM, ess attention has been paid to deriving the inter-as TM. One method of deriving this is through measurement at each vantage point. The authors in [9] describe a methodoogy to compute inter-as TM using Cisco s NetFow and BGP routing data. In addition to measurement, the inter-as TM may be estimated. The authors in [1] propose a methodoogy to estimate inter-as pubisher and web traffic matrices using server ogs from content deivery networks and packet eve traces from arge user sets. These estimated traffic matrices are routing prefix basis, which can be used as the inter-as TM in this paper. C. Intra- and Inter-AS Traffic Engineering A genera intra-as TE probem can be summarized as foows: given a network topoogy and an intra-as traffic matrix, determine an appropriate set of OSPF ink weights or MPLS LSPs so as to optimize the network performance such as bandwidth consumption and oad baancing within the network. Many techniques exist to sove this probem. For exampe, Fortz and Thorup [11] proposed a tabu search technique to derive optima ink weight settings. Xiao et a [12] proposed a MPLS-based TE heuristic. More intra-as TE work can be found in [26,28] and their cited references. On the other hand, the genera inter-as TE probem [6] can be summarized as foows: given a network topoogy, an inter- AS traffic matrix and BGP routing prefixes, seect ingress/egress points for the traffic so that the network performance is optimized. A common inter-as TE obective is to baance the oad over inter-as inks. A number of outbound inter-as TE agorithms have been proposed recenty [6,16,17]. In addition, a number of inbound inter-as TE agorithms [18,19] have been proposed very recenty based on AS-path prepending [21]. This is a commony used technique that purposey makes a route ess attractive to its upstream ASes by adding severa instances of its own AS-Number to the ASpath attribute so as to increase the AS-path ength of that route. In current practice, inter-as TE is enforced by adusting BGP route attributes such as oca-preference, AS path ength, etc. The reader is referred to [21] for the expanation of these techniques. In addition, ongoing work in inter-as MPLS [22] provides an aternative method to enforce inter-as TE. III. INTERACTIONS BETWEEN INTRA- AND INTER-AS TE The existing iterature considers intra- and inter-as TE independenty. However, the overa network performance may not be truy optimized when they are considered separatey. In this section, we expain with exampes two interaction effects between intra- and inter-as TE which woud ead to suboptima overa network performance. Athough, as mentioned in the introduction, this paper focuses on MPLS-based intra-as TE and outbound inter-as TE, we present the two interaction effects in a genera way that is appicabe to other intra- and inter-as TE approaches. A. Effect of Inter-AS TE on Intra-AS TE The first interaction is the effect of inter-as TE on the performance of intra-as TE. We take outbound inter-as TE as an exampe in our discussion athough the effect is aso appicabe to inbound inter-as TE as both can infuence intra- AS TE performance by changing the intra-as traffic matrix. Reca from Section II.B that the intra-as TM is derived from both oca intra-as and inter-as traffic. Inter-AS TE assigns egress points to inter-as traffic in order to baance the oad over inter-as inks, for exampe. Consequenty, the traffic wi be routed on different ingress-to-egress paths according to the assigned egress points. This resuts in changing the intra-as TM, with varying traffic oad for each ingress-egress node pair. For exampe, referring to the scenario in Section II.B, if s(i,k) was changed to, the two eements of intra-as traffic matrix woud then become t_intra(i,) = t_oc(i,) and t_intra(i, ) = t_inter(i,k) + t_inter(i,k ). However, a change in intra-as TM has a consequence. There are two cases where if: inter-as TE is performed before intra-as TE: by performing inter-as TE, different egress point seections for the inter-as traffic wi resut in different intra-as TMs because the traffic oad between ingress and egress node pairs is varied. However, different intra-as TMs cause intra-as TE to produce soutions with different performances, even though identica network topoogy

4 and TE agorithm are used. This argey depends on the topoogica connectivity and characteristics, e.g. ink capacity. In this case, a suboptima network performance resuts if the intra-as TM does not ead the intra-as TE to produce a gobay optima performance. inter-as TE is performed after intra-as TE: when inter- AS TE is run, inter-as traffic is shifted to use different ingress-to-egress paths, causing increased intra-as ink utiization and possiby congestion. This worsens the performance initiay achieved by the intra-as TE k i 2 t _oc(i,) = 1 t _oc(i, ) = 5 t _inter(i,k) = 4 We iustrate the first interaction effect through the foowing exampe of Figure 2(a). We denote inter-as costs 1 at egress point and by Cost() and Cost( ), and intra-as costs 1 of route r(i,) (the best route that coud be found by intra-as TE between ingress point i and egress point ) and r(i, ) by Cost(i,) and Cost(i, ) respectivey. These routes are assumed to be the utimate intra-as TE soution, regardess of whether the inter-as TE performs before or after the intra-as TE; this corresponds to the abovementioned two consequences. Assume there are two oca intra-as traffic fows t_oc(i,) and t_oc(i, ), and one inter-as traffic fow t_inter(i,k). Their traffic oads are 1, 5 and 4 (unitess) respectivey as shown in the figure. The number besides each ink is the ink capacity. We define the cost 1 of each ink to be the ink s tota traffic divided by its capacity. For inter-as TE, there are two possibe egress points where t_inter(i,k) can be assigned. If it is assigned to, Cost() becomes 4 / 1 =.4 and the intra-as TM becomes: t_intra(i,) = t_oc(i,) + t_inter(i,k) = 14 and t_intra(i, ) = t_oc(i, ) = 5. In this case, Cost(i,) and Cost(i, ) are (.9 x 2 hops) and.25 respectivey. Hence, the overa TE cost is = On the other hand, if the traffic fow is assigned to, Cost( ) becomes.44 (4 / 9) and the intra-as TM becomes: t_intra(i,) = t_oc(i,) = 1 and t_intra(i, ) = t_oc(i, ) + t_inter(i,k) = 9. In this case, Cost(i,) and Cost(i, ) are 1. and.45 respectivey. Thus, the overa TE cost is = For inter-as TE, egress point is chosen primariy because of the owest resuting inter-as cost. However, as can be seen, the owest overa TE cost soution is to assign the traffic fow to egress point as the intra-as cost can now be significanty reduced with ony a sighty increase in the inter-as cost. Even if the resuting inter-as cost of both assignments are the same, the first one may sti be a resut since intra-as route optimization is not considered during inter-as TE. In fact, 1 ^ >9 1 9->11 i t _inter(i,k) = 4 Figure 2(a) and (b). Interaction effects between intra- and inter-as TE there are many issues infuencing the performance of intra-as TE. An important issue which has been shown in the exampe is the amount of traffic routed on each ingress-to-egress path. Varying the traffic oad between ingress and egress node pairs caused by inter-as TE resuts in different traffic demands in the network. If a arge traffic demand can ony be routed on a ow capacity ink or a onger path, a high intra-as TE cost usuay resuts, for exampe t_intra(i,) and Cost(i,) in the scenario where t_inter(i,k) is assigned to egress point. Regardess of whether the routes have been optimized or are to be optimized by intra-as TE, shifting the traffic egress points by inter-as TE yieds different intra-as TE soutions. We therefore propose to operate on oca intra-as TM and inter-as TM coectivey in order to find a good intra-as TM so as to improve the overa network performance, rather than merey assuming that an intra-as TM is given. B. Effect of Intra-AS TE on Inter-AS TE The second interaction effect between intra- and inter-as TE is caused by hot-potato routing. We expain this effect using the exampe in [2] iustrated by Figure 2(b). The definition of each eement in the figure is the same as those in Figure 2(a), except that the numbers on the three arrows towards the three egress points are the IGP costs of r(i,), r(i,^) and r(i, ). Assume that ingress point i has three equay good routes to routing prefix k through egress points, ^ and. Under hot-potato routing, i directs traffic to the cosest egress point - the router with the smaest IGP cost (i.e., router ). As a resut, Cost(), Cost(^) and Cost( ) are equa to (.,.,.4) respectivey. However, suppose the IGP cost to changes from 9 to 11, in response to a change in ink weights for a TE purpose. Athough the route through is sti avaiabe, the IGP cost change woud cause i to seect the route through egress point since this has the owest IGP cost. In this case, the inter-as costs are changed to (.66,.,.). Consequenty, athough the intra-as TE cost coud be improved, the change of IGP costs affects the utiization on some inter-as inks and possiby causes congestion. In this case, intra-as TE can not ony affect the performance of intra- AS but aso the inter-as performance. Note that this effect woud not have an infuence on MPLS-based intra-as TE since the traffic forwarding decision does not depend on IGP costs but LSPs and abes. Nevertheess, irrespective of which intra-as TE approach is used, the performance of intra-as TE may not be optimized without reassigning some inter-as traffic fows to other ingress/egress points that coud eventuay ead the traffic to be routed on better-performed paths in the network. In order to reduce the high inter-as cost in the above exampe, the status of both intra- and inter-as ink utiization need to be simutaneousy considered. For exampe, one coud change the IGP cost of r(i, ^) from 12 to 9 if this change does not affect the overa intra-as cost whie sti meeting the TE obectives. As a resut, the traffic fow wi be assigned to egress point ^ and the inter-as costs now become (.,.4,.). This produces the same inter-as cost as the origina case where IGP costs have not changed. 1 The cost here is simpy a performance metric that approximates the ink utiization. It is not the IGP cost or weight that is used by OSPF routing.

5 C. The Need for Joint Optimization The two interaction effects have shown that inter-as TE indeed can affect the performance of intra-as TE and vice versa. A recent study has aso shown that the Internet botteneck is approximatey equay distributed on intra- and inter-as inks [1]. Therefore, a good TE soution shoud perform satisfactoriy with respect to both intra- and inter-as TE obectives. It is important that inter-as TE aspects shoud be considered during intra-as TE and vice versa. For exampe, if we ook for the best overa TE resut, we can see that egress point and the suggested soution of the preceding two exampes woud be chosen if intra- and inter-as TE can be ointy optimized. In the next section, we therefore propose a bi-criteria integer programming formuation for the Joint intraand inter-as TE optimization (Joint-TE) probem. IV. JOINT INTRA- AND INTER-AS TE OPTIMIZATION A bi-criteria optimization formuation is that one can express two notions that are of concern in defining what represents an optima soution. Their obectives are typicay expressed in a form of cost functions. In this paper, we formuate a bi-criteria Joint-TE probem by taking into account both intra- and inter-as TE cost functions. A. Cost functions We empoy the commony used cost function proposed by Fortz and Thorup [11]. This is a piecewise inear function of ink utiization, which imitates the response time of M/M/1 queues to represent the cost of network inks. By using the piecewise inear cost function, two obectives of minimizing bandwidth consumption and achieving oad baancing are taken into account simutaneousy. In this paper, we use the piecewise inear cost function for both intra- and inter-as TE for consistency and generaity, as we as for its abiity to express the common key obectives of each TE. Nevertheess, these cost functions may be different by domains according to their operationa obectives. B. Bi-Criteria Probem Formuation The obective of the Joint-TE probem is to minimize both overa intra- and inter-as costs: Minimize ϒ + Ψ (1) Eintra J Subect to u intra = y i, t_ intra(, i ) C intra, Eintra (2) i I J uinter = xik, t_ inter(, i k) C inter, J () i I k K w ip,, = 1, ( i I, J) (4) p Pi, f u ϒ = ( ) (5) intra Ψ = f ( u ) (6) inter f () θ = θ, θ 1 (7) 2 f () θ = θ, 1 θ 2 (8) 16 f () θ = 1 θ, 2 θ 91 (9) 178 f () θ = 7 θ, 91 θ 1 (1) 1468 f ( θ) = 5 θ, 1 θ 11 1 (11) 1618 f ( θ) = 5 θ, 11 1 θ (12) Equations (2) and () define the utiization of intra- and inter-as inks. Constraint (4) ensures that each intra-as traffic fow t_intra(i,) is routed aong a singe LSP within the network. This is an optiona constraint but we consider preserving scaabiity and minimizing compexity on network management by avoiding excessive LSPs to be managed and arbitrary traffic spitting, though such spitting coud permit better network performance [2]. The remaining constraints define the cost of each intra- and inter-as ink as a function of its utiization based on the piecewise inear cost function [11]. Given the ossess property of the inks, a genera constraint is a fow conservation constraint which ensures that the tota traffic demand incoming to an intermediate node is equa to the tota traffic for this demand outgoing from the node. Fundamentay, the Joint-TE probem is the combination of intra- and inter-as TE probems. The probem formuation of intra-as TE consists of the reduced obective function Minimize Eintra ϒ as we as constraints (2), (4), (5) and (7)- (12). On the other hand, the probem formuation of inter-as TE consists of the reduced obective function Minimize Ψ as we as constraints () and (6)-(12). J C. Optimization Criteria An optima soution of the bi-criteria optimization probem is that each of the two TE obectives simutaneousy attains an optima vaue. However, in genera, either it is not possibe to find such optima soutions or they do not exist. In other words, the two obective functions are conficting. For exampe, the cost of a particuar egress point may be ow but the cost of the intra-as path towards the egress point is high. Moreover, it is not possibe to compare the two obective vaues mathematicay and sensiby. For exampe, we cannot distinguish mathematicay which is a better TE soution, (1,) or (,1), where (x,y) represents intra- and inter-as costs respectivey. However, we can observe that (1,) is better than (2,) or (2,4). On the other hand, the vaue of cost function varies with the number of inks and their utiizations. It does not make sense to compare the two obective vaues when the number of intra- and inter-as inks and their capacities are different, as is typicay the case. Consequenty, this eads us to finding non-dominated soutions, which is a primary goa when soving a muticriteria optimization probem. A soution is caed nondominated if there is no any other soution that is stricty better in one of the obective functions, and has the same or better vaues in the others [24]. Thus, soution (1,) in the above exampe is a non-dominated soution of (2,) and (2,4). There are mutipe ways to identify non-dominated soutions. A commony used method is to design a metric or cost function that combines both intra- and inter-as TE obectives. However, it is often uncear how to determine the reative weights between the two obectives. A more intuitive

6 approach, which we consider in this paper, is to search nondominated soutions in such a way that the inter-as cost remains at east near-optima whie substantiay improving the intra-as cost. In other words, inter-as TE is assumed to be more important than intra-as TE. The rationae for this assumption is the foowing: intra-as overprovisioning has been empoyed by ISPs as an effective means to provide high-quaity service to a traffic on their IP backbone networks [2]. inter-as inks are common points of congestion in the Internet [6,1]. For exampe, peer-to-peer traffic consumes the maor part of inter-as ink bandwidth [25]. Moreover, an inter-as ink is reativey more difficut to extend than an intra-as ink due to timeconsuming and compicated negotiation between two ASes. Therefore, the ASes need to contro traffic especiay on inter-as inks. The method of predefining a exicographic importance order is commony used in soving muti-criteria optimization probems. It aows us to generate Joint-TE soutions that can be mathematicay distinguished and sensiby compared. D. TE Agorithm Seection A Back-Box Approach Intra- and inter-as TE agorithms wi be used to sove the Joint-TE probem. In this paper, we deiberatey treat both intra- and inter-as TE as back-boxes that we combine in a pug-and-pay manner. Both sides use TE agorithms that are based on previousy estabished techniques and can achieve near-optima soutions. The intention of using a TE agorithm that produces near-optima soutions is to minimize the possibiity that any performance improvement is soey caused by a arge performance gap between the optima soution and the soution achieved by the agorithm. We use the optima aware heuristic proposed by Sridharan et a in [26] as the intra-as TE agorithm. The agorithm soves a MPLS-based intra-as TE probem with the piece-wise inear cost function, which is the intra-as TE probem considered in this paper. The basic idea of the agorithm is that Linear Programming (LP) formuation of the intra-as TE probem is soved to obtain an optima routing soution. This optima soution permits arbitrary traffic spitting but it is not the soution required by the intra-as TE that does not aow such spitting. Therefore, a greedy heuristic based on traffic demand sorting is then performed to transform the optima routing soution to the traffic non-spittabe soution whie attempting to maintain its optimaity. We ca this intra-as TE agorithm intra-optima-aware-ag throughout this paper. The reader is referred to [26] for more detais of the agorithm. The idea of the optima aware heuristic agorithm has aso been used by prior TE work [27,28]. Their simuation resuts showed that the agorithm can achieve near-optima soutions. Hence, we aso propose using it to sove the inter-as TE probem. We ca it inter-optima-aware-ag throughout this paper. The agorithm works as foows: 1. A LP of the inter-as TE probem is soved. The outcomes are an optima inter-as cost and the desired utiization of each inter-as ink that achieves this optima cost. This soution permits arbitrary spitting of inter-as traffic over mutipe egress points. 2. Set the desired inter-as ink utiization as capacity constraints. This constraint ensures that the tota traffic on each inter-as ink does not exceed the desired utiization.. Sort inter-as traffic fows in descending order according to traffic demand. Assign each traffic fow in that order to the egress point (i.e. inter-as ink) that has the owest utiization whie not vioating its capacity constraint If there exists unassigned inter-as traffic fows, remove the capacity constraint on a the inter-as inks and re-run step unti a the traffic fows have been assigned. The LP formuations (i.e. the first step) of intra-optimaaware-ag and inter-optima-aware-ag were modeed in AMPL and soved with CPLEX optimization engine. To enabe us to show that these two agorithms produce nearoptima soutions, we incude their optima soutions (i.e. by ony soving the LP of the corresponding TE probem) in our experiments for comparison. V. STRATEGIES FOR JOINT-TE Based on the probem formuation, assumptions and agorithms described in the preceding section, we present three strategies to sove the Joint-TE probem, namey sequentia, nested and integrated optimization. A. Sequentia Optimization Sequentia optimization soves the Joint-TE probem sequentiay where the optima soution of one TE probem becomes the input for the other one. It can be regarded as the current practice in the coective use of intra- and inter-as TE. As with the two interaction effects in Section III, sequentia optimization can be divided into two approaches with different execution sequences in the TE probems: 1) SeqOpt-Inter-Intra: egress points are first assigned to inter- AS traffic using inter-optima-aware-ag so as to optimize the inter-as cost. Then the intra-as traffic matrix is computed by taking into account the oca intra-as TM and the inter-as TM with the assigned egress points. Finay, intra-as TE is performed using intra-optima-aware-ag to optimize the intra-as cost. This strategy is ogica in the sense that intra- AS TE is not performed unti both ingress and egress points of the inter-as traffic have been determined. 2) SeqOpt-Intra-Inter: initiay, intra-as TE is first performed using intra-optima-aware-ag to optimize the intra-as cost based on the intra-as TM that is computed by combining the oca intra-as TM with the inter-as TM that has the egress points randomy assigned. This random egress point assignment mimics the situation where inter-as TE has not been performed, thus the inter-as cost is not optimized. At this stage, the outcome of intra-as TE is a set of ingress-toegress LSPs. The next step is to perform inter-as TE using inter-optima-aware-ag to optimize the inter-as cost. Since inter-as TE does not consider route optimization within the network, each inter-as traffic fow is eventuay routed on the LSP between the ingress and the assigned egress point, which has been determined by the intra-as TE. SeqOpt-Intra-Inter 2 If there are severa such egress points, the seection tie-break is in order of maximum residua capacity and visited sequence. Information about AMPL and CPLEX can be found at and respectivey.

7 takes the TE approach that inter-as TE is performed on top of an intra-as traffic engineered network. The advantage of sequentia optimization is that different anaysis techniques can be appied to each of the TE probems. However, neither of the above sequentia optimization approaches consider intra-as route optimization during inter- AS TE nor inter-as egress point seection during intra-as TE. B. Nested Optimization Sequentia optimization generates ony one soution from each TE agorithm, and this is used as input to the other agorithm. However, the initia soution may not be a good input to the subsequent TE probem. In fact, for many optimization probems, there exists more than one optima soution. Hence, there may exist inter-as TE soutions that are neary optima and are aso good with respect to intra-as TE. Thus, we seek for those optima soutions from a TE probem and then input them to the subsequent TE probem one at a time unti the soution with the best overa cost is found. We propose a nested optimization to impement the abovementioned idea. It can be regarded as an enhanced and iterative version of sequentia optimization. The agorithm proposed for the nested optimization in a simiar sequence to SeqOpt-Inter-Intra: 1. A Genetic Agorithm (GA) is used to identify a set of owest-cost inter-as TE soutions. When the GA converges, a chromosomes are the soutions that have neary identica owest cost but they may have different resuts. In fact, in order to expore a arger soution searching space for intra-as TE, these soutions shoud not be restricted to an identica cost. In this paper, the soution with inter-as cost not exceeding.1% of the visited owest cost is considered. 2. For each of this set of inter-as TE soutions, we perform intra-as TM computation and then intra-as TE using intra-optima-aware-ag. During intra-as TE optimization, the best and the worst visited soutions are recorded. We ca them and. The and soutions refect respectivey the extent to which the sequentia optimization soution can be further optimized and how worse coud be the soution. We modified our GA [17] previousy proposed to sove the inter-as TE probem. The GA has incuded a heuristic simiar to step of inter-optima-aware-ag to enhance the quaity of the soution. In fact, the number of candidate inter-as TE soutions may be quite arge and we did identify a arge number of such soutions as aternative inputs for the intra-as TE. We observed that many of those inter-as TE soutions have significanty different egress point seection resuts, which produced intra-as TE soutions with very different performances. The nested optimization maintains a simpy interdiscipinary partitioning used in sequentia optimization whie attempting to obtain better overa TE soutions. C. Optimization optimization aims to sove the Joint-TE probem by simutaneousy optimizing the intra- and inter-as TE obectives. We propose an integrated approach that requires as starting soutions an inter-as and an intra-as routing configurations with known egress points and ingress-to-egress paths. The starting soutions can be any quaity, regardess of whether they are optimized by TE or not. The integrated approach then proceeds to enhance the quaity of the starting soution using neighborhood search agorithm. The integrated approach guarantees that the produced soutions are no worse than the input soutions and in practice are much better. Agorithm NSA 1. Obtain the starting soutions 2. iter. Whie iter < MAX_ITER /* stopping criteria */ 4. iter iter If no significant cost improvement for a certain number of iterations then 6. perform intra-as TE on the current soution /* diversification */ 7. For each inter-as traffic fow t_inter(i,k) 8. f(i,k) s(i,k) 9. s(i,k) Φ αψ sik (, )( bwinter) + ϒ( bwintra ) p Pi, : wip,, = 1 Φ ' αψ ( s(i,k) + t_ interik (, )) + ϒ ( + t_ interik (, )) sik (, ) bw inter p Pi, : wip,, = 1 1. Φ Φ ' /* deta is the saved cost */ 11. For each Out(k) which does not constitute a move in the memory ist 12. Ω αψ ( ) + ϒ ( ) bw inter Figure. Neighborhood search agorithm 1) Neighborhood Search Agorithm a) Overview Neighborhood Search Agorithm (NSA) is widey regarded as an important too to sove hard combinatoria optimization probems efficienty. The primary reasons for the widespread use of neighborhood search techniques in practice are their intuitive appea, fexibiity and ease of impementation, and their exceent empirica resuts [29]. The basic steps of NSA are as foows. Consider a current starting soution x. NSA expores the soution space by identifying the neighborhood of x, N(x). The neighbors of x are soutions that can be obtained by appying a singe oca transformation (aso caed a move) on x. The best soution in the neighborhood is seected as the new current soution. This neighborhood searching iterates unti the stopping criteria is satisfied. Finay the agorithm returns the best visited soution. p P i, : wip,, = 1 1. virtuay add t_inter(i,k) to t_intra(i,) 14. virtuay reease the resource used by t_inter(i,k) on p Pi, : wip,, = re-compute intra-as path z between ingress point i and egress point 16. Ω ' αψ ( bw t _ inter( i, k)) + ϒ ( bw ) inter intra z 17. If Ω Ωthen ' /* if saved cost >= added cost */ 18. Ω Ω ' /* ines 17-2 aim to find the minimum added 19. f(i,k) cost, i.e. to find the maximum profit cost */ 2. se_path z 21. If f(i,k) s(i,k) 22. break the most outer For oop /* Impementation of FM */ 2. If f(i,k) s(i,k) 24. update resource utiization on intra- and inter-as ink with respect to the new assignment 25. repace p Pis,(, ik) : wis,(, ik), p = 1 by se_path 26. add t_inter(i,k) to t_intra(i,f(i,k)) 27. s(i,k) f(i,k) bw bw Remarks: Ψ (Φ)Represents the cost of inter-as ink with residua bandwidth Φ γ (Φ)Represents the cost of intra-as ink with residua bandwidth Φ intra intra

8 During neighborhood search, NSA can move the current soution to the best neighbor that either improves or worsens the quaity of the soution. To avoid cycing, a speciay designed memory ist is used to store previousy visited soutions or certain attributes of them for a certain number of iterations. A neighbor soution is reected if it is aready in the ist. In order to make the neighborhood searching more effective, an intensification or diversification technique is used to force the agorithm to expore parts of the soution space that has not been searched yet. Our NSA is outined in Figure and its fundamenta components are described as foows. b) Non-TE Starting Soutions Starting soutions for inter- and intra-as routing can be respectivey obtained by randomy assigning egress points to the inter-as traffic and then routing each intra-as traffic fow on the shortest hop paths. They are regarded as non-te soutions. Nevertheess, we wi aso evauate the impact of using TE optimized starting soutions on network performance. c) Neighborhood Structure and Search Strategy We consider a neighborhood structure that is based on shifting inter-as traffic to different egress points whie at the same time rerouting the corresponding ingress-to-egress paths. Detais of this oca transformation are as foows: We define path cost to be sum of the cost of the inter-as ink and the cost of each ink on the intra-as route to which an inter-as traffic fow has been assigned. In order to pace more importance on optimizing inter-as cost reative to the intra- AS cost, we introduce α as a factor with arge vaue to scae the inter-as cost. Note that α itsef has no particuar meaning to the Joint-TE probem. It is an intermediary to identify a the non-dominated combinations of intra- and inter-as costs. For each inter-as traffic fow, cacuate the profit cost, i.e. the saved cost minus the added cost. The saved cost (ine 1 in Figure ) is the path cost of the traffic fow minus the path cost of the traffic fow that woud have been removed. This saved cost refects how much cost woud have been reduced if the traffic fow was removed on the path. The added cost (ine 17) is cacuated for each potentia egress point except for the one that is currenty assigned to the inter-as traffic fow. It is the cost of a new path towards the potentia egress point that the traffic fow woud have been assigned minus the path cost of the origina path towards this egress point. The new path is the resut of rerouting the origina path taking into account the traffic fow (ines 1-15) and it is a minimum cost path that can be found by Dikstra s agorithm using the instantaneous intra-as ink cost as the routing metric. Consequenty, the added cost refects how much cost woud have been increased when the traffic fow is assigned onto a new egress point. The neighborhood search strategy specifies which soution in the neighborhood is chosen at each iteration. The foowing two methods are commony used: Best Method (BM): Compute the profit cost for each inter-as traffic fow. Choose the one yieding the soution with the highest profit cost as the next move. First Method (FM): Compute the profit cost for each inter-as traffic fow. Choose the first one yieding a soution with positive profit cost. It is of great importance for the soution quaity and the search efficiency. We have found in our experiments that BM can achieve approximatey 5%-1% performance improvement over FM, but the computationa compexity of BM is severa orders of magnitudes higher than the FM, which makes it impractica to use. We therefore decided to use FM in our NSA. Our finding is consistent with the prior work that has evauated the tradeoff between quaity and efficiency of BM and FM []. d) Use of Memory List The memory ist is operated as a first-in-first-out queue. The first eement in the ist is removed and then a new soution is pushed into the tai of the ist. As suggested in [29], the size of the ist depends on the size and the characteristics of the probem. We define the size of the ist to be a arge vaue (1) in order to avoid ooping. This number does not significanty affect the performance that can be achieved by the NSA because the number of potentia traffic-to-egress-point assignments that are not in the memory ist is sti very arge. e) Diversification We notice that if the NSA works on the initia soution for many iterations, this may ead the agorithm get stuck in a oca optima. To overcome this, a diversification is needed. If there is no obvious improvement in the soution for a certain number of iterations, we modify the current soution by rerunning intra-as TE on it. We define the threshod of obvious improvement to be 1% and the number of iterations to be 5. f) Stopping Criterion Many stopping criteria can be deveoped depending on the nature of the probem being studied. The most common criterion, which is empoyed in this paper, is a maximum number of iterations (MAX_ITER). However, we do not arbitrariy seect this number since the performance of the NSA is mainy dependent on how many times inter-as traffic fows can be reassigned. Therefore, the maximum iteration number shoud be reated to the number of inter-as traffic fow. In our experiments, we found that setting the maximum iteration number to be 4 times the number of inter-as traffic fows gives us a sufficienty good resut. g) Neighborhood search agorithm compexity The worst-case time compexity of the NSA is anayzed. We denote n and e the number of inter-as traffic fows and the number of egress points. The NSA cacuates the profit cost for each inter-as traffic fow by evauating each potentia egress point (ines 11-2). The most time consuming step in this bock is to find a new minimum cost route by using the Dikstra s agorithm (ine 15). The time compexity of Fibonacci-heap impementation of the agorithm is O( V og V + E ) where V and E are the number of nodes and edges in the network [1]. Since the worst-case is to examine a the inter-as traffic fows unti the first positive profit cost soution is found, the whoe step (ines 7-2) coud take O(n e ( V og V + E)) time. The NSA then iterates unti the maximum iteration number (MAX_ITER) has been reached. Therefore, the overa worst-case time compexity of the NSA can be summarized as O(MAX_ITER n e ( V og V + E)).

9 VI. EVALUATION METHODOLOGY In this section, we present our methodoogy to evauate the performances of different proposed Joint-TE strategies. A. Network Topoogy We use the Rocketfue [4] Point-of-Presence (POP) eve maps pubished by the authors, shown in Tabe 2, as our topoogies. For each topoogy, POPs correspond to cities. Some POPs have inter-as inks connected to other ASes and we ca them border POPs. Without oss of generaity and for simpicity, we assume that each border POP is associated with a virtua inter-as ink that is the abstraction of one or mutipe physica inter-as inks. Since the Rocketfue data do not contain ink bandwidth, we set the intra- and inter-as inks to be OC-48/STM-16 and OC-12/STM-4 respectivey. Tabe 2. Rocketfue topoogies used in evauation AS Name # POP nodes # Intra-AS inks 129 Sprint (US) AT&T (US) Abovenet (US) BT Backbone (Europe) # Border POP nodes B. Internet Routing prefixes For scaabiity and stabiity, inter-as TE can focus on ony a sma fraction of routing prefixes which are responsibe for a arge fraction of the traffic [14]. In this paper, we consider 2 such popuar routing prefixes. Nevertheess, each of them may not merey represent an individua prefix but aso a group of distinct routing prefixes that have the same set of candidate egress points [5] in order to improve network and TE agorithm scaabiity. Hence, the number of routing prefixes we consider coud actuay represent even a arger vaue. Each border PoP can be an ingress or egress point. In order to evauate the effect of inter-as TE on the performance of intra-as TE, we consider the situation where if a border POP receives a route advertisement towards routing prefix k from adacent AS Y, then AS Y cannot inect traffic for k into it. This corresponds to muti-hop traffic [5] in which the traffic traverses across the network instead of being directed to another egress ink of the same border POP. As a resut, we cannot assign a the routing prefixes on each border POP as route advertisements. Instead, we consider haf of the routing prefixes are randomy seected as route advertisements and the other haf as inter-as traffic in each border POP. We note that this routing prefix generation process is ust a best effort attempt to mode prefix distribution, as no synthetic mode for the actua behavior of rea networks was found in the iterature. C. Traffic Matrices We generate synthetic traffic matrices for our evauation. We generate inter-as traffic from each POP towards each of the considered routing prefixes. Note that if the POP is a border POP, it can ony inect traffic headed towards the routing prefixes that have not been seected as route advertisements. Previous work has shown that intra- and inter- AS traffic are not uniformy distributed [2,5]. According to [], AS traffic voumes are top-heavy and can be approximated by Weibu distribution with shape parameter.2-.. We therefore generate the inter-as TM with traffic demand using this distribution with the shape parameter.2. We use the Gravity Mode (GM) outined in [4] to generate the oca intra-as TM. The GM approach was proposed based on the findings in [8]. Foowing the suggestions in [5], we randomy cassify 4% of POPs as sma, 4% as medium and 2% as big. The amount of incoming traffic at a POP is proportiona to its size. D. Agorithm Parameters For the GA in the nested optimization, we use the suggested vaues from previous GA research to achieve satisfactory effectiveness and convergence rate of the agorithm [6]. The popuation size is 2 and the probabiity of mutation is.1. We set the GA to produce maximum 2 distinct inter-as TE soutions to compute and. E. Performance Metrics The foowing performance metrics are used to evauate the Joint-TE strategies. For these metrics, ower vaues are better than high vaues. Overa Intra- and Inter-AS cost: these metrics capture the overa network cost of the obective function (1). Tota bandwidth consumption: the amount of bandwidth needed to accommodate a traffic fows within the network, being the sum of the traffic oads over a the intra-as inks. Maximum intra- and inter-as ink utiization: the maximum intra-as (inter-as) ink utiization is the maximum utiization on a the intra-as (inter-as) inks in a network. Minimizing this vaue ensures that traffic is moved away from congested to ess utiized inks and is baanced over the inks. VII. SIMULATION RESULTS A the resuts presented in this paper are an average of trias. For the argest network topoogy, the simuation took approximatey two hours in average. A. Evauation of Overa Inter-AS TE Cost Overa inter AS cost Optima Inter AS traffic demand (Mbps) x 1 4 Maximum inter AS ink utiization (%) Optima Inter AS traffic demand (Mbps) x 1 4 Figure 4(a) and (b). Overa Inter-AS cost and maximum inter-as ink utiization (Sprint topoogy) We have evauated the overa inter-as TE costs achieved by a the strategies for a the network topoogies. We found that their resuts exhibit a common characteristic their overa inter-as TE costs are neary identica and very cose to the optima soution that is obtained by soving the LP of the inter-as TE probem. We therefore arbitrariy present and anayze the resuts of one network topoogy for brevity. Figure 4(a) shows the overa inter-as cost (y-axis) achieved