Scaling the Internet Routing System through Route Aggregation. Electrical and Computer Engineering

Transcription

1 Scaling the Internet Routing System through Route Aggregation André Correia Magalhães Sousa Thesis to obtain the Master of Science Degree in Electrical and Computer Engineering Supervisor: Prof. João Luís da Costa Campos Gonçalves Sobrinho Examination Committee Chairperson: Prof. Nuno Cavaco Gomes Horta Supervisor: Prof. João Luís da Costa Campos Gonçalves Sobrinho Member of the Committee: Prof. Teresa Maria Sá Ferreira Vazão Vasques October 2014

2 ii

3 Acknowledgments During all these years in college while I have been completing my degree, I would often look at this dissertation as the final period that would mark the finishing line of my academic life, and for this reason it has been a simultaneous source of anxiety and hopefulness for a long time. To be able to say that I have finally finished it, is a source of many mixed feelings, among which fulfillment and joy stand out the most. I would not have been able to come this far so soon and so successfully if not for the consistent presence and support of various people, to whom I feel some paragraphs in a single page of acknowledgments on my final thesis is a minuscule token of appreciation considering the massive importance they all have. Nevertheless, even if it is a small gesture, it is still a significant one that I believe is more than called for. First and foremost, a special word of appreciation goes out to my thesis advisor, Prof. João Luís Sobrinho. Not only was the enthusiasm with which he lectured the courses of Computer Networks and Network Algorithms an important element to help trigger my interest in this area, but it was also because of his support, his enlightened advice, his helpful suggestions, his patience, and his availability to answer my questions that this thesis came to be. Secondly, I want to express my deepest thanks to all the teachers that I have encountered during this educational ride, from the ones which taught me how to sum and multiply in first grade to the ones which taught me how to integrate and how to program at Instituto Superior Técnico. I was fortunate enough to have encountered a large number of teachers that were inspirational, extremely intelligent, helpful and, in summary, absolutely quintessential for making me grasp and comprehend many of the subjects that I now understand well. I would also like to thank my family, which have supplied me with so many opportunities in life, and have allowed me to fully dedicate myself to my academic performance. For all their kind and comforting words, for all the support they have given me, for all the joyous family time, and for giving me the freedom to spread my wings and fly in whichever direction I choose, I am extraordinarily grateful. Finally, I want to address my group of friends. It is often said a person s value may be measured through the value of its friends. If that is the case, I must be among the most highly regarded people. To have been blessed with such remarkable friends which are very caring and supportive, and that are able to accept me in all of my idiosyncrasies, peculiarities, and strangely exotic mix of personality traits, is something I am extremely thankful for. For all the fun moments, for all the walks, for all the talks, and for the company throughout the years, I thank you deeply. To all the dissertation readers that are venturing on going through the dozens of pages that follow, I give you a personal thank you as well for showing interest in what I have done, and hope that you will find it useful, or at least mildly interesting. iii

4 iv

5 Resumo A Internet tem-se expandido significativamente nos últimos anos, tanto no número de redes (Sistemas Autónomos, SAs) que incorpora como no número de prefixos IP (blocos de endereços) atribuídos às redes. De acordo com práticas operacionais atuais de encaminhamento, cada prefixo IP atribuído deve ser propagado a todos os SAs, o que constitui um fardo pesado para o sistema de encaminhamento da Internet. Esta dissertação estuda o problema da escalabilidade do encaminhamento na Internet, utilizando agregação de rotas para reduzir a informação de estado que é necessária manter e trocar entre SAs sem comprometer a qualidade dos caminhos percorridos pelos pacotes de dados. Mecanismos de agregação de rotas têm como objectivo uma substituição judiciosa de conjuntos de rotas associadas a prefixos IP longos por rotas individuais associadas a um prefixo IP mais curto que sumariza o espaço de endereços dos prefixos IP longos. Inicialmente, analisa-se o estado atual do sistema de encaminhamento, recolhendo várias estatísticas sobre a topologia da Internet, relações comerciais entre SAs, e atribuições de prefixos. Posteriormente, apresenta-se um grupo de estratégias para uma agregação de rotas distribuída, que acomodam diversas políticas de encaminhamento e diferentes fases de execução parcial. Finalmente, avalia-se a eficiência desse grupo de estratégias em modelos realistas de topologias da Internet, relações comerciais, e atribuições de prefixos. As experiências realizadas mostram que a agregação de rotas reduz o tamanho das tabelas de expedição da Internet em aproximadamente 78%. Palavras-chave: Internet, escalabilidade BGP, agregação de rotas, encaminhamento interdomínio, prefixos IP v

6 vi

7 Abstract The Internet has significantly expanded in recent years, both in the number of networks (Autonomous Systems, ASs) it incorporates and in the number of IP prefixes (blocks of addresses) that have been assigned to those networks. According to current routing operational practices, every assigned IP prefix has to be propagated to all the ASs, and this puts a heavy burden on the Internet s routing system. This thesis studies the Internet routing scalability problem, using route aggregation to reduce the state information that ASs need to maintain and exchange with other ASs without compromising the quality of the AS-paths traversed by the data packets. Route aggregation purports to judiciously substitute sets of routes pertaining to long IP prefixes by single routes pertaining to a shorter IP prefix which summarizes the address space of the long IP prefixes. We first analyze the current state of the routing system, collecting several statistics concerning the Internet topology, business relationships between ASs, and prefix assignments. Then, we present a class of distributed route aggregation strategies, accommodating different varieties of routing policies and different stages of partial deployment. Lastly, we assess the efficiency of that class of route aggregation strategies for realistic Internet topologies, business relationships, and prefix assignments. Our experiments show that route aggregation reduces the size of the Internet s forwarding tables by approximately 78%. Keywords: Internet, BGP scalability, route aggregation, inter-domain routing, IP prefixes vii

8 viii

9 Contents Acknowledgments iii Resumo v Abstract vii List of Tables xi List of Figures xiv List of Abbreviations xvi 1 Introduction Previous Work Our Approach to Routing Scalability Structure of the Thesis Fundamentals of Inter-Domain Routing AS Topology and Business Relationships Addresses and Prefixes Inter-Domain Routing: BGP and the Gao-Rexford model Isotonicity: A Special Type of Routing Policies Stability and Connectivity Snapshot of the Internet Data Sources Structures and Algorithms AS Topology and Hierarchy Paths and Routes Prefixes and Addresses Statistics and General Analysis AS Topology and Hierarchy Paths and Routes Prefixes and Addresses Distributed Route Aggregation on the Global Network (DRAGON) Multi-Homing and Provider-Independent Addresses ix

10 4.2 Filtering Prefixes Route Consistency Forwarding Consistency Aggregation Prefixes Evaluation of DRAGON Filtering Efficiency without Aggregation Prefixes Filtering Efficiency with Aggregation Prefixes Partial Deployment Stretch on AS Paths Conclusions Achievements and Results Future Work Bibliography 75 x

11 List of Tables 2.1 Gao-Rexford export policy Operation table for the Gao-Rexford routing policy Operation table for Gao-Rexford with AS-Path length routing policy Modeled tier 1 ASs Topology statistics (ASs and Links) Stub statistics Forwarding Neighbors statistics Prefix statistics FIB reduction factor statistics for RT CONS and FWD CONS FIB reduction factor results for Unicast and Anycast aggregation prefixes Differences in FIB reduction factor for a partial deployment of Tier 1 & 2 ASs Statistics concerning loss of consistency in partial deployment Stretch on AS-Path after filtering Statistics for path lengths when stretch on AS-Path > xi

12 xii

13 List of Figures 2.1 Classification of ASs according to tier and as stubs Generic Trie (left) and Tree of Allocated Prefixes (right) Construction of a FIB from the RIB Usable and unusable paths on the Gao-Rexford model Representation of policy-connectedness Execution example of the RemoveCycles algorithm Execution example of the DetermineElectedRoutes algorithm Lack of isotonicity in Gao Rexford with path length routing policy Execution example of the ElectedRouteswASPath algorithm Distribution of ASs through tiers Distribution of the elected paths among pairs of ASs Cumulative distribution of AS Path length Classification of modeled prefixes Distribution of prefixes by mask size Distribution of prefixes by number of less specific prefixes Representative network with multi-homing Impact of route aggregation on routing behavior Implicit filtering and churn reduction caused by route aggregation Illustration of the impact of route consistent filtering on q-r nodes Illustration of the impact of forwarding consistent filtering on q-r nodes Illustration of the impact of aggregation prefixes on the number of filtered prefixes Representation of the set of aggregation prefixes that an AS may generate Illustration of the effectiveness of Theorem Comparative example of UNI and ANY aggregation prefix announcements Cumulative distribution of FIB reduction factor across ASs Cumulative distribution of FIB reduction factor across non stubs Cumulative distribution of FIB reduction factor with aggregation prefixes Cumulative distribution of FIB reduction factor with customer-only aggregation prefixes Cumulative distribution of FIB reduction factor with partial deployment (Tier 1 ASs) xiii

14 5.6 Cumulative distribution of FIB reduction factor with partial deployment (Tier 1 & 2 ASs) Loss of Route Consistency in AS Path length after implementation of DRAGON xiv

15 List of Abbreviations ANY Anycast ASIC Application Specific Integrated Circuit AS Autonomous System BGP Border Gateway Protocol CAIDA Cooperative Association for Internet Data Analysis CIDR Classless Inter-Domain Routing CO AGG Customer-Only Aggregation Prefixes DFS Depth First Search DRAGON Distributed Route Aggregation on the Global Network FIB Forwarding Information Base FWD CONS Forwarding Consistent Filtering GL AGG Global Aggregation Prefixes HLP Hybrid Link-State Path-Vector IANA Internet Assigned Numbers Authority IPv4 Internet Protocol version 4 IPv6 Internet Protocol version 6 IP Internet Protocol MBW Multibit Burrows-Wheeler ORTC Optimal Routing Table Constructor RIB Routing Information Base RIPE Réseaux IP Européens RIS Routing Information Service RT CONS Route Consistent Filtering SMALTA Saving Memory and Lookup Time via Aggregation TCP Transmission Control Protocol UCLA University of California, Los Angeles UNI Unicast xv

16 c2p p2c p2p customer-to-provider provider-to-customer peer-to-peer xvi

17 Chapter 1 Introduction The Internet may arguably be regarded as one of the most defining elements of the 21 st century. It has facilitated communication between geographically distant individuals, it has allowed for a widespread propagation of various types of information and services such as news or games, and it is the foundation for a large number of social platforms. Considering the diverse type of content and services that one might obtain through the Internet, it is no wonder that there is an increasing number of individuals and companies interested in making use of it, be it for business purposes or for personal use. However, given the high demands of the fast-paced modern society, it is of crucial importance that the process of providing end-users with their requested services and content remains quick and faultless. The scalability of the Internet refers to the ability of the Internet s infrastructure to adapt to its growing size without constraining its overall performance in supplying end-users with their requested content. The Internet is an interconnected set of more than Autonomous Systems (ASs) [1], also called domains. Each AS is a network composed of end-hosts, routers and links operated by a single administrative entity. ASs may represent Internet Service Providers, content distribution networks, military networks, campus networks, as well as many other types of networks. In order to distribute content throughout the end-hosts, ASs exchange messages and data packets among them, which travel through the Internet by traversing the various links that constitute it in such a way that a path is formed connecting the source of the content to its final destination. In order to uniquely identify end-hosts of the network, each of them possesses a distinct string of bits of a fixed length called an IP address. IP prefixes are strings of bits with a shorter length which encompass all the IP addresses in which the first bits are the same as the IP prefix. For example, for IP addresses which are 4 bits long, IP prefix 01 refers to addresses 0100, 0101, 0110 and 0111, which are all the IP addresses whose initial sequence of bits is equal to the prefix. Each AS is allocated a set of IP prefixes by the Internet Assigned Numbers Authority (IANA). The AS then disseminates the IP prefixes it locally holds to the whole Internet, so that packets which are destined to an IP address contained in one of the AS s IP prefixes may reach this AS. Given the growing number of computers and other devices which are connected to the Internet, the quantity of allocated IP 1

18 prefixes has increased significantly as well. This phenomenon has lead to other problems such as the exhaustion of IPv4 addresses [2], and it is one of the elements that currently hinders the scalability of the Internet. Since each AS is continuously in need of sending packets to their destination, the AS must be aware of its set of neighboring ASs to which it should forward the packets so that they are carried through a path that will eventually reach the packets associated destination. For this purpose, ASs communicate through the Border Gateway Protocol (BGP), which is the standardized protocol used to exchange reachability information across ASs. Using BGP, ASs exchange routes, which are pieces of state information that associate an attribute to a destination IP prefix. This attribute is used to represent a specific characteristic of the path through which the route has been propagated, and routes pertaining to the same destination prefix are ordered by preference based on their attributes. Every AS collects the set of routes announced by its neighbors in a routing table, the Routing Information Base (RIB). Using the information in the RIB, a forwarding table is created, the Forwarding Information Base (FIB), which contains, for every reachable prefix in the network, which of the neighboring AS(s) announced the most preferred of all routes learned for this prefix. For every packet that traverses the AS, its destination prefix is looked up in the FIB and the packet is then sent to the corresponding forwarding neighbor(s). The decisions regarding what are the specific metrics each route uses as its attribute and what is the preference that should be given to each of them constitute the routing policy. Routes with the most preferred attributes will have been exported through paths which possess the highest quality according to the criteria of the corresponding routing policy. For example, in shortestpath routing, the attributes reflect the length of the path, and the most preferred routes would be the ones which possess the smallest length. One of the main problems that prevents a scalable inter-domain routing structure lies in the growing size of forwarding tables. The increasing number of ASs and allocated IP prefixes lead to a larger quantity of exchanged routes, which in turn will create new entries in these tables and therefore occupy more memory space. Considering that FIBs are queried on a packet-by-packet basis, it is fundamental that routers can access their FIB and determine the entry associated to a packet s destination as quickly as possible, which is why routers use the fastest hardware such as ASIC memory chips to locally store the FIBs [3]. Therefore, its growth in size, management burden and lookup time present a clear bottleneck in the performance of routers. There has been a significant growth in the number of routes that occupy BGP routing tables, with an increase of approximately entries per year for the last couple of years [4]. Recently, in August of 2014, the amount of BGP routes in the Internet reached 512K (or ), which is the default superior memory limit in some of the older Cisco routers routing tables capacity, having lead some of them to slow down or even shut down completely [5] [6]. Given the dynamic nature of the Internet, there are various types of network events that occur constantly which require ASs to communicate via BGP to inform one another of routing updates. For example, link failures or additions, as well as the allocation of new IP prefixes may often demand ASs 2

19 to announce newly generated routes to their neighbors, withdraw previously announced routes or send routes with updated attributes. The rate at which these updates are announced via BGP is called churn, and it can have a noticeable impact on scalability [7]. As an illustrative case, consider the withdrawal of a route: if the churn is too high, it may increase the time taken for this update to reach an AS and, during that time, the AS will still contain that route in its routing tables, and will therefore be sending data packets to a route which in actuality no longer exists. In summary, the main issues concerning the inter-domain routing scalability which constitute some of BGP s main vulnerabilities are in the increasing size of routing and forwarding tables, and in the increasing rate of BGP updates. The scalability problem is becoming critical due to the growing number of ASs and IP prefixes, since BGP does not have the proper mechanisms to handle this expansion. 1.1 Previous Work The vulnerabilities of BGP prevent it from scaling well with the expansion of the Internet, such that routing scalability has been said to be the most important problem facing the Internet today [8]. Hence, various solutions that aim to solve this problem have been suggested and studied. Many solutions focus on optimizing the representation of the FIB, since this is the main data structure on routers that needs to have a fast lookup time and low memory space occupation. FIB aggregation is a software solution that operates locally on each AS and focuses on reducing the amount of FIB entries by eliminating redundancies such that the overall routing behavior in terms of traversed paths and route attributes is not affected by the removal of those entries. Many variations of FIB aggregation algorithms have been around, including the Optimal Routing Table Constructor (ORTC) suggested by Draves et al. [9], which is provably optimal on the number of FIB entries. This solution, however, is not adaptable to incremental route updates, meaning that it must be executed on the full FIB periodically in order to account for routes which may have been added or withdrawn meanwhile. However, strategies to make ORTC adaptable to incremental FIB updates have been proposed, such as SMALTA (Saving Memory and Lookup Time via Aggregation) [10]. Another solution is FIB compression, which is a strategy that, besides reducing the number of FIB entries, also focuses on optimizing the data structure utilized for the storage of the FIB to a minimal memory occupation, such as the Multibit Burrows-Wheeler transform (MBW) [11]. The main disadvantages of FIB aggregation and FIB compression solutions are that they are local to each AS and focus exclusively on the FIB, not taking into account other important scalability issues such as the size of RIBs and churn. Other approaches to this problem include the creation of new inter-domain infrastructures and protocols which do not contain the scalability vulnerabilities present in BGP, and would therefore substitute BGP. These are called clean-slate routing architectures [12]. One of these is the Hybrid Link-State Path-Vector (HLP) protocol [13], which defines distinct hierarchies of ASs such that the rate of routing updates between ASs of different hierarchies is lower than the exchange among ASs of the same hierarchy, thereby reducing the overall churn in the network when compared to BGP. The main disadvantage 3

20 of this technique is that, by suggesting a completely new inter-domain routing protocol, it would require an extremist redesign of the Internet. Exploring through the aforementioned solutions, we would be hard-pressed to find one that simultaneously aims to optimize the representation of routing tables (FIBs and RIBs) as well as minimize churn on the network, while being compatible with BGP, which is what we wish to accomplish. 1.2 Our Approach to Routing Scalability The approach we use to handle the scalability problem is centered on the idea of having ASs filter prefixes. We say that an AS filters a prefix if it chooses not to store any of the learned routes pertaining to that prefix in its local FIB, nor does it export any of those routes to its neighboring AS(s). Filtering prefixes, however, suggests loss of information. Having some ASs cease the announcement of all routes pertaining to a prefix, it stands to reason that some of the other ASs will no longer learn these routes. Therefore, how can we be sure that filtering these prefixes will not affect the paths traversed by packets to reach their destination? The FIBs of ASs may contain more than one prefix which will refer to the same destination address. For example, considering 4-bit IP addresses, 0110 is included in the range of addresses encompassed by prefixes 0, 01 and 011. The longest prefix match rule states that, if a destination address matches more than one prefix of the FIB, the entry with the longest prefix should be used for forwarding decisions. Therefore, in the previous example, only the route pertaining to prefix 011 would be considered for packets destined to Taking advantage of the existence of various prefixes in the FIB which match the same destination address, we use a formal model of routing to prove that filtering prefixes according to a carefully outlined criteria will maintain the routing quality of the paths followed by data packets. Another important aspect to consider is that the Internet is decentralized, meaning that there is no central system of coordination to manage how the decisions made at each AS affect the remaining ASs. Hence, one might ponder on the impact that one AS s decision to filter prefixes might have on the routes of other ASs, and in the overall routing behavior of the Internet. Also, even though this would contribute to global scalability, is it in each AS s best interest to filter prefixes, in terms of economic gain? Is this kind of mechanisms compatible with BGP, or will it require additional protocol features? In this thesis, we give a thorough study on route aggregation mechanisms, which are strategies involving filtering prefixes, and we will see that the answers to the previous questions are remarkably optimistic. Route aggregation mechanisms purport to substitute sets of routes pertaining to long prefixes by single routes pertaining to shorter prefixes which encompass the addresses of the longer prefixes. Filtering prefixes constitutes an inviting technique to handle scalability, since it simultaneously improves the size of FIBs and RIBs, by reducing the number of entries they will possess, as well as the network churn, given that there will be an inferior quantity of route announcements and updates involved. The idea behind route aggregation was introduced by Sobrinho and Le in 2012 [14]. In their article, various types of route aggregation mechanisms are analyzed. Recently, in 2014, a distributed route aggregation algorithm was suggested, called DRAGON (Distributed Route Aggregation on the Global 4

21 Network) [15]. This thesis will look further into DRAGON, studying the results we might expect from its network-wide implementation on the Internet. If we see that this strategy is compatible with BGP, maintains the global inter-domain routing behavior, and filters a considerable number of prefixes, then it is clearly an appealing method of mitigating the scalability problem of the Internet routing system. 1.3 Structure of the Thesis We start in Chapter 2 by presenting the basic concepts and definitions necessary for describing the Internet s AS-level topology, the announced set of IP prefixes, and the inter-domain routing system, as well as the notation used throughout the remaining sections. Chapter 3 is dedicated to obtaining an inferred topology of the Internet, as well as a tree of allocated prefixes, through various datasets publicly available online. In this chapter, we develop algorithms used to transform these datasets into data structures out of which we can collect statistics to develop a better understanding of the main aspects that characterize the Internet s AS-level topology and the set of allocated IPv4 prefixes. In Chapter 4, we study route aggregation mechanisms from a theoretical point of view, and analyze the impact they might have on reducing routing table sizes and churn. We look into techniques for filtering prefixes without compromising the routing quality of the network, which constitute the distributed algorithm called DRAGON. In the final section of the chapter, we explore the concept of aggregation prefixes, a measure used to improve the effectiveness of route aggregation by announcing additional prefixes in the network. For Chapter 5, we evaluate the effects of DRAGON on the previously developed Internet model, with and without the use of aggregation prefixes. We define a metric used as reference for measuring the impact of DRAGON on routing scalability, and present the obtained results of implementing this algorithm both network-wide and by only some ASs. Final conclusions on the obtained achievements and results, as well as considerations on what future work could be done to further continue the study on route aggregation are presented in Chapter 6. 5

22 6

23 Chapter 2 Fundamentals of Inter-Domain Routing 2.1 AS Topology and Business Relationships The Internet s AS-level topology is based on the ASs and the different types of business relationships that can be formed between two distinct ASs. The current flow of data packets across the Internet is mostly determined by two contractual agreements that pairs of ASs may establish between themselves, which determine the types of economic benefits that each of the involved parties obtains and under what conditions, and they are customer-provider and peer-peer [16]: Customer-Provider - A customer AS settles a form of payment with a provider AS in order to have access to the ASs that the provider can reach and to be able to exchange packets with them. Peer-Peer - Two ASs form a peering settlement between them by agreeing to exchange their customers packets free of charge. In order to create a mathematical model that accurately portrays the topology of the Internet, we resort to graph theory and choose to represent the AS-level topology as a graph G = (V, E) with a set of vertices, or nodes, V = {u 0, u 1,..., u n } and a set of links E V V, where each node represents an AS and each link represents a business relationship established between the connected ASs. By modeling the business agreements as links in the graph, we may characterize each link (u 0, u 1 ) in one of three categories: p2c, provider-to-customer (u 0 is a provider of u 1 ); p2p, peer-to-peer (u 0 is a peer of u 1 ); and c2p, customer-to-provider (u 0 is a customer of u 1 ). For every p2c link, there is a c2p link in the opposite direction (if u 0 is a provider of u 1, then u 1 is a customer of u 0 ), and for every p2p link, there is also a p2p link in the opposite direction (if u 0 is a peer of u 1, then u 1 is a peer of u 0 ). A path in the Internet is interpreted as an ordered sequence of ASs in which every sequenced pair of ASs has established a business relationship between them. Using the graph model, we define paths as ordered sequences of distinct nodes (u n,..., u 1, u 0 ) such that each pair (u i, u i-1 ) for 0 < i n is a link in E. 7

24 Each AS has a set of neighboring ASs with which it has directly established one of the aforementioned business relationships. In terms of the graph model, we represent this set as a given node u s set of neighbors, and they contain all nodes which have a link connecting them to node u. The set is represented as N(u). Besides the neighbors of node u which are its customers, we may still define a node v as an indirect customer of node u (equivalently, node u is an indirect provider of node v) if there is a path (u n,..., u 1, u 0 ) connecting node u n = u to node u 0 = v, where all links (u i, u i-1 ) for 0 < i n are of type p2c. We have found it useful to pinpoint which ASs have greater economic advantages in the inter-domain routing system, and for that purpose we establish a criteria for each individual AS that reflects the leverage it possesses in the overall topology, and whether the AS needs to pay other ASs in order to reach every other destination in the Internet. To this end, we use the concept of tiers. A tier is a number with which each AS is classified in order to loosely represent its economic status in the routing system. We define tier 1 ASs as those which possess no providers, and they are the most economically powerful elements, since they do not pay to any other AS for the exchange of data packets. Even though there is no consensus on the rigorous definition of tiers, this is among the most common definition for tier 1 ASs [17]. In the case of any arbitrary AS which is not tier 1, we determine the value of its tier from the shortest path that connects this AS to a tier 1 AS exclusively through c2p links. Therefore, ASs which are customers of tier 1 ASs will be tier 2 ASs, ASs which are customers of tier 2 ASs but not of tier 1 ASs will be tier 3 ASs, and so forth. This definition represents how close an AS is to being a tier 1 AS, and can be mathematically formulated as such: 1 if u has no providers Tier(u)= 1 + min v provider of u Tier(v) otherwise Besides the presented function, other metrics can be used for the definition of an AS s tier, such as the number of direct and indirect customers or the total number of destinations that an AS can reach through their customers. Despite the suggestion of various definitions for tiers, we have seen that for most of them, we arrive at a similar characterization of the Internet s AS-level structure as a hierarchy established by the customer-provider business relationships where tier 1 ASs are a small part of the topology which are at the top of the hierarchy and a greater number of domains belong to an intermediate tier of 3 or 4, lower in the hierarchy [18] [19]. Another important element when describing the Internet topology is the set of ASs located in the edges of this hierarchy. We call these ASs stubs, and they are defined as ASs that do not have any customers, and are therefore entirely dependent on their peers and providers for the exchange of data packets. Figure 2.1 illustrates the classification of ASs according to their tier and as stubs in a generic network. 8

25 Figure 2.1: Classification of ASs according to tier and as stubs 2.2 Addresses and Prefixes An IP address is a string of bits of a fixed size (32 bits for IPv4 addresses) used to label and identify devices on the Internet, and it is interpreted as an indicator of the respective end-host s location while exchanging data packets with the Internet Protocol (IP). An IP prefix is a string of bits of a smaller length than that of an IP address which represents the set of IP addresses with the same initial bits. The set of IP addresses that an IP prefix encompasses is called its address space. For example, if we consider IP addresses of length 3, some possible IP prefixes in this network would be 0 (whose address space would contain addresses 000, 001, 010 and 011) and 10 (whose address space would contain addresses 100 and 101). IP addresses are presented in human-readable notations: IPv4 addresses use the dot-decimal notation (represented as x 1.x 2.x 3.x 4, where each x n is the decimal represent of the nth set of 8 bits in the string). IP prefixes are usually represented in the format <First Address>/<Mask>, where <First Address> is the lowest IP address included in the prefix s set of addresses, and <Mask> is the prefix s length in number of bits. An example of an IPv4 prefix is /24, which embraces addresses to An important concept associated to prefixes is specificity: a prefix p is said to be less specific than prefix q (equivalently, q is more specific than p) if p has a length shorter than the length of q, and it coincides with the first bits of q. For instance, prefix 10 is less specific than Less specific prefixes encompass a larger address space than the address space embraced by its more specific prefixes. Another important concept is parent-child prefix pairs. A prefix p is a parent of prefix q (equivalently, q is a child of p) if, within the set of all allocated prefixes in a network, which we denote as, p is the longest of all the prefixes that are less specific than q in. For instance, if = {0, 1, 10, 1000}, then the parent of 1000 is 10 as, out of all the prefixes less specific than 1000 present in (1 and 10), it is the longest one. However, if prefix 100 were allocated and the set became = {0, 1, 10, 100, 1000}, then the parent of 1000 would be 100. When wishing to denote a parent-child prefix pair, we use two alphabetically sequential letters: in (p,q), p is a parent of q. Prefix trees aim to represent the existing prefixes in a network by means of a tree structure. In this tree, each node represents a prefix, and their ramifications point to more specific prefixes. As convention, 9

26 we define the top of all prefix trees as the empty prefix (-), which is the parent of all the prefixes with no less specific prefix in the set. We will focus mostly on two types of tree representations: 1. Trie - In this tree, for every allocated prefix, the entire set of less specific prefixes, even the ones not announced in the network, is represented as well. Each node may have, at most, two ramifications, which represent the prefixes obtained by appending one extra bit (0 or 1) to the prefix represented at this node. A node only exists in this structure if it represents an allocated prefix or if there is at least one prefix which is more specific than it that exists in. 2. Tree of Allocated Prefixes - This structure only uses the set of prefixes which have been allocated in a network (the prefixes that constitute ), along with the empty prefix at the top. A node s ramifications represent the respective prefix s child prefixes. Figure 2.2: Generic Trie (left) and Tree of Allocated Prefixes (right) The two possible prefix tree representations on a network in which the announced prefixes are {0, 1, 00, 10, 110, 111, 0101, 0110, 0111, 1100, 1101} are presented in Figure Inter-Domain Routing: BGP and the Gao-Rexford model The core elements of the inter-domain routing system are routes, which are pieces of state information that associate an attribute to a destination prefix. Each AS may locally hold a set of prefixes, for which the AS is the destination of data packets, and generates a route pertaining to each of those prefixes. Routes pertaining to a prefix p are called p-routes, and we denote the node in the graph model representing the AS that is the destination for a prefix p as t p. The AS then exports this generated set of routes to a subset of its neighbors, which in turn will export the learned routes to their neighbors, and so forth, propagating the routes throughout the network. The criteria that determines to which of its neighbors should an AS announce each route is called the export policy. The export policy is a part of the routing policy, whose definition is centered on two main aspects: operations between links and routes and relative preference between routes. 10

27 Whenever an AS exports a route to its neighbor, that neighbor locally recalculates the route, whose new attribute value depends on the link that the route was exported through. Hence, as a route is propagated across the network, it might change its attribute depending on the types of links it traverses. In order to account for this phenomenon, links in the network are associated to labels, which are functions that transform a route attribute into another route attribute. The set of all possible route attributes in a routing policy is represented by Θ, and labels will be denoted as functions L: Θ Θ. The label associated to a specific link (u,v) will be represented as L[u,v], and the set of all labels within a routing policy will be represented by L. The notation L[u,v](α) represents the transformation that occurs on route α once it is exported from node v through link (u,v) to node u. Each AS may learn various different routes from its neighbors pertaining to a given prefix. However, each pair of routes can be compared in terms of relative preference based on their attribute: a given route A will be either preferred to route B, less preferred than route B, or equal to route B. This may be extended to a full set of routes, meaning that each AS elects only one route for a certain prefix: the route which is preferred over all other routes learned pertaining to the same prefix. We use the symbol as a representation of the absence of a route, and it is always included in Θ, being the least preferred of all routes. The attribute of the route elected at a given node u for prefix p is given by R[u;p]. The following symbols are the notation we use when comparing two routes A and B in terms of preference: A B - A is either less preferred than B, or equal to B A B - A is less preferred than B A B - A is preferred to B A B - A is either preferred to B, or equal to B This preference ordering is transitive, meaning that if route A is preferred to route B, and route B is preferred to route C, then route A is preferred to route C: A B C A C. In order to avoid exporting too many routes on a network, the basic structure of inter-domain routing is centered on the fact that, of all the routes it has learned for a given prefix p, an AS only announces its elected route for this prefix, i.e. for every prefix p, each node u only announces R[u;p]. We say that a network has reached a final stable state for p-routes when there are no p-routes traversing links, and there are no nodes which still have to export a p-route to one of its neighbors. Once this state has been reached in the network, we know that each node will have elected the p-route with the most preferred attribute of all p-routes it locally recalculated from the p-routes announced by its neighbors, and therefore we may define this basic property of a node s elected p-route: v N(u) R[u;p] L[u,v](R[v;p]) Another important concept is forwarding neighbors. For each prefix p, an AS s set of forwarding neighbors is the subset of its neighbors which announced the p-route that was locally transformed into the AS s elected p-route. These neighbors are the ones to which the AS forwards the data packets 11

28 destined to p. A node u s set of forwarding neighbors for a certain prefix p is represented by FN[u;p], and it may be formally defined as: v N(u) v FN[u;p] R[u;p] = L[u,v](R[v;p]) v / FN[u;p] R[u;p] L[u,v](R[v;p]) We define a forwarding path for prefix p as a path (u n,..., u 1, u 0 ) in which each of the nodes is a forwarding neighbor for p of the previous node, i.e. u i-1 FN[u i ;p] for 0 < i n. This is the path that the packets at u n will traverse when their destination address is within the address space of prefix p, since each of the nodes will send it to its forwarding neighbors after receiving the packet. Each AS possesses two main data structures that aid in the process of choosing and collecting routes for every prefix: The Routing Information Base (RIB) is a dataset which contains a table for every one of the AS s neighbors. Each of these tables contains an entry for every prefix p, containing the route attribute announced by this neighbor for prefix p after being locally recalculated at the AS. The Forwarding Information Base (FIB) is a table which contains an entry for every prefix p, containing the forwarding neighbor(s) to reach it. The FIB is constructed from the RIB by selecting, for each prefix p, the neighbor(s) whose associated RIB entry pertaining to that prefix contains the most preferred p-route attribute of all the p-route attributes in the tables of the RIB. Figure 2.3: Construction of a FIB from the RIB Figure 2.3 illustrates the FIB and RIB of a generic AS X, which has ASs A, B, Y and Z as its neighbors. In the example, we only show the entries of the tables pertaining to two arbitrary prefixes /24 and /24. For this AS, the elected routes have attribute 1 for prefix /24 and for prefix /24. The sets of forwarding neighbors are {Y,Z} for prefix /24 and {A} for prefix /24. For this particular example, we have a multipath for prefix /24, given that data packets destined to this prefix may be forwarded at AS X to more than one AS (Y and Z). The information present in an AS s FIB determines its behavior regarding the forwarding decisions 12

29 for the data packets it receives and generates. A packet will contain a destination address, and in order to make sure that the packet reaches its destination through the most preferred route, the AS searches through its FIB in order to find the prefix whose address space includes the packet s destination address. In the event that there is more than one prefix under these conditions, ASs abide by the longest prefix match rule, which states that it should choose the FIB entry with the longest prefix that encompasses the destination address. Once this FIB entry is determined, the AS proceeds to send the packet to the prefix s associated set of forwarding neighbors. The standardized protocol in the Internet to exchange routing data among ASs is called the Border Gateway Protocol (BGP). This protocol is used to exchange messages for announcing routes and for withdrawing routes that were previously announced, which we generically call route updates. A standard BGP route will contain the destination prefix and route attributes. Although there are various attributes in a BGP route, we will focus on the two most relevant ones, LOCAL PREF and AS PATH [20]: LOCAL PREF - This stands for local preference and is the main route attribute used for comparing and electing routes. The elected route at an AS is the one which has the highest LOCAL PREF value. AS PATH - This is the sequence of ASs that constitute the path traveled by the route. It is not the primary route attribute used for comparing routes, but it may be the next element used for tie-breaking. In this situation, among routes that have an equal value for LOCAL PREF, the AS may choose to prefer the ones which have an AS PATH with the least number of ASs. The Gao-Rexford model [21] is the definition of a routing policy on BGP which accurately portrays the possible types of paths that the customer-provider and peer-peer agreements impose on the network. The routing export policy can be determined by taking certain aspects of these business relationships into consideration: 1. Since the customer-provider relationship provides the customer with total access to the provider s routes, all routes are exported to customers. 2. Since ASs gain nothing from scenarios in which there are data packets flowing from one of its peers/providers to another one of its peers/providers, then ASs only export to peers and providers the routes learned from customers. This is the basic Gao-Rexford routing export policy which is put into practice in BGP, and it is summarized in Table 2.1. This export policy creates restrictions in the amount of existing paths in the AS topology network that can be used to send packets, since there are some paths in the network which an AS may not be aware of, because its neighbors did not announce the routes learned from them. Considering the types of agreements that ASs can establish between themselves, there are only three possible types of paths that packets may use to travel throughout the Internet from a node u n to a node u 0 (all other paths are unusable): 13

30 Route announced to Route learned from Customer Peer Provider Customer Yes Yes Yes Peer Yes No No Provider Yes No No Table 2.1: Gao-Rexford export policy Customer Path: (u n,..., u 1, u 0 ) is a customer path if all links (u i,u i-1 ) for 0 < i n are of type p2c, or if the path is composed entirely of one node. Peer Path: (u n,..., u 1, u 0 ) is a peer path if link (u n,u n-1 ) is of type p2p and the path (u n-1,..., u 1, u 0 ) is a customer path. Provider Path: (u n,..., u 1, u 0 ) is a provider path if there is a value 0 k < n such that all links in the path (u n,..., u k ) are of type c2p and the path (u k,..., u 0 ) is either a peer path or a customer path. These are all the possible paths one may conceive in the network in order to eliminate the possibility of data packets flowing from an AS s peer/provider to another peer/provider. This is referred to as valley-free routes. Figure 2.4 illustrates some examples of usable and unusable paths. Figure 2.4: Usable and unusable paths on the Gao-Rexford model The established inter-as agreements also conduct the preference level which is given to each of the routes learned from an AS s neighbors. 1. A customer path is preferred to peer and provider paths 2. A peer path is preferred to a provider path 3. A provider path is less preferred than peer and customer paths We have built this preference ordering based on the Gao-Rexford model: while customer paths are the most preferred types of paths since the customer is paying to carry the traffic, we also add that peer paths should be preferred to provider paths given that a path which incurs in no payment (peer path) ought to be preferred to a path which will lead to some expense (provider path). An AS classifies 14

31 the routes it imports in one of three categories, considering the agreement it has with the AS which announced them. If a route is locally generated or learned from a customer, it is a customer route (C): it represents a customer path in the network. Similarly, routes learned from peers (which represent peer paths) and routes learned from providers (which represent provider paths) are peer routes (R) and provider routes (P), respectively. Resorting to our definition of a mathematical model for representing the inter-domain routing system, we can formally define this routing policy. The set of route attributes will be given by Θ = {C,R,P, }, in which the preference ordering is C R P, and the set of labels will be L = {c2p, p2p, p2c}. Considering the network on Figure 2.4, examples of c2p labels would be L[E,B] and L[G,C], whereas examples of p2p labels would be L[B,C] and L[A,B], and examples of p2c labels would be L[C,F] and L[D,H]. A possible representation of a routing policy is through its operation table, a table which translates the effect that each of the labels in L has over each of the route attributes in Θ. The first row of this table contains all elements in Θ, and the first column contains all elements of L. All remaining entries contain the output of the label in the corresponding row when the input is the route attribute in the corresponding column. The operation table for the Gao-Rexford model is presented in Table 2.2. C R P p2c C p2p R c2p P P P Table 2.2: Operation table for the Gao-Rexford routing policy One can see that Table 2.2 accurately reflects all main characteristics of the Gao-Rexford model, including the export policy. Customer routes (C) are exported to all neighbors, i.e. through all possible link labels: p2c(c), p2p(c) and c2p(c). Also, peer routes (R) and provider routes (P) are only exported to customers, i.e. the only labels L for which L(R) and L(P) are c2p (the ones in which the extremity receiving the route is the customer). It should also be noted that, if a route is exported, the value it is converted to depends only on the AS which sends the route, i.e. whenever the output of the label is not, we will have p2c(α)=c, p2p(α)=r and c2p(α)=p, labels for links in which the extremity exporting the route is a customer, a peer and a provider, respectively. 2.4 Isotonicity: A Special Type of Routing Policies An important property that routing policies may have is isotonicity. This property states that the relative preference between two routes is maintained when applying the same link label to both of them, and can be formally expressed as: α, β Θ L L α β L(α) L(β) Isotonicity is important due to the fact that it is a sufficient condition for the route selected locally at 15

32 each node to represent the best existing route at this node pertaining to the associated prefix [22]. Let us consider a situation without isotonicity, i.e. there being at least a pair of attributes α and β in Θ, and a label L in L such that α β and L(α) L(β). If a certain node v has routes with attributes α and β for p, then it will only export α to its neighbors, as it is the most preferred route attribute at node v. If node u learns of a route with attribute α from node v through a link with label L, then it calculates a route with attribute L(α) for prefix p, which is less preferred than L(β), meaning that it does not obtain its most preferred route in the network to reach the destination for p. One way to verify if a certain routing policy is isotone is by analyzing its operation table, like the one presented in Table 2.2. By ordering the routes in the first row from the least preferred to the most preferred, isotonicity is ensured if each of the following rows also maintains that same order in the routes, which means that the relative preference between routes does not change after applying to them a common link label. We can therefore conclude that the Gao-Rexford routing policy is isotone. Let us now look into the Gao-Rexford routing policy with consideration for the AS-Path length. In this routing policy, nodes opt between routes which have the same Gao-Rexford attribute (customer, peer or provider) by choosing the one which has the smallest number of ASs in its path. The routes may now be formally represented as a pair of route attributes (k,n) where k is the Gao-Rexford attribute (customer - C, peer - R, provider - P) and n is the length of the AS-Path. The preference between routes is therefore given by: (k 1,n 1 ) (k 2,n 2 )if k 1 k 2 or if k 1 = k 2 and n 1 < n 2 For this routing policy, the operation table is presented in Table 2.3. The lack of isotonicity occurs due to provider routes. As we can see, the last row (provider routes) is the only one that breaks the ordering of routes from least preferred to most preferred, meaning that there are routes which, after being exported from a provider to a customer, may change their preferential ordering. For example, a customer route with 3 links is preferred to a peer route with 1 link: (C,3) (R,1). However, after being exported to a customer through a c2p-link, they become, respectively, a provider route with 4 links and a provider route with 2 links, changing their relative preference: c2p(c,3) c2p(r,1) (P,4) (P,2). (C,0) (C,1)... (C,n)... (R,1) (R,2)... (R,n)... (P,1) (P,2)... (P,n)... p2c (C,1) (C,2)... (C,n+1) p2p (R,1) (R,2)... (R,n+1) c2p (P,1) (P,2)... (P,n+1)... (P,2) (P,3)... (P,n+1)... (P,2) (P,3)... (P,n+1)... Table 2.3: Operation table for Gao-Rexford with AS-Path length routing policy 16

33 2.5 Stability and Connectivity In order to ensure that a routing policy reaches a final correct state, it is important that it does not allow for the occurrence of route oscillations. A route oscillation is a specific scenario in the network in which a sequence of route updates is continuously sent among the same ASs repeatedly. The possibility of route oscillations is supported on the existence of cycles. A cycle is a specific type of path (u n,..., u 1, u 0 ) in which the first and final nodes are the same (u n = u 0 ). Cycles enable the continuous exchange of routes among them since, once node u 0 receives the route, it may be possible for the route to once again traverse the same sequence of links in the path repeatedly. So as to ensure that a routing policy converges to a final stable state, cycles in the network topology must obey a certain property which we call freeness [22]. We say that a cycle (u n,..., u 1, u 0 ) is free for a given prefix p if all nodes possess a route pertaining to it and there is at least one node u i in which the elected route attribute is preferred to the route attribute calculated from the route learned from the following node u i-1, formally expressed as: 0 < j n R[u j ;p] 0 < i n R[u i ;p] L[u i,u i-1 ](R[u i-1 ;p]) It is possible to show that, if all cycles are free for every prefix in the network, then the routing policy converges to a stable final state without route oscillations [22]. The stability of inter-domain routing on the Internet is therefore supported on the hypothesis that no such cycles exist in the topology. By definition, we see that a cycle is not free if all nodes in the cycle elect the route they learn from the next node. In the case of the Gao-Rexford model, the only cycles that could lead to this phenomenon are provider-customer cycles. These are cycles (u n,..., u 1, u 0 ) in which all links (u i,u i-1 ) for 0 < i n have the label p2c (or, if we analyze the cycle in the reverse direction, all links have the label c2p). Consider, for instance, a simple provider-customer cycle (u 3, u 2, u 1, u 0 ) where u 0 = u 3. Since the Gao-Rexford export policy states that all customer routes are exported to a node s providers, if u 0 receives a customer route, it exports this route to u 1, which, in turn, exports this route to u 2, which ends up exporting the route back to u 3 = u 0. Therefore, since u 3 is now one of u 1 s forwarding neighbors for this route, once u 1 sends packets to it, they will be continuously traveling through the sequence u 3, u 2 and u 1 : this is called a forwarding loop. It is ordinarily safe to assume that no provider-customer cycles exist in the Internet, since ASs usually choose as providers other ASs which have a larger scope and reachability across the Internet. In fact, the existence of such cycles would defeat the hierarchical structure of ASs and their grouping in tiers, since it would create the possibility of all ASs having at least one provider, thereby precluding the classification of tier 1 ASs. For instance, it is usual for an AS serving a metropolitan area to be a customer of a regional provider AS, which in turn could be a customer of a national provider AS - it is unlikely for the nationwide AS to derive any economic advantage from being a customer of the metropolitan area AS [21]. Another common assumption about the Internet topology is that it is policy-connected, which is defined as meaning that there is a usable path connecting every possible pair of ASs. Policyconnectedness ensures that there will be no prefix for which the elected route at any AS is. For 17

34 the Gao-Rexford model, this means that every source-destination AS pair will have either a customer path, a peer path or a provider path connecting them. Figure 2.5: Representation of policy-connectedness Figure 2.5 represents two networks, in which the rightmost network is policy-connected. On the leftmost network, we can see that there is no usable path connecting, for example, ASs C and E. Even though it is possible to form a path from C to E using the links in the network, such as (C, A, B, F, E), it will not be a usable path according to the Gao-Rexford routing policy: in the path given as example, AS B would not export to its peer A the route learned from its provider F. Given that the Internet is commonly deemed as an interconnected set of networks in which every AS can reach destinations at every other AS, a policy-connected topology is a realistic assumption. 18

35 Chapter 3 Snapshot of the Internet In order to take conclusions on the effectiveness and efficiency of route aggregation mechanisms, an important initial step is to analyze the current state of the inter-domain routing structure. To this end, we have collected some data concerning the Internet s AS-level topology and the overall set of IPv4 prefixes which are announced in it. We then carefully examine its most relevant characteristics, taking into account how accurately they portray the current Internet structure, and how we may make the most of them when putting a distributed route aggregation algorithm into practice. 3.1 Data Sources The dataset detailing the inferred types of inter-as relationships on the Internet was obtained from the UCLA archive [23], and the existing IPv4 prefixes, as well as the corresponding ASs which announce them, were obtained from the CAIDA prefix to AS mappings [24]. These datasets rely mainly on the use of BGP data collectors, which consist of strategically located sets of routers that establish a BGP session with other ASs through a peer-peer relationship, that make their routing tables and BGP route updates publicly available. Among the most relevant ones used are RouteViews [25] and RIPE-RIS [26]. Analyzing the sets of routes in the BGP data collectors routing tables, it is possible to execute algorithms which infer the type of existing inter-as relationships on the Internet, as well as algorithms that detect which AS announces each prefix. 3.2 Structures and Algorithms AS Topology and Hierarchy The first step in analyzing the current Internet s AS-level topology is in representing it and storing it. Making use of the graph model, we represent the network as a graph G = (V, E) where each AS is a node in V and each inter-as relationship is a link in E. The data structure used to represent the graph 19

36 is a vector, where each of its entries is a node. At each node, there are three distinct lists: a list of customers, a list of peers and a list of providers. Since the number of ASs in the network is considerable (approximately ), we need to optimize the program so that the analysis may be executed with a relatively small run-time complexity and so that the data structures do not overflow the temporary storage memory. Therefore, we allocate 2 16 entries for the vector, which was deemed the most appropriate size since it is not too large as to occupy too much memory and it is not too small allowing for each AS s numeric identifier to be used directly as index. Tier 1 ASs In order to establish the hierarchical structure of the topology, we start by identifying the AS numbers which can be considered tier 1. Even though we have defined tier 1 ASs as those which have no providers, this designation is used to classify ASs which are considered to be the most economically powerful of the Internet, in the sense that they are among the ASs with the greater amount of customers, therefore being able to reach every other AS in the network without resorting to provider routes. The identification of tier 1 ASs was done using a heuristic approach: an AS is considered to be tier 1 if it has no less than MINCL direct customers and no more than MAXPR direct providers. Testing possible values for MINCL and MAXPR, we then proceeded to verify how the set of obtained tier 1 ASs is ranked according to CAIDA [1], which ranks each AS by the total amount of direct and indirect customers. Settling on the values 400 for MINCL and 5 for MAXPR, we obtained the set of ASs presented in Table 3.1 and then removed the links which connected these ASs to their providers, if they had any, so that they meet our definition of tier 1 ASs by having no providers. AS Number Organization Number of Number of AS Rank Name customers providers (CAIDA) 174 Cogent Communications Qwest Communications Company, LLC MCI Communications Services, Inc. d/b/a Verizon Business Sprint TeliaSonera International Carrier XO Communications NTT America, Inc Tinet SpA Level 3 Communications, Inc Level 3 Communications, Inc Tata Communications Abovenet Communications, Inc AT&T Services, Inc Table 3.1: Modeled tier 1 ASs 20

37 Provider-Customer Cycles When creating a topology which accurately represents the Internet, it is important to make sure it satisfies the assumptions we have established about it. One of the main aspects which is important for the convergence of the Gao-Rexford routing policy is the absence of provider-customer cycles. We have detected some provider-customer cycles in the UCLA dataset, which may be due to some imperfections in the inference algorithm, as well as ASs which have different economic advantages in distinct parts of the globe (for example, an AS may be a provider of another AS in one continent, but a customer of that same AS in another continent). In order to remove provider-customer cycles from the topology, we resort to a DFS (Depth-First Search) which uses only p2c links. The algorithm explores paths in the network formed by a sequence of ASs where each AS is a customer of the previous one. Once it encounters an AS which is a customer of the AS at the end of the path, but already belongs to the path, then this implies the existence of a provider-customer cycle, which is then eliminated by removing the appropriate p2c link. A pseudocode of this is presented in Algorithm 1. This algorithm uses two auxiliary variables at each AS to aid in the global DFS: visited and open. The visited variable is used to distinguish the ASs which have already been explored at some point during the algorithm (visited = 1) from those which have not (visited = 0), and its main purpose is to avoid exploring p2c links more than once. The open variable identifies the ASs which belong to the current path of ASs being tested, in which open = 1. Algorithm 1 Algorithm for removing provider-customer cycles in a network 1: procedure REMOVECYCLES(u) u is the last node in the path 2: u.visited 1 All nodes are previously initialized with visited as 0 3: u.open 1 All nodes are previously initialized with open as 0 4: for all nodes v in u.customers do 5: if v.open = 1 then 6: u.customers u.customers - {v} Removes the link to eliminate the cycle 7: v.providers v.providers - {u} (from both lists) 8: else if v.visited = 0 then 9: RemoveCycles(v) 10: end if 11: end for 12: u.open 0 node u is no longer in the path now 13: end procedure The main procedure of the algorithm can then be described as follows: Once an AS is added to the path, it is marked with visited=1 (line 2), since it has now been explored in the course of the algorithm, and with open=1 (line 3), since it belongs to the current path. Afterwards, its set of customers is swept sequentially and, if one of them already belongs to the current path, then this forms a provider-customer cycle and, consequently, the p2c link which connects the AS being explored to this customer is removed from the network (lines 6 and 7). Customers with visited=0 are added to the path and all subsequent p2c links are then tested for provider-customer cycles by recursively executing the algorithm (line 9). Customers with visited=1 and open=0 are disregarded because all provider-customer cycles which could include them have already been removed. Figure 3.1 presents an 21

38 execution example of the algorithm. Figure 3.1: Execution example of the RemoveCycles algorithm In an attempt to preferentially remove the p2c links which connect ASs at the bottom of the hierarchy (high tier ASs) to ASs near the top (low tier ASs), we initially execute the algorithm using the tier 1 ASs as the first ASs of the AS path being tested, and then proceed to execute it for all ASs which still have visited=0 until all ASs have been explored. Policy-Connectedness Another important characteristic we must ensure is policy-connectedness. Our network will be policyconnected if each AS can reach each of the other ASs through either a provider path, a peer path or a customer path. Making use of the tiered hierarchy of the Internet, Theorem 1 provides one way of ensuring this: Theorem 1. A tiered network with no provider-customer cycles is policy-connected iff there is a p2p link connecting every possible pair of tier 1 ASs. Proof. The proof that having all tier 1 ASs establish a peer-peer relationship is a necessary condition is straightforward: since tier 1 ASs have no providers, the only way to connect two tier 1 ASs is through a peer path consisting of a single p2p link. By definition, tier 1 ASs should be the only ASs in the network hierarchy with no providers. It follows, therefore, that there is at least one customer path connecting a tier 1 AS to any AS which is not tier 1. 22

39 This can be easily proven by incrementally constructing a path made exclusively of c2p links starting at any arbitrary AS which is not tier 1, leading us to the realization that, since the number of ASs in the network is finite and there are no provider-customer cycles, we will eventually reach an AS from which there is no further c2p link to explore - an AS with no providers (i.e., a tier 1 AS). Taking this into consideration, one may prove that having all tier 1 ASs establish a peer-peer connection between them is also a sufficient condition for policy-connectedness. Consider any two arbitrary ASs X and Y. If both of them are tier 1 ASs, then they are connected through a single link peer path. If we assume that only AS X is a tier 1 AS, then AS X will have either a customer path to reach AS Y, as stated in the previous paragraph, or it would have a peer path to reach it, by having a peer-peer relationship with a tier 1 AS which reaches AS Y through a customer path (the situation is equivalent if AS Y were the tier 1 AS instead of AS X). Let us now assume that neither AS X nor AS Y are tier 1. If AS X is a direct or indirect customer of AS Y, then there is a provider path connecting the source-destination pair X-Y (the situation is equivalent if AS Y is the direct or indirect customer of AS X). If this is not the case, then by knowing there is a customer path connecting a tier 1 AS (which we will call T X ) to AS X, and that there is a customer path connecting a tier 1 AS (which we will call T Y ) to AS Y, then we can be sure there is at least one provider path connecting ASs X and Y. If T X =T Y, this path consists of the concatenation of paths X T X and T Y Y. If T X T Y, then the condition that all pairs of tier 1 ASs have a peer-peer agreement between them guarantees the existence of a provider path connecting X to Y through the concatenation of the path X T X, the p2p link T X -T Y and the path T Y Y. Therefore, this is enough to ensure a policy-connected network. After adding p2p links between the tier 1 ASs which did not have a peer-peer agreement, we proceed to ensure that the topology satisfies the hierarchical structure in which only the tier 1 ASs have no providers and all remaining ASs are reachable from a tier 1 AS through a customer path. Since all provider-customer cycles have already been eliminated, this can be done by executing a simple loop in which ASs without providers that were not classified as tier 1 ASs are removed from the network, as well as all the links in which they were one of the extremities. When the only ASs without providers in the network are the tier 1 ASs, the loop terminates Paths and Routes Elected Routes among pairs of ASs In order to take into consideration what could be the most common routes for prefixes in the FIBs of the Internet, we start by analyzing the existing customer, peer and provider paths in the network connecting every possible source-destination AS pair. To this end, we have devised an algorithm called DetermineElectedRoutes which calculates what is the elected route at every possible source AS to reach prefixes announced at every other AS. This is done by analyzing the paths of the network, taking into consideration that customer paths are preferred to peer paths, and peer paths are preferred to provider paths. 23

40 The DetermineElectedRoutes algorithm adds a new data structure: now, at each node, there is a vector called ElectedRoutes in which each of its entries contains the elected route to reach prefixes generated at the AS whose numeric ID was used as index. In order to be able to use the AS s number as index, we have opted once more for vectors of 2 16 entries. However, allocating this vector for each of the ASs proved itself to be too heavy memory-wise, overflowing the temporary storage capacity that was being used. Therefore, this vector is not allocated at stubs without peers (ASs that have no customers and no peers), since we know beforehand that, at these ASs, the preferred path to reach each of the other ASs will be a provider path, as no other type of path exists. This measure led to a significant reduction in the required memory space, allowing for a standard execution of the algorithm without any memory allocation problems. The main steps of DetermineElectedRoutes are described below, and they are executed at every source AS (other than stubs without peers): 1. We initially fill each of the entries in the source AS s ElectedRoutes vector with the attribute P (provider route). This is done because, since the network is policy-connected, then there will necessarily be a path connecting this AS to each of the other ASs which, at worst, will be a provider path. 2. A DFS is now executed starting from the AS being analyzed, in which only the p2c links are explored. Each of the ASs encountered during the DFS will be a direct or indirect customer of the source AS, and therefore for each of them the associated entry in the ElectedRoutes vector is filled with the route attribute C (customer route). 3. Afterwards, we execute a DFS of p2c links starting at each of the source AS s peers, thereby encountering all ASs which are direct or indirect customers of one of the peers of the source AS. For each of these encountered ASs, we initially check to see if its associated entry in the source s ElectedRoutes vector is not already filled with C (since customer routes are preferred to peer routes). If it is not, then we fill it with the attribute R (peer route). Figure 3.2 shows an illustrative example of an execution of this algorithm. Figure 3.2: Execution example of the DetermineElectedRoutes algorithm 24

41 Forwarding Neighbors Another aspect which we deemed relevant is the number of forwarding neighbors for each route. This could give us some insight as to how frequently does the need to consider multiple forwarding neighbors become relevant. Let us consider an arbitrary node u and its elected route to reach a prefix p. Considering each of the possible routes in a Gao-Rexford model, we can establish the following statements regarding the forwarding neighbors for p: R[u;p] = C (customer routes) : A customer route is a route learned from a customer. Considering that the Gao-Rexford export policy states that a node will only export to its providers their customer routes, this means that the customers of node u will only export a route for p to node u if their own elected route for p is a customer route. Therefore, the forwarding neighbors for p will be customer nodes v in which R[v;p] = C. R[u;p] = R (peer routes) : A peer route is a route learned from a peer. Considering that the Gao-Rexford export policy states that a node will only export to its peers their customer routes, this means that the peers of node u will only export a route for p to node u if their own elected route for p is a customer route. Therefore, the forwarding neighbors for p will be peer nodes v in which R[v;p] = C. R[u;p] = P (provider routes) : A provider route is a route learned from a provider. Considering that the Gao-Rexford export policy states that a node will export to its customers all of their routes, this means that the providers of node u will export a route for p to node u if they possess an elected route for p. Therefore, the forwarding neighbors for p will be provider nodes v in which R[v;p]. Within the context of a policy-connected network, we know by definition that there will be no node v in the network for which R[v;p] =, for any prefix p. Considering the forwarding neighbors for provider routes, this leads us to the following result: in a policy-connected network, all of an AS s providers are forwarding neighbors for its provider routes. Hence, the procedure done for obtaining the set of forwarding neighbors associated to each of the routes of an AS is straightforward: the forwarding neighbors of a customer route are the customers of the AS which also reach the destination through a customer route, the forwarding neighbors of a peer route are the peers of the AS which reach the destination through a customer route, and the forwarding neighbors of a provider route are all the providers of the AS. Accounting for AS Path Length Since the AS PATH attribute is a common tiebreaker used between BGP routes which have the same Gao-Rexford route attribute, we have decided to analyze the AS path lengths across the Internet for customer, peer and provider paths. Given that the Gao Rexford routing policy with consideration for the path length is not an isotone policy, this means that, for an arbitrary prefix p, the elected route at a given node u R[u;p] may not 25

42 necessarily reproduce the optimal path in the network from node u to the destination for prefix p, i.e. there may exist a path in the network (u n,..., u 1, u 0 ) from u = u n to t p = u 0 such that the route value α learned from it, α = L[u n,u n-1 ](L[u n-1,u n-2 ](...L[u 1,u 0 ](R[t p ;p])...)), is preferred to the value of R[u;p]. Figure 3.3: Lack of isotonicity in Gao Rexford with path length routing policy Consider the example in Figure 3.3. As we can see, AS B has a customer route through AS C to reach prefixes at AS F, which it prefers to the provider route through AS E. Therefore, AS B will only export to AS A its customer route. However, if the provider route of AS B were to be exported to AS A as well, it would be preferred to the customer route of AS B, since they are both converted to provider routes at AS A, and the provider route through AS E possesses a smaller number of links (3) comparatively to the route through AS C (4). This example shows how, for the Gao Rexford with path length and other routing policies which are not isotone, a prefix s elected route will not necessarily represent the most preferred route pertaining to it. Because of this routing policy s lack of isotonicity, the algorithm for calculating the elected routes at each AS must not be an optimal paths algorithm. The approach we now use for every possible destination AS is to compute the route exports that occur after the AS generates a route pertaining to a locally held prefix, taking into consideration that every AS will only export to its neighbors the most preferred of all the routes that it learns. The algorithm is run for every destination AS and its main data structure is an ordered list. Each element of this list contains three fields: a source AS, a Gao-Rexford route attribute and the number of links of the AS path. An element of this list represents a route received at the AS in the field source AS which was there calculated as a route whose attributes are the remaining fields. The list is ordered from the most preferred routes, at the beginning of the list, to the least preferred routes, at the end of the list. Another important element for this algorithm is the auxiliary variable visited present at each AS. Since an AS may receive various routes pertaining to a given destination, we may encounter it as a source AS more than once while sweeping through the ordered list. Therefore, we use the variable visited to distinguish ASs which already received their elected route (visited = 1) from those which have not (visited = 0). Since routes are now pairs of attributes (k,n), the vector ElectedRoutes at each AS now possesses 26

43 two values at each entry: one for the Gao-Rexford attribute, and one for the number of links. At ASs which are stubs without peers, this vector is not allocated, since we know beforehand that they will all be provider routes. At these ASs, we compute the number of links of the elected route for a given destination by determining which of their providers elected routes for that destination has the smallest number of links, and then increment this value. The algorithm for calculating the elected routes among all AS pairs with consideration for the AS path length is presented as a pseudocode in Algorithm 2. We refer to the macro AddToOrderedList(v,k,n) as adding a new element to the ordered list with the fields v, k and n as the source AS, the Gao-Rexford route attribute and the number of links of the AS path, respectively, and refer to the macro Remove- FirstElementOrderedList() as removing the first element of the ordered list (the one with the most preferred route), while attributing its fields values to local variables. Algorithm 2 Algorithm for detecting all routes in a Gao-Rexford with path length routing policy 1: procedure ELECTEDROUTESWASPATH(dst) dst is the destination AS currently being tested 2: for all nodes v in V do 3: v.visited 0 Initialization (does not include stubs without peers) 4: end for 5: AddToOrderedList(dst,C,0) 6: while OrderedList not empty do 7: (u,k,n) RemoveFirstElementOrderedList() 8: if u.visited=0 then 9: u.visited 1 10: u.electedroutes[dst] (k,n) 11: if k = C then Only C routes are exported to peers and providers 12: for all nodes v in u.providers do 13: AddToOrderedList(v,C,n+1) 14: end for 15: for all nodes v in u.peers do 16: AddToOrderedList(v,R,n+1) 17: end for 18: end if 19: for all nodes v in u.customers do 20: AddToOrderedList(v,P,n+1) 21: end for 22: end if 23: end while 24: end procedure The correctness of this algorithm is centered on the ordered list. Accounting for the preference ordering on which the list of routes is founded upon, the first time we encounter a source AS in this list, we know already that this route will be its elected route for prefixes at the destination AS we are analyzing. Based on the operation table for this routing policy, presented in Table 2.3, we can make the following observations for each route: Customer Route with N links - An AS with this elected route will have learned it from a customer which elects a customer route with N-1 links (preferred to a customer route with N links). Peer Route with N links - An AS with this elected route will have learned it from a peer which elects a customer route with N-1 links (preferred to a peer route with N links). 27

44 Provider Route with N links - An AS with this elected route will have learned it from a provider which elects either a provider route with N-1 links, a peer route with N-1 links or a customer route with N-1 links (all of which are preferred to a provider route with N links). We can therefore conclude that an AS s elected route is supported on the elected route of one of its neighbors which is actually preferred to the AS s elected route. Therefore, by sweeping the list from the most preferred routes to the least preferred routes, we know that, once we encounter the first route learned by an AS, it will be its elected route, since all of the routes which are preferred to it have already been swept beforehand. It should be noted that the algorithm starts at the destination AS, which elects the most preferred of all routes (customer route with 0 links), for this reason. An execution example of the algorithm is given in Figure 3.4. Figure 3.4: Execution example of the ElectedRouteswASPath algorithm Prefixes and Addresses The representation of the set of IPv4 prefixes which are announced on the Internet was done through a Tree of Announced Prefixes. The Trie representation was discarded because it would add nodes to the tree which are unnecessary for the analysis, thereby increasing the necessary memory space to store it, and because the identification of parent-child (p,q) pairs is not as straightforward as it is in the Tree of Announced Prefixes. 28

45 Each node in the tree represents an IPv4 prefix which is announced on the network, therefore the data structure of each node contains the prefix it represents, as well as the AS which generates it. Other than that, each node possesses a list of child prefixes and a pointer to the parent prefix. Provider-Allocated Prefixes One of the underlying notions behind the allocation of prefixes in the Internet is that blocks of IP addresses are delegated from providers to customers. This means that an AS s locally held prefixes should be more specific than one of its providers prefixes, or contain no parent prefix. Since each AS that locally holds a prefix announces it to its entire set of neighbors, this implies that the route generated for locally held prefixes is equivalent to a customer route. Therefore, so as to satisfy this prefix delegation from provider to customer in the topology, then for every prefix p, we must have R(t p ;q) = C for all of the child prefixes q of p. In order to aid with this verification, we use the previously determined ElectedRoutes vector at each AS. We sweep through every prefix q in the prefix tree and, for each, we mark them as valid or invalid, regarding whether their announcement satisfies the aforementioned condition or not, respectively. This classification is done as follows: If q has no parent, or if it has a parent p such that R(t p ;q) = C, then q is valid. If q has a parent p such that R(t p ;q) C, then q is invalid. Once each prefix is dutifully classified into one of these two categories, we sweep through the entire prefix tree once more and, for every prefix p, we count the number of child prefixes q which were marked as invalid. If this quantity is not nil, we execute one of the following procedures: 1. Remove p - If the number of invalid child prefixes is more than or equal to a heuristically defined threshold LIMPREF, then prefix p is removed from the tree. In this situation, the child prefixes q of p now become children of p s parent prefix o. Therefore, once p is removed, we reclassify all the q prefixes considering their new parent prefix o, and then reexamine prefix o, since its number of invalid child prefixes might change as a consequence of this procedure. 2. Remove invalid q - If the number of invalid child prefixes is less than a heuristically defined threshold LIMPREF, then all the invalid q prefixes are removed from the tree. In this situation, the child prefixes r of each of the q prefixes now become children of p. Therefore, once the invalid q prefixes are removed, we reclassify their children r considering their new parent prefix p, and then reexamine prefix p, since its number of invalid child prefixes might change as a consequence of this procedure. The value of LIMPREF is configurable, and was heuristically set at each prefix to 66% of its number of child prefixes, since this was the approximate value to which the amount of removed prefixes was minimal. 29

46 3.3 Statistics and General Analysis AS Topology and Hierarchy Once we have established the tiered hierarchy of the topology, removed all provider-customer cycles, ensured policy-connectedness and removed the prefixes violating the delegation from providers to customers, a few statistics concerning the topology were extracted, which allowed us to take some conclusions on some of its most noteworthy aspects. Quantity in Original File Quantity in Final Model Percentage Maintained Autonomous Systems % Stubs % Customer-Provider Links % Peer-Peer Links % Table 3.2: Topology statistics (ASs and Links) By analyzing, in Table 3.2, the amount of nodes and links in our final graph which were maintained from the original UCLA file, it is fair to assume that our model is a relatively accurate portrayal of the Internet s AS-level topology it represents. We should also take into consideration the fact that the original file has some inaccuracies, including some possible short-life links which may not actually exist, and that it may be incomplete, underestimating peer-peer links and having a limited scope on their BGP data collection [23] [27]. Multi-homing refers to ASs which have more than one direct provider, and it is relatively frequent occurrence in the network. We have found that approximately 51.16% of all modeled ASs are multihomed. Stubs Multi-Homed % Only 1 provider % With peers 3252 (8.82%) Without peers (38.71%) With peers 5014 (13.60%) Without peers (38.87%) Table 3.3: Stub statistics Stubs are a particularly important part of the Internet topology, since they represent a very large portion of all ASs: in our model, about 83.86% of all ASs are stubs. In Table 3.3, we take a closer look at stubs in terms of how many of them are multi-homed and how many have at least one peer-peer relationship. Figure 3.5 shows the distribution of ASs through different tiers. It allows us to see that the vast majority of ASs (97.81%) are at a distance of, at most, three c2p links from a tier 1 AS. Tier 3 ASs are the predominant ones in the network. 30

47 Figure 3.5: Distribution of ASs through tiers Paths and Routes Considering a situation in which each AS announces only one prefix, we have done an analysis on the global elected routes at every AS for each possible destination, so as to have it reflect the predominant existing types of path in the network: customer, peer or provider. The final results are presented in the pie charts of Figure 3.6. The left-side chart accounts for every possible source-destination AS pair, while the right-side chart accounts only for the paths in which the source is not a stub. Considering that all paths starting at a stub will necessarily be either a peer path or a provider path, the weight of customer paths increases in the right-side chart. However, overall, it is evident that the vast majority of the preferred paths in the network connecting pairs of ASs are provider paths, even at non-stub ASs, where it accounts for approximately 70% of all paths. Figure 3.6: Distribution of the elected paths among pairs of ASs Figure 3.7 represents a distribution of the AS Path lengths in the network. In this graph, we present the AS Path lengths for customer, peer and provider paths in distinct columns from the accumulated distribution for all paths. The values are presented in a percentage relative to the total amount of paths (within that type), and for a cumulative distribution, meaning that the total amount of paths at each AS Path length represents the total amount of paths with an AS length which is equal or lower than the presented value. For example, the point (4;40%) for the peer paths distribution means that approximately 40% of the elected peer paths have an AS Path length which is either less than or equal to 4. 31

48 Figure 3.7: Cumulative distribution of AS Path length By analyzing Figure 3.7, we see that, even though the largest AS Path length value found was 35, 90% of all paths have an AS Path length of 7 or less, and about 70% have, at most, an AS Path length of 5. For any type of elected path (customer, peer or provider), more than half of the paths have either 6 or less links. Accounting for some possible inaccuracies in the UCLA file, this allows us to conclude that the vast majority of the preferred paths in the network consist of relatively few links. Table 3.4 shows some additional statistics about the forwarding neighbors associated to the elected routes pertaining to prefixes at every destination AS. Even though, for most routes, the percentage which contains only one forwarding neighbor is relatively high, the average and maximum amount of forwarding neighbors per route indicates that there may exist several of them with a large quantity of forwarding neighbors. Therefore, there is a quite evident multipath potential at various source ASs. Average amount of forwarding neighbors Customer Paths Peer Paths Provider Paths (4.365 for non-stubs) (2.573 for non-stubs) Maximum amount of forwarding neighbors 170 (1 source-destination pair) 130 (2 source-destination pairs) 41 (18448 source-destination pairs) Percentage with only 1 forwarding neighbor 65.60% 63.57% 50.54% Table 3.4: Forwarding Neighbors statistics Prefixes and Addresses After reading through all the prefixes in the original file, we detected a total of distinct prefixes. Out of these prefixes, were initially discarded: 5 prefixes were discarded because they were contained within the range of a private address space. The prefixes considered private are /8, /12 and /16 [28], 32

49 and we discarded them because they are not used for inter-domain traffic data exchange prefixes were discarded because they had a mask longer than /24. Since /24 is the largest allowed mask size on BGP [29], these prefixes are directed at traffic exchange within the same AS prefixes were discarded because they were announced by ASs which do not belong to our final model of the Internet s AS-level topology. Table 3.5 represents the amount of prefixes maintained, after ensuring they all satisfied the conditions for provider-allocated prefixes. As we can see, the total number of prefixes which remained in the final model is , approximately 97.53% of the non-discarded prefixes, which is an indicator that the amount of prefixes which already verify the provider-to-customer prefix delegation in the Internet is extremely high. Of all the prefixes in the original file (including the discarded prefixes), we maintained about 92.46%. Quantity in Original File Quantity in Final Model Percentage Maintained Non-discarded prefixes % Prefixes announced by stubs % Prefixes announced by tier 1 ASs % Table 3.5: Prefix statistics An important aspect to consider while analyzing the announced prefixes is that there is a large quantity of ASs that announce both a prefix and some of its child prefixes, as well as a large number of prefixes which possess no parent. Figure 3.8 shows the distribution of prefixes according to this classification. Figure 3.8: Classification of modeled prefixes Figure 3.9 shows the total distribution of prefixes by their mask sizes. The left-side columns (associated to the left axis) represent the total amount of prefixes in the network of that particular mask size. The right-side columns (associated to the right axis) represent, for each mask size, the ratio between the quantity of announced prefixes and the total amount of prefixes of that mask size which can be announced (which, for a mask size of N, is equal to 2 N ). As we may see from analyzing this figure, prefixes of mask /24 are the vastly predominant ones in the network (about slightly under ), yet a much larger portion of the prefixes of mask /16 have been utilized (slightly under 19%). For each prefix, we counted the amount of modeled less specific prefixes it has and present the results in Figure Besides the half which has no less specific prefix, as previously observed in Figure 3.8, the vast majority of the remaining prefixes (about 36%) have exclusively its parent as the only less specific prefix. 33

50 Figure 3.9: Distribution of prefixes by mask size Figure 3.10: Distribution of prefixes by number of less specific prefixes In order to account for the number of prefixes announced at each AS, the algorithms executed to obtained the statistics presented in this section were rerun with the actual set of announced prefixes in the network, obtained after removing the prefixes that violated the criteria for provider-allocated prefixes. We have found, however, that the final results and statistics were approximately equal to the ones we obtained by considering only source-destination AS pairs. This allows us to assume that the announcement of prefixes is relatively uniform across the ASs of the Internet, i.e. there is no specific AS or set of ASs which locally hold a much larger quantity of prefixes than the remaining ASs. 34