PERFORMANCE EVALUATION OF ROUTE-BASED DISTRIBUTED PACKET FILTERING FOR DDOS PREVENTION IN LARGE-SCALE NETWORKS. A Thesis. Submitted to the Faculty

Transcription

1 PERFORMANCE EVALUATION OF ROUTE-BASED DISTRIBUTED PACKET FILTERING FOR DDOS PREVENTION IN LARGE-SCALE NETWORKS A Thesis Submitted to the Faculty of Purdue University by HyoJeong Kim In Partial Fulfillment of the Requirements for the Degree of Master of Science December 2003

2 ii ACKNOWLEDGMENTS I would like to thank my advisor Professor Kihong Park for his persistent guidance from technical details to mentoring. His keen criticism on my research has improved my attitude exploring science, and his earnest devotion to science has been a source of my motivation. I present my gratitude to my friends and colleagues at Network Systems Lab. In particular, I am grateful to my friend Bhagya for her valuable feedback on my research; I remember enjoyable nights we spent together facing deadlines. I would like to thank my friend Humayun for his help during my study which include numerous discussions on the subject, implementation of the design, and proof-reading of my thesis. I would also like to thank Ali for his participation in protocol design; I thank Yan for his participation in implementation. I am grateful to Sunwoong who has carefully examined my thesis and provided me vaulable comments. Finally, I thank to my parents and brothers for their life-long love and support. I also thank to my friends in Korea, especially Eun-Ju and Seung-Hyub who have given me warm encouragements throughout my study.

3 iii TABLE OF CONTENTS Page LIST OF TABLES vi LIST OF FIGURES vii ABSTRACT x 1 Introduction Motivation Objective Technical Challenges Contribution Related Work DoS Attacks and Prevention Mechanisms Methods for Computing Source Reachability Scalable Network Simulation Power-Law Network Topology Route-based Distributed Packet Filtering Protocol Overview of Route-based Distributed Packet Filtering Protocol Design Issues Filter Look-up Filter Update BGP and Its Extension Route-based DPF Protocol Architecture DPF-lookup Protocol Semi-maximal Filter Table BGP Extension

4 iv Page DPF-update Protocol Improvement of DPF-update Protocol for Fault-tolerance Performance Evaluation of Route-based DPF Protocol Overall Objective Performance Measures Stability Safety Violation Staleness Containment Traceback Experimental Setup Stability Safety Violation Staleness Containment Traceback Dynamic DPF Simulator Overview of DaSSFNet DaSSFNet-based Parallel Network Simulation Environment Description Automatic Model Configuration and Partitioning Measurement Framework Protocol Modeling Large-scale Network Simulation Experimental Setup System Configuration Benchmark Topologies Simulation Setup

5 v Page 6.2 Performance and Utility of Comprehensive Measurement Subsystem Methodology Memory Requirement Monitoring Memory Consumption by Tables Memory Consumption by Messages Memory Consumption and Counting of Major Kernel Events CPU Load Monitoring Communication Cost Scalability of Partitioning Completion Time Balanced Memory Offloading Conclusion and Future Work LIST OF REFERENCES

6 vi LIST OF TABLES Table Page 6.1 Statistics of the benchmark topologies Parameter settings for TCP

7 vii LIST OF FIGURES Figure Page 3.1 Illustration of route asymmetry Protocol stack of route-based DPF protocol Illustration of BGP Extension mechanism BGP-REFLECT message format BGP routing stability as a function of simulation time Consistency of filter tables as a function of simulation time Safety violation as a function of simulation time Staleness as a function of simulation time Containment as a function of simulation time Traceback as a function of resolution Traceback as a function of simulation time A simple network specification in DML, which consists of one router Mapping of partition groups onto distributed machines A DML snippet of a point-to-point link Network protocol models in the Dynamic DPF Simulator System architecture of the Dynamic DPF Simulator A sample Meta-DML input file Growth of AS-level Internet graph node AS-level Internet graph Pseudo code of phase Pseudo code of phase Local and remote message passing procedures A DML snippet for global measurement A DML snippet for local measurement at IP

8 viii Figure Page 5.14 An illustration of a peering relationship between one border router in AS 1 and another in AS The architecture of BGP protocol model with its tables The class hierachy of BGP message types The architecture of BGP protocol model with its timers A DML snippet of BGP-4 model Interaction of BGP and DPF-update A DML snippet of DPF-update model A DML snippet of DPF-lookup model A DML snippet of UDP model A DML snippet of CBR traffic generator A DML snippet of CBR massive attacker Mechanism of ShutDown model A DML snippet of ShutDown model Hardware configuration of a Linux cluster used for AS-level benchmarking Power-law connectivity of the benchmark topologies Memory consumption as a function of simulation time. (a)m Memory consumption is classifies into three categories tables, messages, and kernel events. (b) The categories are further subdivided into fine granular components Memory consumption by protocol tables. BGP Adj-RIB-In, BGP Loc- RIB, and IP Memory consumption by protocol messages. BGP, TCP Send buffer, TCP Receive buffer, and IP Major types of KernelEvent objects. (a) shows cumulative counts of KernelEvent object creation as a function of simulation time for the major types. (b) shows total memory consumption by KernelEvent objects and memory consumption by major types of KernelEvent objects as a function of simulation time CPU load distribution over 16 Linux workstations

9 ix Figure Page 6.8 (a) Completion time as a function of parallelism for different benchmark graphs. (b) Completion time as a function of problem sie for 16, 20, and 24 machines (a) Total memory watermark as a function of parallelism for different benchmark topologies. (b) Average and maximum memory watermark as a function of parallelism for different benchmark graphs (a) Total memory watermark as a function of problem size for 16, 20, and 24 machines. (b) Average and maximum memory watermark as a function of problem size for 16, 20, and 24 machines

10 x ABSTRACT Kim, HyoJeong. M.S., Purdue University, December, Performance Evaluation of Route-based Distributed Packet Filtering for DDoS Prevention in Large-scale Networks. Major Professor: Kihong Park. This thesis studies performance evaluation of route-based distributed packet filtering (DPF) for spoofed distributed denial of service (DDoS) attack prevention in large-scale networks under dynamic network conditions. Our contribution is threefold. We design and implement a route-based DPF protocol which computes routebased filter tables dynamically in the presence of IP (Internet Protocol) routing table updates governed by BGP (Border Gateway Protocol), Internet s inter-domain routing protocol. By introducing an additional signalling message type to BGP, our solution discovers source reachability information despite the destination-based and policy-based characteristics of BGP that is prone to generating asymmetric routes. We evaluate proactive protection performance of route-based DPF under dynamic network conditions including node failures and resulting transient system states. Benchmarking is carried out in large-scale Internet measurement topologies where we show that route-based DPF is robust and effective with respect to both proactive (containment) and reactive (traceback) performance. To facilitate large-scale simulation-based DDoS performance evaluation, we built the Dynamic DPF Simulator as an extension of DaSSFNet. By incorporating automated network configuration, partitioning, and run-time measurement and monitoring, we show that scalable network simulation is effected by enabling efficient memory, CPU, and communication load balancing in workstation clusters.

11 1 1 INTRODUCTION 1.1 Motivation By clogging the Internet, denial of service (DoS) attack impedes availability of resources and services. Severity and prevalence of DoS attacks is increasing, and their consequent malfunctions are experienced by the Internet population at large. Moore et al. [1] observed that more than 12,000 denial of service attacks occurred against 5,000 distinct targets during a three-week period. Whereas past incidents mostly targeted commercial web sites, universities, and organizations, more recent attacks have targeted the network infrastructure such as major root DNS servers [2 5]. In addition, Internet worms self-replicating malicious code have been used as an agent for launching massive distributed DoS (DDoS) attacks [6]. Route-based distributed packet filtering (DPF) [7] is a recently advanced solution that provides proactive and reactive protection against spoofed DDoS attack when the source address of attack packets is forged. Given the routing information of the Internet, route-based DPF at strategic border routers inspects the source address of an incoming IP packet. If its source address turns out to be valid, i.e., unspoofed, with respect to the routing information, it forwards the packet to IP for routing. Otherwise, it discards the packet proactively. As many DDoS attacks use IP spoofing to hinder source identification, proactive prevention guards the core and end points of the network from attack. In the case when spoofed DDoS attack succeeds at penetrating the proactive shield, route-based DPF s reactive protection localizes its physical source within a few sites. Route-based DPF has been evaluated in large but static network environments assuming availability of relevant routing information, and shown to be effective. However, a protocol for calculating route-based filter

12 2 tables has been missing, including its effectiveness and robustness in the presence of dynamic route changes. 1.2 Objective The objective of this thesis in three fold. First, we design and implement a routebased DPF protocol for computing valid source address information. Route-based DPF filters at distributed filter sites are updated in the presence of dynamic route changes caused by BGP, Internet s inter-domain routing protocol. Second, we carry out performance evaluation of the route-based DPF protocol under dynamic network conditions, including system failures. We evaluate fault-tolerance of route-based DPF in large-scale autonomous system (AS)-level Internet measurement topologies. Third, in order to perform large-scale AS-level Internet benchmarking, we built a scalable simulation environment extending DaSSFNet, a distributed simulation platform for workstation clusters. We perform comprehensive simulation benchmarks to determine scalability in large-scale networks. 1.3 Technical Challenges Route-based DPF has to infer source reachability from inter-domain routing information in order to compute filter tables at distributed filter sites. However, IP routing is based on destination reachability. When a packet arrives at a router, the router is interested in only the destination address of the packet, not its source address. Hence, existing routing protocols, in particular, BGP (Border Gateway Protocol), do not provide source reachability information. Moreover, due to asymmetry of IP routing, we cannot infer source reachability directly from the destination reachability information. That is, we cannot ascertain that the path from a source node to a destination node is the same as from the destination to the source in reverse order. In addition, BGP uses administrative policies for determining routes that are not necessarily of technical nature. As a result, routing information received at a border router

13 3 may be biased by its upstream routers. Thus, a major challenge when implementing route-based DPF to the global IP Internet is to infer source reachability from BGP. For large-scale performance evaluation involving dynamic network simulation, a scalable simulation environment that is capable of providing necessary system support including monitoring and measurement, memory management, and partitioning is crucial. Since we are aiming to carry out performance evaluation of route-based DPF on large-scale AS-level Internet measurement topologies, our environment must support up to 12,000- node/26,000- edge networks that may contain 144,000,000 routing entries. A critical problem in scalable network simulation is memory consumption stemming from both static requirements such as routing tables and dynamic requirements such as protocol messages. In AS-level Internet simulation, each node represents an AS where each node is modelled as a single border router. Thus, in addition to IP routing tables there exists the memory requirement of BGP s internal tables. For our performance evaluation, route-based DPF filter tables need to be maintained along side IP and BGP tables. Hence, we need to build a scalable simulation environment for AS-level Internet simulations supporting large-scale memory requirements which achieving parallel speed-up. 1.4 Contribution We have designed a route-based DPF protocol, which updates filter tables dynamically in the presence of BGP which changes IP tables. The route-based DPF protocol relies on a BGP Extension that allows source reachability information to be deduced from routing related signalling messages. Based on observable information obtained from the BGP Extension, the route-based DPF protocol at filter-deployed border routers infers validity of source addresses for each interface and updates filter tables accordingly. In addition, a counter-based table design facilitates incremental filter update in tandem with incremental BGP routing update.

14 4 We have implemented the route-based DPF protocol in a process-oriented simulation environment, DaSSFNet [8]. The process-oriented abstraction allows simulation models to be almost as comprehensive as actual system-level protocol implementations. Since the filter update component of the route-based DPF protocol resides above the transport layer as in BGP, it interacts with modules in the lower layers of the protocol stack via BSD-like socket interface. Thus, the simulation model is built independently from its underlying simulation environment. This working protocol model and design decisions we made in the implementation phase is useful for future system building work in network processor platforms. We have carried out performance evaluation in large-scale AS-level Internet measurement topologies [9]. Our performance evaluation results on fault-tolerant protection of route-based DPF in Internet measurement topologies is useful for assessing its effectiveness and robustness in dynamic environments. Moreover, the scalable simulation environment serves as a base for researching filter placement issues as well as more thorough performance evaluation with respect to infrastructure attack against route-based DPF. In the context of distributed simulation, partitioning of a given simulation topology affects the simulation s completion time. Memory requirement imbalance may cause swapping in the virtual memory management system, resulting in increased execution time. We have devised a new partitioning algorithm for power-law network topologies, characteristic of Internet AS measurement graphs, which achieves balanced distribution of memory requirement as well as utilization of CPU resources.

15 5 2 RELATED WORK In the following, we review related work across key areas relevant to the thesis. 2.1 DoS Attacks and Prevention Mechanisms Denial of Service (DoS) attacks overburden a target system or network by demanding more resources than they can provide. As presented in classical types of DoS attacks [2,10,11], resources can be network bandwidth, process, or network connections. Typically network-based distributed DoS (DDoS) attack forges the source address of DoS attack packets [12] called spoofing. Although some recent attacks used agents to generate DoS attack traffic with unspoofed source addresses, initial attack packets for launching remote agents employ IP source address spoofing [13]. Recent incidents reveal that the Internet infrastructure such as core routers or Domain Name Servers (DNS) have become a target of DoS attack [5]. Research on source identification also called IP traceback [14] have looked at ways to localize the physical source of attack traffic. Manual, recursive link testing [15], audit trails approach which use traffic logs at routers or gateways [16], and packet-based traceback mechanisms [14, 17 20] belong to source identification mechanisms. Contrary to route-based DPF, these approaches are inherently reactive dmage must occur before traceback can be initiated and cannot provide proactive protection where attack packets are discarded before they reach a victim. Packet filtering at ingress or egress points of a domain prevent DDoS attack traffic proactively [21 23]. For example, a firewall at an egress points can check the source address of an exiting IP packets. If the source address of an arriving packet is not from the address space of its domain, the firewall can discard it determining that the packet is spoofed. A limitation of egress filtering is that with partial deployment

16 6 there are still too many domains from which spoofed DDoS attack can be initiated by compromised hosts. Ingress filtering only works at transit providers vis-à-vis stub ASes which limits its effectiveness. In this sense, route-based DPF can be viewed as a generalization of ingress filtering. Recently, Mayday [24] has demonstrated a key aspect of route-based DPF, where it functions as non-cryptographic authentication mechanism. Mayday applied source address (or port number) based network layer packet filtering as a light weight authentication mechanism for protecting servers of Secure Overlay Services (SOS) [25]. In this framework, router-based packet filters are deployed around the server, and, by inspecting source address (or port number) of every incoming packet, they forward only valid packets to SOS servers. 2.2 Methods for Computing Source Reachability A major difficulty in route-based DPF table calculation lies in inference of source reachability from destination-based IP routing, and in the presence of routing asymmetry. Whereas our approach extends BGP-4 introducing an additional signalling message type, SAVE [26] and OPCA [27] address the same problem in different domains with different design principles. The objective of SAVE [26] is to enforce IP packets to carry valid source addresses at a router-level network. Since SAVE is designed to be routing protocol independent, it uses additional data structures to maintain source reachability information at each SAVE router and is generic in nature where the main contribution lies in performing efficient incremental update. The key technical challenge with respect to protocol implementation is to realize routebased DPF semi-maximal or maximal filtering as a minimal footprint companion protocol of an underlying routing system. In the case of the route-based DPF protocol advanced in this thesis, this is done in the context of BGP utilizing its AS-PATH message information. OPCA [27] proposes an overlay network on top of BGP as a policy control architecture, which is applied for faster route fail-over and inbound

17 7 traffic load balancing. In the case of OPCA, a central repository maintains inter-as relationships and Internet hierarchy that is used to enhance routing performance. 2.3 Scalable Network Simulation Simulating large-scale networks such as the Internet has been a challenging problem due to the size and complexity of the global IP Internet [28, 29]. ns-2 [30], a packet-level discrete-event network simulator, has been used widely for research including TCP congestion control, multicasting, and wireless networks. Although ns-2 is well-suited for small scale simulation, due to memory requirement of routing tables, messages, and timer events in large-scale networks it is ill-suited for scalable network simulation. Moreover, the lack of process-oriented abstraction hinders flexible and accurate evaluation of dynamic protocols including those pertaining to dynamic routing. To tackle these challenges and limitations, several studies have looked into a fluid-based simulation approach [31 33] which represents network traffic as a fluid flow in such a way simulator only keeps track of changes in rates of network flow without maintaining individual packet events. A number of projects have studied parallel/distributed simulation techniques in order to utilize parallel/distributed resources in multi-processor and distributed memory environments for large-scale simulation [8, 34 36]. A principle focus of these studies has been on synchronization and parallel speed-up issues. Recent work proposed techniques aimed at enabling large-scale network simulation by using memory resource thriftily, emphasizing that routing-related information inherently requires large amount of memory that can become a bottleneck [37,38]. Our environment, the Dynamic DPF Simulator, facilitates large-scale network simulation by extending DaSSFNet [8], a C++ based realization of SSF for workstation clusters and multi-processor systems. SSF is a process-oriented discrete event simulation framework aimed at flexible, accurate, and efficient simulation. Adopt-

18 8 ing DaSSF s scalable simulation environment together with a process-oriented world view [39], DaSSFNet provides a network simulation infrastructure which is amenable to full-fledged network protocol implementation on commodity workstation/pc clusters. The existing tools, including DaSSFNet, provide a parallel simulation kernel capable of efficient synchronization and exporting standardized APIs(e.g., SSFNet s object classes and a BSP-like socket API), however, they lack tools for automated network configuration, partitioning, and efficient dynamic monitoring. A key problem is large-scale topology partitioning which has a dominant influence on performance with respect to memory, CPU, and communication load balancing. Our system building work addresses these issues. 2.4 Power-Law Network Topology Recent measurements of various information networks, including Internet domain networks [40], the World Wide Web [41, 42], metabolic networks [43], and a variety of social networks [44 46] have shown that connectivity in these networks follow a distinct pattern: most are connected to a few, but a few are connected to many. These networks are sometimes collectively referred to as power-law networks as there is a power-law relation between the degree and frequency of nodes of that degree: Pr{deg(u) = k} k β In [7] the impact of power-law network connectivity on route-based DPF s protection performance has been studied. It is shown that power-law connectivity of Internet AS-level topology plays a crucial role in route-based DPF s effectiveness for achieving proactive and reactive while achieving sparse filter placement. Theoretic studies that extend classical random graph theory to power-law graph based on degree sequences include [47].

19 9 3 ROUTE-BASED DISTRIBUTED PACKET FILTERING PROTOCOL Route-based Distributed Packet Filtering (DPF) [7] has been proposed as a proactive and reactive solution for distributed denial of service (DDoS) attacks which use source IP address spoofing. Aiming at Internet autonomous system (AS) level protection against DDoS attack, we have designed a protocol for route-based DPF. This chapter is organized as follows. First, we introduce the concept of routebased DPF for technical background. Next, we discuss several protocol design issues surrounding route-based DPF. This is followed by the protocol specification. Finally, we describe an improvement of the protocol for enhancing fault-tolerance. 3.1 Overview of Route-based Distributed Packet Filtering This section presents the idea of route-based distributed packet filtering (DPF) for technical background, summarizing [7]. First, the concept of route-based packet filtering is described, followed by two filter types maximal and semi-maximal filters. Next, the concept of distribution of route-based packet filters and their synergistic effect are described. Finally, issues regarding filter placement are discussed. Route-based DPF assumes that each node 1 has complete knowledge of routing over the entire network. With this assumption, each node verifies if an incoming IP packet is valid, i.e., non-spoofed, when it arrives through a specific link. If the packet is deemed spoofed from the routing information, the packet is discarded. On the other hand, if the routing information cannot conclusively determine the validity of the source address, the node forwards the packet following IP. 1 A node can be an AS in an AS-level network or a router in a router-level network. We will focus on AS-level network in this section.

20 10 Route-based DPF includes two types of filters maximal and semi-maximal. A filter is a mechanism for determining if a packet is valid or not over a link on which the packet arrives. Given a graph G = (V, E) which represents the Internet AS topology and routing information R, a maximal filter F e at a link e = (u, w) E is defined as 0, if e R(s, t); F e (s, t) = 1, otherwise. Here, R(s, t) represents a set of routes from a source IP address s to a destination IP address t. The output 0 means that a given packet is valid i.e., non-spoofed, and the output 1 means that it is invalid i.e., spoofed. A maximal filter evaluates validity of an incoming packet M(s, t) based on the existence of a path from s to t going through the link e based on R(s, t). Each node maintains a separate table per link for storing the validity flag of incoming packets for all source and destination pairs. This requires in general O(n 2 ) space, where n is the number of nodes in the network. A semi-maximal filter over a link e = (u, w) E is defined as follows. 0, if e R(s, v) for some v V ; ˆF e (s, t) = 1, otherwise. A semi-maximal filter uses the source IP address of a packet for determining its validity. It checks if a link, where a packet M(s, t) arrives, belongs to a routing path from the source IP address s to some destination node, irrespective of the destination IP address t inscribed in the IP header. In this setting, a semi-maximal filter over a link maintains a table which keeps validity information based on source address only. This requires O(n) space. When more than one node of a network enable their filtering functionality, filtering is distributed. The collaborative effect of route-based distributed packet filtering (DPF) is two-fold proactive and reactive. Proactive protection (a.k.a., containment) means that route-based DPF discards spoofed IP packets proactively before they can

21 11 reach their target. Reactive protection (a.k.a., traceback) is in effect when routebased DPF cannot proactively filter out spoofed attack traffic. Reactive protection means that, upon receiving an IP packet spoofed or non-spoofed route-based DPF can localize its physical source. Distributed filter placement involves two issues coverage ratio and selection of nodes for a given coverage ratio. Coverage ratio is defined as the fraction of nodes where filtering is enabled. For a given coverage ratio, the strategy for selecting the filter nodes affects proactive and reactive protection performance. AS-level Internet topology has been shown to exhibit power-law connectivity [40]. This implies that there are a few high degree nodes which are connected to many low degree nodes. Thus, exploiting the power-law nature of AS-level Internet topology, we can reduce the coverage ratio by placing filters at high degree nodes. Conversely, we can apply power-law connectivity information as a strategy for selecting a set of filter nodes while minimizing coverage ratio. 3.2 Protocol Design Issues Sharing a common protocol architecture with IP, the route-based DPF protocol is composed of two major parts filter look-up and filter update. The filter look-up component does line rate packet processing to determine source address validity. As with IP, the filter look-up component functions on the data plane at line rate. On the other hand, the filter update component updates route-based DPF filter tables as routes in the network change dynamically. Similarly to routing protocols such as BGP, the filter update component operates in the control place which occurs at slower time scales. In the next section, we discuss issues in designing filter look-up and filter update components, and present our approach for solving these issues.

22 Filter Look-up Filter look-up resides between the network interface layer and internet layer in the Internet Reference Model. Filter look-up implements packet forwarding/discarding depending on the validity of a packet s source address, i.e., it is a semi-maximal filter. Semi-maximal filtering shows comparable protection with O(n) space requirement to that of maximal filtering, which requires O(n 2 ) space [7]. Note that, in the context of AS-level route-based DPF protocol, we interpret an IP address of an AS node as an IP prefix within the administrative domain of the AS node 2. In other words, we assume that every AS node has a unique non-overlapping IP prefix and we can check the originating AS node from a source IP address by inspecting the prefix of the IP address. We have designed a counter-based semi-maximal filter, which is suitable for incremental filter update. Each entry within the counter-based semi-maximal filter table includes a separate counter, and the counter represents the number of destinations where the link e is traversed from each source IP address. The counter value is interpreted as false if its value is positive. Otherwise, the value is interpreted as true. A straightforward semi-maximal filter design is to maintain a table which consists of entries for all source IP prefixes, where each entry includes a Boolean flag. If it is true 1 in the definition we consider it as invalid (spoofed). Otherwise, we consider it as valid (non-spoofed). If a link e belongs to a set of routes from a source IP address s to some destination IP address, we set the entry of the source s IP prefix as false. Otherwise, the entry has true as its value. Thus, when a packet, whose source IP address is s, arrives, filter look-up will check its validity by performing maximum prefix matching. BGP handles dynamic state changes incrementally, whereas OSPF and RIP do so periodically. The route-based DPF protocol, in the AS context, follows BGP and 2 In reality, some AS node may include more than one IP prefix within its domain. In some cases, more than one AS node may have a common IP prefix within their domains.

23 13 hence its updates are carried out incrementally. The following example shows that route-based DPF must be extended to handle incremental changes. When network state is dynamic, a source IP address, which was valid earlier, may become invalid later. Formally, let R 0 (s, t) be the set of routes from s to t calculated at time 0 and let R 1 (s, t) be the set of routes calculated at time 1. When the state of the network is transient, it is possible that e R 0 (s, t) but e / R 1 (s, t). When a filter entry has false as its flag value, it is difficult to tell whether the in question link is used to reach from source s to the destination t only, or the link is also used to reach to some other destination address. In the first case, the entry should be updated as true, however, in the presence of continual changes and uncertainty this may cause violation of safety if the link is used to reach from source s to some other destination. As mentioned in [7], route-based DPF is safe in the sense that it never discards valid, i.e., non-spoofed packets. In the second case, the entry should remain false. However, if the link is in fact not used by any other source-destination pair, the entry becomes stale. An entry for a source IP address s of a filter table at a link e is stale when a semi-maximal filter cannot filter out spoofed DDoS packets, whose source IP address is forged as s. When the above transient change happens, our counter-based semi-maximal filter decrements the corresponding entry s counter value. If the link e is used to reach from the source IP address s to some other destination, the counter value remains positive. Thus, safety is not violated. If the link is used only for a single (s, t) pair, the counter will reach 0 after decrement. Hence, the entry is prevented from being stale Filter Update As mentioned in [7], the most challenging problem, in the context of designing a filter update protocol, lies in IP focusing on destination reachability, however, not necessarily source reachability. This can induce asymmetry of routing, which makes

24 14 it difficult to infer source reachability. Asymmetry of IP routing is common in interdomain routing where non-technical, administrative policies are applied due to BGP implementing policy routing. In this section, we focus on these challenges. Our approach, a BGP Extension, for solving these challenges is introduced in the next section. For background, let us describe BGP s route selection and advertisement procedure. According to RFC 1771 of BGP-4 [48], Decision Process selects routes for subsequent advertisement by applying policies to route updates stored in Adj-RIB-In (a table containing received route updates per peer). The output of Decision Process is a set of routes to be advertised, and they are stored in Adj-RIB-Out. Decision Process includes three distinct phases Phase 1, Phase 2, and Phase 3. Phase 1 is a step for calculating the degree of preference for each received route update by applying policies. Let us call them update policies. During Phase 2, a best route is selected out of the received route updates for each distinct destination. When there exist more than one candidate route update for a destination with the same preference, a tie-breaking procedure is applied. Each BGP speaker has a unique BGP Identifier, which is set to an IP address assigned to it. When a BGP peering relationship is established, participating BGP speakers exchange their BGP Identifier values. Among routes selected during Phase 2, some routes are chosen for advertisement to peer BGP speakers per policy during Phase 3. Let us call them advertisement policies. As with other routing protocols, BGP focuses on destination reachability, which is sufficient for calculating IP routing tables. Since IP routing relies on destination IP address only, destination reachability is a sufficient condition. A BGP speaker at a destination AS creates and forwards reachability information (i.e., route update) to its neighbors. Once reachability information is sent out, the BGP speaker at the destination AS does not know which route was selected by BGP speakers at source ASes. Each source AS selects a route to the destination AS based on destination reachability information only, without taking into account routing path from the des-

25 15 tination AS back to itself. Moreover, source ASes need not send routing information back to the destination AS. Destination-based IP routing may induce route asymmetry, and routes calculated by BGP can exhibit the same problem. Given a route from a source IP address s to a destination IP address t, routes between s and t are asymmetric if a route from t to s is not the same as one from s to t. Route asymmetry is a characteristic feature of destination-based IP routing, and it has been observed since the mid 1990s both at the AS-level and at the router-level [49]. Figure 3.1 illustrates route symmetry in a simple network. Let us use hop count as the metric in this example. When a routing protocol lets node 1 know that there are two route candidates for reaching node 6 with the same metric 3, node 1 will choose one of them. Similarly, node 6 will select one of two route candidates for reaching node 1. Depending on the routing protocol and local information available, route asymmetry can arise as depicted in Figure 3.1. Figure 3.1. Illustration of route asymmetry.

26 16 Due to route asymmetry, source reachability may not be inferable directly from destination reachability information. In Figure 3.1, given a route from node 1 to node 6, we cannot construe that the route from 6 to 1 is the same route in reverse. On the other hand, link-state routing protocols such as OSPF can detect this asymmetry and infer source reachability using its global knowledge of the entire network. However, BGP does not provide to each node global knowledge for calculating source reachability. Phase 3 of BGP s Decision Process chooses route updates (1) among routes selected during Phase 2, and (2) according to its advertisement policy. In other words, route update information received from one s peer may be biased and restricted by the update policy (in case of 1) and advertisement policy (in case of 2) of the peer. Hence, the current BGP is not suited for inferring source reachability for use in route-based DPF filter table update. To calculate correct filter tables at distributed filter sites, augmentation of BGP or introduction of a new protocol for propagating source reachability information is required. In reality, BGP may not compute correct routing tables. Sometimes, it leads packets to black holes where packets cannot be forwarded any further. Even if this is the case, route-based DPF protocol should infer source reachability from the consistent image of routing tables which the BGP protocol calculates. Hence, both methods are required to interact with BGP for synchronization. The latter requires additional overhead for coordinating with a separate protocol, BGP. We extend BGP for disseminating source reachability information BGP and Its Extension We extend BGP by defining a new message type BGP-REFLECT. A BGP- REFLECT message contains source reachability information, and it is disseminated back to destination ASes, where its corresponding route update message is initiated. BGP-REFLECT includes two internal types ADD and DELETE. In the case when a BGP-REFLECT message carries potential source AS information which becomes

27 17 reachable, its internal type is set as ADD. In the case when it carries source AS information which becomes unreachable, its type is set as DELETE. As BGP presents destination reachability using AS-PATH in BGP route update messages, source reachability information in BGP-REFLECT messages is represented as AS-PATH. A BGP route update message includes AS-PATH as an attribute for a destination IP prefix. AS-PATH, which contains destination reachability information, is a sequence of AS numbers starting with that of the originating AS. As it is propagated from the originating AS, AS numbers are prepended to the AS-PATH attribute in BGP route update messages. Similarly, a BGP-REFLECT message includes source reachability information as part of AS-PATH. The AS number of the source AS is prepended to AS-PATH of selected BGP route updates from a destination AS. Generation of BGP-REFLECT message is triggered during BGP s Decision Process. First, a BGP-REFLECT ADD message is initiated when a route update is selected for a destination IP prefix. This may be caused by a newly-received BGP route update or by expiration of the BGP Hold Timer, which checks liveness of its peer BGP speaker. In the first case, when the newly-received BGP route update is selected as the best route for the destination IP prefix, the BGP-REFLECT ADD message for the new route update is instantiated. In the latter, expiration of BGP Hold Timer initiates selection of another best route for the destination IP prefix. For the newly-chosen route update, BGP-REFLECT ADD message is initiated. Next, BGP-REFLECT DELETE message is created when a route update, which was selected for a destination IP prefix, becomes invalidated. This can be caused by a newly-received route update or by expiration of the BGP Hold Timer. Triggered BGP-REFLECT messages are forwarded back to the destination AS, which originated the corresponding BGP route update message. BGP consults AS- PATH within BGP-REFLECT message to determine where to forward. Once the message reaches the destination AS, it is destroyed. From the received BGP-REFLECT message, the destination AS comes to know source reachability.

28 Route-based DPF Protocol Architecture Based on the discussion in Section 3.2, we define route-based DPF protocol DPF-lookup and DPF-update together with BGP Extension. The DPF-lookup protocol challenges every incoming packet consulting the semi-maximal filter table to determine validity. Filter update is composed of BGP Extension and DPF-update protocol. As mentioned earlier, BGP Extension disseminates source reachability information to the entire network. The DPF-update protocol interprets source reachability from BGP-REFLECT messages received by the BGP Extension, and it updates semi-maximal filter tables according to the information obtained. Figure 3.2 shows the overall architecture of the route-based DPF protocol. As mentioned in Section 3.2, DPF-lookup exists between the network interface layer and internet layer in the Internet Reference Model, and DPF-update operates at the application layer on top of TCP. The DPF-update protocol updates semi-maximal filter tables which are used by the DPF-lookup protocol for filtering at smaller time scales. Figure 3.2. Protocol stack of route-based DPF protocol.

29 DPF-lookup Protocol The DPF-lookup protocol performs filtering, inspecting every incoming IP packet. When a packet arrives through a network interface, DPF-lookup fetches its source IP address. Then, DPF-lookup finds the best matching IP prefix 3 in the semi-maximal filter table corresponding to the network interface. If an entry for a source IP address stores a positive counter value, which means that the packet with the source IP address is valid, it is passed to IP for routing. Otherwise, the packet is discarded Semi-maximal Filter Table The DPF-lookup protocol defines semi-maximal filter tables as protocol components. A semi-maximal filter table contains an entry for each source IP address. For reduction of size (the number of entries) of a semi-maximal filter table, it is recommended to maintain an entry only for valid source IP address. In this case, if there is no matching entry for a source IP address, it implicitly means that the source IP address is invalid. Filtering functionality can be enabled selectively. For each network interface where filtering is enabled, DPF-lookup maintains a separate semi-maximal filter table BGP Extension BGP Extension is an augmentation of BGP-4 to assist the DPF-update protocol calculate semi-maximal filter tables. The main reason for this extension is to overcome route asymmetry by sending source reachability information back to the originating destination AS. For this, a new type of message, BGP-REFLECT, is employed. Figure 3.3 illustrates the BGP Extension mechanism. Let us consider three border routers: 1, 2, and 3; their AS numbers are 1, 2, and 3, respectively. In this example, 3 In this thesis, we assume that each AS has a unique, non-overlapping IP prefix. As mentioned in 3.2, some ASes have more than one IP prefix, and some IP prefixes are shared by more than one ASes. In that case, this table searching method might cause safety violation or staleness problem.

30 20 Figure 3.3. Illustration of BGP Extension mechanism. every border router installs BGP-4 for routing, and the BGP Extension functionality is enabled. First, BGP route updates are propagated to the entire network according to BGP-4. After establishing a TCP connection, BGP at 1 sends a route update for itself to BGP at 2. Here, a route update corresponds to a BGP UPDATE message which includes Network Layer Reachability Information (NLRI) and AS-PATH. NLRI contains a list of IP prefixes of the triggering AS. On receiving the UPDATE message, BGP at 2 passes it to Decision Process, and it is decided as a best route to reach 1. Then NLRI and AS-PATH information are inserted into Loc-RIB (storage for selected route updates) of BGP at 2, and a new BGP UPDATE message is sent to BGP at 3. When a new route update is selected and inserted into Loc-RIB, a BGP-REFLECT message for the route update is created and sent back to the destination. As shown in Figure 3.4, BGP-REFLECT includes IP prefix, AS-PATH, and TYPE as its major fields. In this illustration, BGP-REFLECT ADD messages are created for the newly received route updates. The IP prefix field stores the IP prefix of the source AS. To

31 21 complete the AS-PATH representing source reachability, the initiating AS prepends its AS number to AS-PATH of the received BGP route update. In Figure 3.3, a BGP- REFLECT message from 2 to 1 and another one from 3 to 2 correspond to instances of this case. On receiving a BGP-REFLECT message from its peer, the BGP router forwards the message to its upstream router based on the AS-PATH field of the BGP- REFLECT message. In Figure 3.3, BGP at 2 receives the BGP-REFLECT message initiated by 3, and it forwards the BGP-REFLECT message without any modification. Again, it refers to the AS-PATH field to find out the corresponding upstream router. When BGP-REFLECT messages arrive at destination ASes, they are removed and not forwarded any further. In Figure 3.3, BGP at 1 receives two BGP-REFLECT messages, and they disappear from the network. BGP-REFLECT Message Format As a BGP message type, BGP-REFLECT contains a 19-byte BGP message header. The BGP message header includes a 1-byte Type field. We assign 5 as the type code of the BGP-REFLECT message. Figure 3.4. BGP-REFLECT message format. Figure 3.4 shows the BGP-REFLECT message format. Following is a description of each field: IP prefix length

32 22 This 1-byte unsigned integer field specifies the length of a source IP prefix. The length is represented in bits. IP prefix The source IP prefix is stored in this field. The length of the IP prefix is variable. For this reason, this field is padded with 0 in order to scale its length into a byte unit. AS-PATH length This 1-byte unsigned integer field specifies the AS-PATH length. The length is represented as a count of ASes in the AS-PATH. AS-PATH This field contains a sequence of AS numbers. Each AS number is represented as a 2-byte unsigned integer. The total length of this field becomes two times the AS-PATH length. When a BGP-REFLECT message is created, an initiating source AS s AS number is prepended to the AS-PATH of a given route update, starting with the destination AS. The AS-PATH field must not be modified by forwarding BGP nodes. TYPE This 1-byte unsigned integer field indicates the type of a BGP-REFLECT message ADD or DELETE. The following codes have been defined: Code Symbolic Name 1 ADD 2 DELETE A BGP-REFLECT ADD message indicates a source AS, which becomes reachable to a destination AS via AS-PATH. BGP-REFLECT DELETE message

33 23 indicates a source AS, which becomes unreachable to a destination AS through a given AS-PATH. Details of each message type is described in the following section. Message Types: ADD/DELETE Whenever BGP chooses a new route update for a destination, a BGP-REFLECT ADD message for the route update is generated. When BGP receives a new route update from its peer, Decision Process starts. If it decides to choose the route, a BGP-REFLECT ADD message is initiated. In another instance, when the BGP Hold Timer for a BGP session expires, BGP withdraws all route updates received from the corresponding peer. In this case, BGP tries to find an alternate route update for each destination. For the newly-selected route updates, BGP-REFLECT ADD messages are generated. IP prefix and IP prefix length fields are filled with the value of the source AS. The AS number of this initiating AS is prepended to the AS-PATH of the route update. At the destination AS, the received BGP-REFLECT ADD message conveys that the destination AS becomes reachable from the source AS through the AS-PATH. A BGP-REFLECT DELETE message is generated when BGP invalidates a route update for a destination. BGP-REFLECT DELETE can be triggered in the aforementioned situations. First, when BGP selects a newly-received route update for a destination AS, withdrawing a previously chosen AS-PATH, a BGP-REFLECT DELETE message is generated for the old route update. On the other hand, when the BGP Hold Timer for a BGP session expires, irrespective of the existence of an alternate route, BGP-REFLECT DELETE messages for all route updates are generated. IP prefix, IP prefix length, AS-PATH, and AS-PATH fields are filled in in the same manner as that of the BGP-REFLECT ADD message. At the destination AS, the received BGP-REFLECT DELETE message is interpreted as the destination AS becoming unreachable from the source AS via the specified AS-PATH.

34 24 Reflect Timer In principle, whenever a new route update is selected and inserted into Loc-RIB, a BGP-REFLECT message for the route update should be initiated and sent back to the destination AS. However, for reduction of message complexity, Reflect Timer is recommended to be employed. In this case, BGP-REFLECT messages are transmitted when the reflect timer expires. BGP checks its Loc-RIB to find a route update for which a BGP-REFLECT message has not yet been triggered, and it generates BGP- REFLECT messages for them. In case when more than one route update for the same destination are selected within the Reflect Timer interval, a BGP-REFLECT ADD message for the last route update is initiated. BGP-REFLECT DELETE messages are sent out only for the invalidated route updates, for which BGP-REFLECT ADD messages was initiated. Hence, we can reduce additional BGP-REFLECT ADD/DELETE messages for other invalidated route updates. Depending on the Reflect Timer interval, the number of BGP-REFLECT messages generated is reduced as intended. In addition, it affects timeliness of semi-maximal filter updates across the entire network. Hence, the Reflect Timer has to be tuned carefully. Interface to DPF-update In the context of defining an interface between BGP Extension and DPF-update, we assume that BGP Extension includes a flag for identifying a site where route-based DPF filtering is deployed (a.k.a., filter site). The route-based DPF protocol DPFupdate and DPF-lookup protocols is deployed at filter sites, where the flag is set to true. Hence, BGP can determine whether a router is a filter site or not. BGP Extension provides a reflect buffer as an interface to the DPF-update protocol. In case of a filter site, all received BGP-REFLECT messages are stored in the reflect buffer. Specifically, when a BGP speaker receives a BGP-REFLECT message, it inserts a copy of the message into its reflect buffer before forwarding to the upstream border router.

35 DPF-update Protocol We assume that the DPF-update protocol at each filter site ascertains the set of filter sites deployed. Note that DPF-update can access semi-maximal filter tables by the AS number of peering ASes, since DPF-lookup maintains filter tables indexed by the AS number of neighboring ASes. The DPF-update protocol decodes received BGP-REFLECT messages for calculating semi-maximal filter tables. As mentioned earlier, BGP-REFLECT messages are handed over by BGP Extension via a reflect buffer. Given a BGP-REFLECT message, the DPF-update protocol operates as follows based on its type code: ADD From the AS-PATH of the BGP-REFLECT message, DPF-update fetches its immediate downstream AS number. A corresponding semi-maximal filter table is accessed. IP prefix and IP prefix length information are used to find an entry for the source AS. Then, the counter value for the entry is incremented. Since the filter table keeps only valid source IP addresses for reduction of table size (a property of power-law networks), an entry for a source IP address may not exist. In that case, DPF-update creates an entry and increments its counter value. DELETE DPF-update accesses a corresponding semi-maximal filter table based on the AS-PATH of the BGP-REFLECT message. Using IP prefix and its length, an entry for the source AS is fetched. Since the semantics of the BGP-REFLECT DELETE conveys that the source AS does not use the given AS-PATH to reach this node, DPF-update invalidates its source IP prefix from the filter table by decrementing the counter. In case when the filter table stores only valid source IP addresses, we need one more check if the counter becomes zero after decrement. If this is the case, the entry is deleted.

36 Improvement of DPF-update Protocol for Fault-tolerance The BGP-REFLECT message forwarding scheme relies on BGP peering relationships. BGP forwards received BGP-REFLECT messages as dictated by their AS- PATH field values. This generates a major issue to be considered: fault-tolerance. When an intermediate border router, which belongs to a path between a certain source AS and a destination AS, goes down, BGP at the source AS runs Decision Process and selects another path (if any). As a result of Decision Process, the original path is invalidated and a BGP-REFLECT DELETE message is initiated. BGP running at a border router, which had a connection with the failed border router loses connection. As a result, in the course of BGP-REFLECT message forwarding by BGP, the BGP-REFLECT DELETE message cannot be forwarded any further. In this case, upstream filter sites do not receive this message, and it causes filter tables at the upstream filter sites to contain stale entries. The new forwarding mechanism offloads forwarding responsibility from BGP on DPF-update. DPF-update maintains connection with other filter sites on demand and forwards BGP-REFLECT messages to its upstream filter sites. However, since only BGP can make a decision for generation of BGP-REFLECT messages, it relies in part on BGP. When a BGP-REFLECT message is generated at a non-filter site, it is forwarded to its upstream router in the same manner as the old mechanism. BGP follows the old mechanism until the message reaches the first filter site. Then BGP at the filter site passes the message to DPF-update so that it may start its forwarding mechanism. We need to consider the case when an upstream filter site may not be reachable as well. DPF-update detects failure of its upstream filter site when TCP connection set-up procedure fails. In this case, it forwards the received BGP-REFLECT message to the next available upstream filter site. The connection failure, however, might be caused by temporary network state. In addition, keeping unreachable upstream filter site information is not suitable for scalability of DPF-update. For these reasons, DPF-

37 27 update does not maintain unreachable upstream filter sites information for future use. Whenever a BGP-REFLECT message is received, DPF-update tries to make a connection with its next upstream filter site. This enhances fault-tolerance without adversely affecting scalability of the DPF-update protocol.

38 28

39 29 4 PERFORMANCE EVALUATION OF ROUTE-BASED DPF PROTOCOL As shown in [7], route-based DPF provides significant degree of proactive and reactive protection against spoofed DDoS attacks in a static network environment. The static network environment represents a network environment where there is no failure or variation of network infrastructure. Here, network infrastructure ranges from hardware such as host, router, or link to software such as routing protocol and name server. In this chapter, we carry out performance evaluation of route-based DPF protocol in a dynamic network environment. Contrary to the static network environment, the dynamic network environment encompasses failure or variation of network infrastructure. Route-based DPF is active during transient periods of the network while BGP updates IP routing tables. Time lags in synchronization between the route-based DPF protocol and BGP may lead to performance degradation. Thus, we measure and analyze the effectiveness of the route-based DPF protocol during transient periods. This chapter is organized as follows. First, we introduce performance measures stability, safety violation, staleness, containment, and traceback for route-based DPF. Next, the experimental setup is presented. Finally, results for the performance measures are shown and analyzed in separate sections. 4.1 Overall Objective During transient periods, route-based DPF may contain incorrect information which may cause safety violation or staleness. That is, route-based DPF may discard

40 30 non-spoofed packets (safety violation) or may not be able to discard spoofed attack packets (staleness) when it is safe to do so. The route-based DPF protocol calculates semi-maximal filter tables based on global knowledge of routing. Thus route-based DPF requires consistent routing information with respect to BGP. During transient periods, BGP Extension generates and handles BGP-REFLECT messages as well as BGP UPDATE messages. Accordingly, the route-based DPF protocol interprets messages from the BGP Extension, in order to be consistent with fluctuations of BGP routing itself. In order to capture potential performance degradation during the transient periods, we measure effectiveness of the route-based DPF protocol with respect to safety violation and staleness. We show route-based DPF s protection performance using two major performance measures containment and traceback, as defined in [7]. In addition, we measure stability of BGP s routing table calculation as well as that of the route-based DPF protocol s filter table calculation. Since the route-based DPF protocol relies on the underlying BGP for its update events, stability of BGP routing affects stability of route-based DPF. 4.2 Performance Measures Let G = (V, E) be an undirected graph representing an AS-level Internet topology. BGP computes routes for all pairs of source and destination; let R be the set of computed routes. A route r an element of R is represented as a 3-tuple < node, destination, nexthop >, where node indicates a routing table where the entry belongs to; the others denote the destination and next hop. Similarly, route-based DPF calculates filter tables based on R, and it generates a set of computed filter entries F. A filter table entry f is represented as a 3-tuple < node, link, source >, where node and link identifies a filter table where the entry belongs to; source denotes the IP source address in semi-maximal filtering. The existence of an entry for a source address represents validity. Let E be the set of events which change network topology

41 31 configuration. In reality, E ranges from addition, deletion, or change of BGP routing policies to addition, failure, or configuration change of hardware infrastructure (host, router, or link). In this section, we focus on a single AS node failure cases, which may come from failure of its border router(s). We define a node failure event e as a pair < time, node >, where time denotes the time of failure; node represents the failed node. Let R 0 and F 0 be the initial set of routes and filter tables before an event e in E occurs. An event e triggers BGP s route update procedure. We assume that there exists a steady state of BGP route calculation, where there is no more BGP route update triggered by an event occurrence. 1 With this assumption, let R s and F s be the set of routes and the set of filter entries in a steady state Stability We define distance of two set of routes, R i and R j, with respect to entry or with respect to node in terms of granularity. The distance of R i and R j with respect to entry is defined as a scalar between 0 and 1, and it denotes the fraction of entries which include inconsistent information. Here, inconsistent means either a route entry for a destination does not exist or nexthop information is not the same. Similarly, we define the distance of R i and R j with respect to node as a scalar between 0 and 1, and it denotes the fraction of nodes whose routing tables include at least an inconsistent entry. When two sets of routes contain exactly the same information, both distance with respect to entry and distance with respect to node become 0. Assuming E is a finite set, we can deduce the resulting network topology G and the corresponding set of routes R, which is calculated by Dijkstra s shortest-path algorithm. Let R be the ideal set of routes. Since BGP may converge to a different set in its steady state, R s and R may not be the same. BGP generates a set of routes R t at each time instance t as a result of BGP route update exchange. After BGP 1 However, it is may not be true, in reality. BGP stability problem and its effects on route-based DPF protocol can be examined separately.

42 32 convergence, R t is the same as R s. By plotting distances of each set of routes R t and R with time, we can observe the evolution of BGP routing calculation and stability. Given two sets of filter entries, F i and F j, we define three types of distances for filter table comparison. First, distance of the given sets of filter entries with respect to entry granularity is defined as a scalar between 0 and 1, and denotes the fraction of entries which include inconsistent filter table information. Inconsistent information means either a filter entry for a pair of a link and a source does not exist or validity information is not the same. Next, we define distance with respect to filter granularity as a scalar between 0 and 1, which denotes the fraction of filter tables which include at least an inconsistent entry. Finally, distance with respect to node granularity is defined as a scalar between 0 and 1, and denotes the fraction of nodes whose filter tables include at least one inconsistent entry. BGP routing table calculation fluctuates during transient periods, and it generates sets of routes as it handles BGP route update messages. Subsequently, BGP s route calculation triggers DPF-update s filter calculation. Since route-based DPF s protection performance is based on underlying routing state, it is important DPF-update to calculate filter table consistent with routing at each time instance. Hence, we define consistency of the route-based DPF protocol s filter calculation at each time instance t as a distance of two set of filter entries, F t and F t, where F t denotes calculated filter information; F t represents theoretically determined filter information from routing R t. For a given routing R, we define F, an ideal set of filter entries calculated from the given R.

43 Safety Violation Given a set of routes R i and a set of filter entries F i at a time instance, we can deduce the ideal set of filter entries F i for the given R i. Here, safety violation of F i for the given R i is defined as follows: 0, if Fi F i = ; SV (R i, F i ) = 1, otherwise. The value 0 represents that F i for the given R i is safe. The value 1 represents that safety of F i for the given R i is violated. The intuitive meaning of the safety condition is that filter tables should contain validity of all unspoofed source addresses, so that unspoofed packets are not discarded by route-based DPF. To quantify the degree of safety violation, we examine safety violation in entry, filter, and node granularity. Let nftr be the number of filter tables, which are identified by a pair, node and link. For given F i and R i, safety violation index in entry granularity SV entry is defined as follows: SV entry (R i, F i ) = F i F i nftr V SV entry is a scalar between 0 and 1, and it denotes fraction of entries, which cause safety violation. Safety violation index in filter granularity SV filter is defined as follows: SV filter (R i, F i ) = {(u, e) : f =< u, e, s > (F i F i )} nftr In this definition, (u, e) represents a filter deployed at u over the link e. SV filter is a scalar between 0 and 1, and it represents fraction of filters, which include at least one safety-violating entry. Safety violation index in node granularity SV node is defined as follows: SV node (R i, F i ) = {u : f =< u, e, s > (F i F i )} V

44 34 SV node is a scalar between 0 and 1, and it denotes fraction of nodes, at least one of whose filter table contains at least one safety-violating entry. SV entry provides the most fine-granular metric for safety violation. SV filter and SV node show distribution of safety-violating entries over all filter tables and nodes, respectively Staleness Given a set of routes R i and a set of filter entries F i at a time instance, we can deduce the ideal set of filter entries F i for the given R i. Staleness of F i for the given R i is defined as follows: 0, if F i Fi = ; ST(R i, F i ) = 1, otherwise. The value 0 represents that F i for the given R i is not stale. The value 1 represents that F i for the given R i contains stale information. In other words, at least a entry for an invalid source address is kept in F i. It implies that DDoS attack packets, whose inscribed source address is the invalid one, cannot be discarded. As with safety violation case, the degree of staleness is measured with staleness indexes in entry, filter, and node granularity. For given F i and R i, staleness index in entry granularity ST entry is defined as follows: ST entry (R i, F i ) = F i F i nftr V ST entry is a scalar between 0 and 1, and it denotes fraction of entries, which cause staleness. Staleness index in filter granularity ST filter is defined as follows: ST filter (R i, F i ) = {(u, e) : f =< u, e, s > (F i F i )} nftr

45 35 ST filter is a scalar between 0 and 1, and it represents fraction of filters, which include at least one stale entry. Staleness index in node granularity ST node is defined as follows: ST node (R i, F i ) = {u : f =< u, e, s > (F i F i )} V ST node is a scalar between 0 and 1, and it denotes fraction of nodes, at least one of whose filter table contains at least one stale entry. As with safety violation measures, ST entry provides the most fine-granular metric for staleness. ST filter and ST node indicate distribution of stale entries over all filter tables and nodes, respectively Containment To observe proactive protection (a.k.a. containment) of route-based DPF during transient periods, we use Φ 2 (τ) defined in [7]. Φ 2 (1) is a scalar between 0 and 1, and it denotes the fraction of AS where no attacker can succeed spoofed DDoS attack targeted at any victim in other ASes Traceback We use Ψ 1 (τ) to measure reactive protection performance (a.k.a traceback) of route-based DPF during transient periods. As defined in [7], Ψ 1 (τ) is a scalar between 0 and 1 for the given parameter τ, and it denotes the fraction of ASes, which on receiving a spoofed IP packet can localize its physical source to within τ sites. 4.3 Experimental Setup We performed experiments on the 3023-node NLANR [9] measurement topology dated 11/08/1997. According to [?] s stub/transit classification on the 3023-node

46 36 topology, around 80% of nodes are classified as stub nodes. We place route-based DPF filter on selected 580 nodes, which compose a vertex cover of the given topology. We considered two single-node failure benchmark scenarios. First, we considered a single degree, stub AS node which is connected to the highest degree node. Due to the power-law nature of AS-level Internet topology [40], most AS nodes correspond to this category. The faulty node (AS3) is assigned to Massachusetts Institute of Technology, and the connected one of highest degree AS nodes (AS1) corresponds to Genuity. Next, we select a degree-9 transit AS node as a faulty node. The faulty node (AS3407) is assigned to Interpath, and the connected 9 high degree AS nodes (AS81, AS286, AS701, AS1239, AS2548, AS2551, AS2914, AS3561, and AS5413). One item to note here is that we do not consider the case when a high degree node is faulty. In that case, basic routing itself will not perform its functionality. Thus, the route-based DPF protocol, which relies on the underlying routing, cannot function correctly either. For both experimental scenarios, Reflect Timer interval is set to 5 seconds; BGP ConnectRetry interval, 120 seconds; Hold Time interval, 90 seconds; KeepAlive interval, 30 seconds; and MinRouteAdvertisementInterval is set to 30 seconds. We executed benchmark simulations from 0 second until 5000 seconds. BGP Loc-RIB tables (storage for selected route update, which is consistent with local IP routing table) and DPF filter tables are dumped every 150 seconds. 2 When simulation starts, BGP and route-based DPF calculate IP routing table and DPF filter table, respectively. These tables converge at around 300 second. A faulty node goes down at 350 second. Thus, transient state transitions start at that time. 4.4 Stability Figure 4.1 shows BGP stability as a function of simulation time. In each plot, entry count and node count represent distance with respect to entry and distance 2 We do not dump routing and filter tables whenever BGP and route-based DPF protocol affect changes. Instead, we log the intermediate state of routing tables and filter tables periodically.

47 37 with respect to node, respectively. From the trajectory of node count in Figure 4.1(a), we observe that more than 40% of nodes converge at time 1500 second, around 1200 seconds after the start of the the transient period. Then it remains stagnant for around 300 seconds, and the rest of them converge in the next roughly 700-second period. In the end, BGP routing stabilizes at around 2500 second. Distance with respect to entry is close to 0 during the whole transient period, although distance with respect to node granularity is significant. It implies that most of the nodes have had few inconsistent entries throughout the whole transient period. In the node count trajectory of Figure 4.1(b), we observe that the pattern of BGP stabilization is similar to that of 4.1(a). The speed of stabilization is slower, and it stabilizes at around 2800 second. Comparatively, the transit node failure case takes longer time for BGP stabilization than the stub node failure case. As with entry count plot of Figure 4.1(a), it is close to 0 during the whole transient period which indicates that each node has few inconsistent entries. 1 entry count node count 1 entry count node count distance of routing distance of routing time time (a) A stub node failure. (b) A transit node failure. Figure 4.1. BGP routing stability as a function of simulation time. Figure 4.2 shows consistency of filter tables as a function of simulation. We measure consistency of filter table information at a time instance by measuring the distance from theoretically determined filter information for the given routing at

48 38 the time instance. Each plot in Figure 4.2 presents distance with respect to entry as entry count ; distance with respect to table, filter count ; and distance with respect to node, node count. In Figure 4.2, we do not observe any transition in node count after the faulty node goes down. Distance with respect to filter table increases during the initial transient period, and it stabilizes at around 2500 seconds. In the same way, distance with respect to entry increases during the initial period, and it stabilizes. The node count plot shows that most filter sites have at least an inconsistent entry. filter count, however, shows that the actual fraction of filter tables which have those invalid entries are less than 20%. Moreover, entry count plot shows that the fraction of invalid entries are significantly less. In conclusion, we find that there exist few invalid entries in a few filter tables. Nevertheless, they are dispersed around the whole filter sites entry count filter count node count entry count filter count node count distance 0.4 distance time time (a) A stub node failure. (b) A transit node failure. Figure 4.2. Consistency of filter tables as a function of simulation time. As explained in Section 4.2, these differences belong to safety violation category or staleness category. In case when there is no entry for a valid source address, this belongs to safety violation category. On the other hand, the case when there exists an entry for invalid source address is classified into staleness category.

49 Safety Violation Figure 4.3 shows safety violation indices SV entry, SV filter, and SV node as a function of simulation time for the two benchmark scenarios. The plots entry count, filter count, and node count denote SV entry, SV filter, and SV node, respectively. In both Figure 4.3(a) and Figure 4.3(b), safety violation happens during the transient period. After BGP s convergence (at around 2500 seconds and 2800 seconds, respectively), safety violation does not happen in both figures. From the node count plots in both figures, we find that around 10% of filter sites violate safety during the transient periods. However, distance values of entry count and filter count indicate that few safety violating entries are scattered over those filter sites entry count filter count node count entry count filter count node count safety violation safety violation time time (a) A stub node failure. (b) A transit node failure. Figure 4.3. Safety violation as a function of simulation time.

50 Staleness Figure 4.4 shows staleness indices ST entry, ST filter, and ST node as a function of simulation time for the two benchmark scenarios. The plots entry count, filter count, and node count denote ST entry, ST filter, and ST node, respectively. In Figure 4.4, entry count, filter count, and node count increase during the transient period and converge after BGP s stabilization. In both cases, staleness remains after the transient period. From comparison with Figure 4.2, we find that inconsistencies most of node experienced throughout the simulation period are due to staleness. As shown in Figure 4.3, safety violation happens only during transient period, and staleness persists even after the transient period. Plots of node count in Figure 4.2, Figure 4.3(a), and Figure 4.4(a) imply that some nodes exhibit both safety violation and staleness during the transient period. The same observation holds in the transit node failure case entry count filter count node count entry count filter count node count staleness staleness time time (a) A stub node failure. (b) A transit node failure. Figure 4.4. Staleness as a function of simulation time.

51 Containment Route-based DPF provides proactive protection (a.k.a. containment) against spoofed DDoS attack by discarding spoofed IP packets in the first place. In this section, we analyze containment of route-based DPF during the transient periods in the presence of safety-violation and staleness. As examined earlier, a node failure causes BGP instability for 2500 seconds to 3000 seconds. During the interim, route-based DPF filter contains incorrect entries, which cause safety-violation and staleness. However, measures for safety violation and staleness do not capture degree of performance degradation experienced by each source and destination pair in reality. Figure 4.5 provides containment index Φ 2 (1) as a function of simulation time. In both experimental scenarios, route-based DPF provides around 99% containment. It means that around 99% of nodes are contained by route-based DPF, and spoofed DDoS attack cannot succeed on the 99% of nodes targeting at any victim at other AS nodes. In particular, in contrast to wide spread of staleness during and after the transient periods, we do not observe performance degeneration Φ 2 (1) Φ 2 (1) time time (a) A stub node failure. (b) A transit node failure. Figure 4.5. Containment as a function of simulation time.

52 Traceback Although containment of route-based DPF does not allow around 99% of nodes initiate spoofed DDoS attack, around 1% of nodes can do spoofed DDoS attack. Reactive protection of route-based DPF (a.k.a. traceback) provides localization of physical sources within few sites, upon reception of spoofed IP packets. Figure 4.6 provides trajectories of traceback index Ψ 2 (τ) with resolution parameter τ. Each trajectories are calculated from dumped routing and filter information at each time instance. Although we do not observe any distinct pattern where resolution parameter τ is less than 5, we find Ψ 2 (4) approaches to 1 in both node failure cases. Over all simulation periods, upon receiving spoofed IP packets, route-based DPF can localize their physical source within 4 nodes. This feature persists even during and after transient periods. Ψ 1 (τ) sec 300 sec 450 sec sec 750 sec 900 sec 1050 sec sec 1350 sec 1500 sec 1650 sec sec 1950 sec 2100 sec 2250 sec 2400 sec sec 2700 sec 2850 sec 3000 sec τ Ψ 1 (τ) sec 300 sec 450 sec sec 750 sec 900 sec 1050 sec sec 1350 sec 1500 sec 1650 sec sec 1950 sec 2100 sec 2250 sec 2400 sec sec 2700 sec 2850 sec 3000 sec τ (a) A stub node failure. (b) A transit node failure. Figure 4.6. Traceback as a function of resolution. Figure 4.7 provides traceback index Ψ 2 (τ), where τ is from 1 to 5, as a function of simulation time. Since staleness increases during transient period, reactive performance with resolution parameter 2 and 3 diminishes in a stub node failure case; reactive performance with resolution 2 decreases in a transit node failure case. As shown in Figure 4.6, nonetheless, Ψ 2 (4) remains close to 1 for the entire simulation

53 43 period. It means that, upon reception of spoofed IP packet, roughly any victim can localize actual attacker within 4 sites at any time instance Ψ 1 (τ) Ψ 1 (τ) τ = τ = 2 τ = 3 τ = 4 τ = time τ = τ = 2 τ = 3 τ = 4 τ = time (a) A stub node failure. (b) A transit node failure. Figure 4.7. Traceback as a function of simulation time.

54 44

55 45 5 DYNAMIC DPF SIMULATOR The Dynamic DPF Simulator is a tool for evaluating performance of the route-based DPF protocol in a dynamic network environment where states of network elements, such as routers and links, and network protocols may change. Aiming at a realistic and scalable evaluation of dynamic performance of the route-based DPF protocol, the Dynamic DPF Simulator is designed to work with large-scale Internet Autonomous System (AS) measurement graphs. The tools contained therein are applicable to general network simulation environments, including router graphs and Internet traffic controls. The Dynamic DPF Simulator is built on top of DaSSFNet, which provides a network simulation environment for workstation clusters and parallel computers. The Dynamic DPF Simulator provides additional functionalities, encompassing automatic partitioning and simulation configuration, various network protocols, and a comprehensive measurement framework. The rest of the chapter is organized as follows. First, we give a brief overview of DaSSFNet. This includes a description of the major features of DaSSFNet. The description of the Dynamic DPF Simulator as a complete system follows. Each major component is discussed in its dedicated section. 5.1 Overview of DaSSFNet DaSSFNet [8] is a DaSSF-based implementation of SSFNet. As a general-purpose simulation environment, the Dartmouth Scalable Simulation Framework (DaSSF) [39] provides a C++ implementation of the Scalable Simulation Framework (SSF). In addition, DaSSF supports shared-memory multiprocessors and distributed-memory machines as its platform, incorporating advanced parallel simulation techniques. SSF

56 46 [50] defines a unified, object-oriented application programming interface (SSF API) as a standard user interface for discrete-event simulation, considering usability and performance as its primary design goal. Supporting a process-oriented world-view of discrete-event simulation, SSF helps make detailed design and implementation of network models including protocols possible. SSFNet [35] provides simulation models of various network elements and network protocols on top of a Java-based implementation of SSF. In the rest of the section, descriptions of the three key features of DaSSFNet scalable simulation kernel, process-oriented world-view, and various network simulation models, are discussed. Then, a brief introduction to the Domain Modelling Language (DML) for network specification is presented. DaSSF, as a base system of DaSSFNet, provides a scalable simulation kernel along with a network specification language DML. DaSSF can be configured either as a stand-alone single process application or as a networked distributed application. As a distributed application, it runs on distributed-memory machines, such as Linux workstation clusters the specific environment we use for benchmarking. For the latter, DaSSF uses the Message Passing Interface (MPI) [51] for synchronization and communication between system components, that implement advanced parallel simulation techniques. The parallel simulation techniques employed by DaSSF include distributed event synchronization, thrifty memory usage, and efficient multi-threading. The threading mechanism reduces the overhead of process context switching cost as well as that of memory consumption [39]. DaSSF exports an application programming interface that is compliant with SSF. A process-oriented world-view is supported by SSF, and hence, DaSSF. The SSF API includes five primary class interfaces Entity, inchannel, outchannel, Event, and Process. Entity is a simulation subject which stores the state of a simulation. inchannel and outchannel define a way of message passing between Entity objects. Event represents a form of message exchanged via inchannel and outchannel objects. Finally, Process is a thread of control, which is scheduled by the simulation framework

57 47 nonpreemptively. As an action of a thread, a Process object, which belongs to an Entity object, can generate and send an Event object to other Process objects. By providing a similar abstraction of processes as that used in multi-tasking operating systems, the SSF API supports a familiar process-oriented world-view to modelers. Thus, we can design, implement, test, and analyze simulation models as we do so in real systems modulo idiosyncracies imposed by hardware characteristics. DaSSFNet comes with a collection of simulation models of network elements and network protocols, specifically IP and TCP. Adopting the process-oriented worldview of SSF and its C++ realization for workstation clusters by DaSSF, DaSSFNet provides a network simulation environment, which is amenable to full-fledged network protocol implementations. Following is a description of the main components: Machine is a logical subject of network simulation components and is modelled as an Entity object of the SSF API. It consists of zero or more network interfaces and a network protocol graph which contains installed network protocols. NIC representing a network interface, is modelled as a Process, containing an in- Channel object and an outchannel object. The pair of inchannel and outchannel objects reflects the bi-directional nature of communication media and MAC protocols. A packet is received and sent through these inchannel and outchannel objects. By modelling an interface as a Process, it can detect and handle incoming packets as soon as the simulation kernel notifies their arrival to the inchannel of the interface. Link models a mapping of inchannel objects and outchannel objects. According to the configuration of a link model, it provides one-to-one or one-to-many mappings between participating inchannel objects and outchannel objects. For example, the point-to-point connection between two machines is represented as two one-to-one mappings between inchannel objects and outchannel objects. On the other hand, in the case of modelling a Local Area Network (LAN), each outchannel object is mapped to all inchannel objects within LAN.

58 48 Hardware is modelled as a logical border between network elements and network protocols. All incoming packets received by a network interface, or outgoing packets sent from higher layer network protocols, are passed to a hardware object. The hardware object passes the incoming packets to the network protocol model installed at the lowest layer. Conversely, outgoing packets sent from the lowest layer protocol are passed to the network interface to be sent via its outchannel object. Network protocols, including IP, TCP, UDP, HTTP, and a BSD-like socket interface, are implemented as separate C++ classes, borrowing the design philosophy of the x-kernel [52]. First, network protocols, installed on a machine, compose a graph of network protocols following protocol layering. Next, every incoming packet is passed to protocols by invoking generic member functions send() and receive() of each protocol according to the network protocol graph structure. Under these design rules, a full-fledged protocol implementation is provided. One thing to note here is that message passing via network interfaces is managed by DaSSF. An instance of class KernelEvents is created and scheduled by DaSSF when an Event object is sent through an outchannel object. While processing the scheduled event, DaSSF creates and schedules another instance of class KernelEvents for its arrival. However, packet handling by network protocol is realized as a chain of procedure calls between related objects, which model network protocols. In other words, the scheme does not require DaSSF to be involved for these packet processing tasks. As a result, packet processing might be finished in zero simulation time unless a protocol model implements a form of processing delay. Although this provides efficiency by reducing the cost for additional event handling related to packet processing, it can also be considered a drawback since processing delay in real systems is ignored. Nonetheless, it provides an abstraction similar to that of network protocol implementations in typical operating systems, such as Linux and Windows.

59 49 DaSSFNet requires a network model the target of a simulation to be described in the Domain Modelling Language (DML) [53]. Figure 5.1 illustrates a simple network specification in DML which consists of one router. Net encloses a configuration of a network. Similarly, Router encloses a configuration of a router. Within the router specification in Figure 5.1, a network interface and a network protocol graph are configured using interface and graph, respectively. The network protocol graph includes IP, TCP, a BSD-like socket interface, and BGP protocol specifications which use ProtocolSession. Ordering of protocol specifications within graph implies protocol layering starting from the highest layer. When more than one protocol exists above a protocol in the protocol stack (e.g., TCP/UDP on top of IP), a special keyword child 1 is used within the upper layer protocol specifications to indicate the lower layer protocol as their common lower layer protocol. DML includes a special keyword, alignment, within Net for describing partitioning information. The keyword takes a string value which identifies a partition group uniquely. More than one network specification can belong to the same partition group. Figure 5.2 presents how a mapping between partition groups and participating distributed machines are specified using MapInfo. A keyword, nnodes, specifies the number of distributed machines participating in a simulation run. A partition group, "group1", is mapped to a machine whose identifier is 1. An identifier of a machine is assigned by MPI starting from 0 to the number of machines - 1. By default, network specifications, which do not have any alignment information, belong to the same partition group, and it is implicitly mapped to a machine whose identifier is 0. Figure 5.3 shows an example DML snippet of a point-to-point link. The link represents a physical connection between two interfaces, whose Network Host Interface (NHI) [53] addresses are 1:1(0) and 2:2(0). The keyword attach is used to specify an interface. Here, 1:1(0) means the interface 0 of the host 1, within the network 1. In the same way, 2:2(0) means the interface 0 of host 2 within network 2. 1 Although this keyword was supported in a past version of DaSSFNet, the latest version does not provide it. The Dynamic DPF Simulator has a modified version of DaSSFNet that supports the keyword.

60 50 Net[ id 1 alignment "group1" Router[ id 1 interface[id 0] graph[ ProtocolSession[ name bgp use SSF.OS.BGP ] ProtocolSession[ name socketmaster use SSF.OS.Socket.socketMaster ] ProtocolSession[ name TCP use SSF.OS.TCP.tcpSessionMaster ] ProtocolSession[ name IP use SSF.OS.IP ] ] ] ] Figure 5.1. A simple network specification in DML, which consists of one router. MapInfo[ nnodes 2 map [ alignment "group1" machid 1 ] ] Figure 5.2. Mapping of partition groups onto distributed machines.

61 51 link[ attach 1:1(0) attach 2:2(0) ] Figure 5.3. A DML snippet of a point-to-point link.

62 DaSSFNet-based Parallel Network Simulation Environment Description As a comprehensive DaSSFNet-based distributed network simulation environment, which considers AS-level Internet graph as its main input, the Dynamic DPF Simulator supports the following features essential for scalable simulation: Automatic model configuration and topology partitioning Since the Dynamic DPF Simulator targets AS-level Internet graphs, which are large and have their own graph representation format, a subsystem, Meta-DML, is provided as an automatic DML configuration tool. At the same time, Meta- DML partitions an AS-level Internet graph into separate partition groups, which are mapped to participating distributed machines. Our own algorithm for partitioning AS-level Internet graphs is employed which aims to achieve scalable simulation with respect to the size of the input graph and efficient use of distributed memory CPU and communication resources. Measurement framework Although the distributed simulation platform supporting automatic network configuration and power-law topology partitioning enables large-scale network simulation that efficiently utilizes distributed system resources, a measurement framework for resource monitoring is needed for analyzing run-time performance of the distributed simulation, especially as it pertains to dynamic memory consumption a key bottleneck and event monitoring for diagnosis and performance analysis. The Dynamic DPF Simulator incorporates a comprehensive measurement framework for large-scale distributed simulation monitoring, which includes memory consumption, CPU load distribution, and communication cost. The Dynamic DPF Simulator provides a collection of measurement routines with a standardized way of specifying measurement configurations in DML.

63 53 Network protocol models First, DPF-update and DPF-lookup protocol models are designed and implemented for route-based DPF s dynamic performance evaluation. Second, BGP is implemented by porting the Java-based implementation of SSFNet [35]. Since DaSSFNet does not provided a dynamic routing protocol models, such as BGP or OSPF, a BGP implementation was required for performance benchmarking on AS-level Internet measurement graphs. Next, a range of application models are supported. They include traffic generators, attackers, and system fault generators. Figure 5.4 presents network protocol models supported by the Dynamic DPF Simulator 2. CBR, Poisson, file trace, MMPP, LRD {attackers, traffic generators, fault models,...} Applications BGP TCP UDP IP Link Layer Figure 5.4. Network protocol models in the Dynamic DPF Simulator. Modelling AS-level Internet graph An AS-level Internet graph represents an AS as a node and peering relationships of ASes as edges between nodes. An AS in the Dynamic DPF Simulator is modelled as a network with one border router. Peering relationships between ASes are modelled as physical and logical connections between border routers. 2 The latest release of DaSSFNet includes the UDP protocol. It was not provided at the time of design and implementation of our prototype system.

64 54 Physical connection stands for physical link between border routers, and logical connection stands for peering relationship established by BGP at run-time. Due to the AS-level viewpoint, an AS where an attacker s machine belongs to is modelled as a network whose border router runs the attack application. Similarly, an AS where a route-based DPF filter is deployed is modelled as a network whose border router is configured with DPF-update and DPF-lookup protocols. The specification of the DPF-lookup protocol includes a list of network interfaces where route-based DPF filters are deployed. Thus, we can selectively deploy route-based DPF filters at physical links of a border router. For simplicity, identifiers of both network and border routers within the network are the same as that of the corresponding AS node of the graph. In case of network interfaces at a border router, serial numbers starting from 0 are assigned. IP address assignment DaSSFNet provides an IP addressing scheme which use subnet addressing. Subnet addressing is assumed within link and Net specifications. Thus, DaSSFNet automatically calculates a subnet mask depending on the number of network interfaces within link or Net. Each network interface is assigned an IP address by adding a unique identifier within its subnet to the subnet mask calculated. However, this addressing scheme is not suited for modelling peering relationships between ASes. As an administrative domain, each AS manages and assigns IP addresses that we allocated to it by InterNIC. Thus, modelling a link between border routers at different ASes as a subnet is not appropriate. For this reason, the Dynamic DPF Simulator has its own addressing scheme. Assuming subnet addressing is used within an AS, the most significant 16 bits are used for the subnet mask, in other words, a common IP prefix. The least significant 16 bits are filled with a unique identifier denoting an interface.

65 55 Figure 5.5 shows the system architecture of the Dynamic DPF Simulator. As mentioned earlier, Meta-DML generates a DML configuration file from a given network partitioning information (i.e., the number of distributed machines), network topology (i.e., AS-level Internet graph), and additional user-specified protocol configuration information. Figure 5.5. System architecture of the Dynamic DPF Simulator. DaSSFNet takes a DML configuration file as input and creates C++ objects of network elements and network protocols as specified in the file. Since the DML configuration file includes partitioning information, C++ objects of a network specification are instantiated at a corresponding machine where their partition group is assigned. Before a simulation starts, there are two phases of preparation for a simulation self-configuration and self-initialization. Self-configuration precedes self-initialization. First, DaSSFNet triggers self-configuration by calling Configure() of the C++ object of the outermost Net specification, which encloses the whole simulation model. In

66 56 turn, it triggers Configure() of all C++ objects of specified models. Next, they trigger Configure() of all C++ objects of models specified within. In this fashion, C++ objects of all simulation models are recursively configured. After configuration, DaSSFNet triggers self-initialization by calling Initialize() of the C++ object of the outermost Net specification. Once all C++ objects are created, configured, and initialized by DaSSFNet, simulation starts. As a discrete-event simulator, DaSSF processes events in the simulation kernel. These events, called kernel events, are generated by applications and network protocol models. After processing events scheduled at a particular time, DaSSF schedules Processes nonpreemptively. After scheduling all eligible Processes, DaSSF advances the current simulation time to that of the nearest future event(s) 3. In case processing of an event requires communication or synchronization with simulation state at other participating machines, DaSSF communicates via MPI to affect coordination Automatic Model Configuration and Partitioning Since the Dynamic DPF Simulator is used with AS-level Internet graphs which are large and have their own graph representation format, an automatic DML configuration tool is provided as a subsystem. At the same time, Meta-DML partitions a given AS-level Internet graph into separate partition groups, which are mapped to participating distributed machines. We devised an algorithm for partitioning ASlevel Internet graphs possessing power-law connectivity properties, aimed achieving scalable simulation with respect to the size of the input graph through efficient use of distributed memory, CPU, and communication resources. Meta-DML accepts network partitioning information (i.e., the number of distributed machines), network topology (i.e., AS-level Internet graph and transit AS node information), and additional protocol configuration. For example, additional 3 Depending on how the initial triggering of events in a simulation is set up, more than one event may occur simultaneously and is scheduled accordingly

67 57 protocol configuration includes distributed filter configuration such as, list of DPF filter-deployed ASes and attack configuration such as, list of ASes, which install attacker models. Parsing a given AS-level Internet graph input, Meta-DML builds an initial graph representation. Meta-DML partitions the given Internet graph over the participating distributed machines and stores the information. Next, distributed filter and attack configuration are processed. Finally, Meta-DML generates a DML file from the information collected. Figure 5.6 shows a sample Meta-DML input file. The variable size denotes the number of nodes in the input graph, machines represents the total number of machines, and dynamic_1_static_0 takes on the values 0 or 1. When dynamic_1_static_0 is set to 0, Meta-DML generates an additional variable routing_tbl_file within the IP DML specification, so that IP routing table entries are loaded from a file. Otherwise, BGP is configured at every node, in order to calculate routing table information dynamically. The variable filtering_1_nofiltering_0 is used to configure filtering functionality of each DPF-lookup protocol centrally using filtering of DPF-lookup DML model. An AS-level Internet graph which is represented as an adjacency list is given as input specified by graph_input. transit_node_input specifies a list of transit AS nodes. Nodes that are not listed in the file are configured as stub AS nodes. The variable filter_node_input takes a list of filter-deployed nodes as argument. attack_input represents a list of attacker nodes. Partitioning Partitioning a given graph for efficiently utilizing distributed memory, CPU, and communication resources is a critical component of large-scale simulation. As presented in Figure 5.7, AS-level Internet graphs based on Oregon RouteViews/NLANR measurements have grown super-linearly during The measurement topology dated 01/01/2002 has 12,514 nodes. With the growth of the number of AS nodes n, O(n 2 ) memory requirement for maintaining O(n 2 ) routing entries is required in

68 58 size 3023 machines 10 dynamic_1_static_0 1 filtering_1_nofiltering_0 1 graph_input ASgraph.3023 transit_node_input AStrans.all filter_node_input VC.3023 attack_input attack.3023 Figure 5.6. A sample Meta-DML input file. case each AS has at least one unique IP prefix. Furthermore, simulation scenarios involving varying traffic patterns imposes additional memory requirements as well as CPU and communication resource requirements # AS year Figure 5.7. Growth of AS-level Internet graph. One important feature of distributed simulation with respect to partitioning is that the slowest participating process determines the execution time of the whole simulation. This implies that effective load balancing is crucial for parallel speedup. Furthermore, overloaded memory resource requirements may cause swapping between main memory and disk if the virtual memory system gets triggered which has a debilitating effect on distributed simulation. A key requirement of partitioning,

69 59 in addition to CPU and communication balancing, is that static and dynamic memory requirement is balanced so that the virtual memory system is kept at bay. In this thesis, we assume a homogeneous workstation cluster environment, where each machine has uniform memory, CPU, and communication resources. Since a major challenge for performing large-scale network simulation comes from significant memory requirement, we focus on balanced distribution of static and dynamic memory requirement. To do so, we identify key factors involved in memory consumption from experiments, and they are offloaded into participating machines. By harnessing locality of message exchange within each distributed machine, overhead of synchronization and message exchange between machines is reduced, and utilization of given CPU and communication resources is increased. From performance results of memory requirement monitoring in Chapter 6, it turns out that routing tables, especially Adj-RIB-In tables of BGP, are the most dominant factor in terms of static memory consumption. The results also show that messages occupy only a small portion of dynamic memory compared to that of tables. Thus, we first focus on evenly distributing BGP Adj-RIB-In table s memory requirement into participating machines for static memory requirement balancing. Since every edge in an AS-level Internet graph represents a BGP peering relationship, a AS node u has Adj-RIB-In tables with O( V deg(u)) space requirement where V represents the total number of nodes. Denoting the number of edges of an AS-level Internet graph and the number of participating machines as E and k, respectively, the total number of Adj-RIB-In tables comes to 2 E, and its space complexity is O( V E ). Thus, our heuristic algorithm limits the total number of Adj-RIB-In tables in each partition group to around 2 E /k. As presented in [40], AS-level Internet graph possess power-law connectivity properties. One of the implications is that there exist a few high degree ASes, which possess peering relationships with many low degree ASes. Figure 5.8 shows a 300-node subgraph of the 3023-node AS-level Internet graph of Oregon RouteViews/NLANR

70 60 measurements from 11/08/1997. We observe a few locally star-like AS clusters that are connected by a more complicated backnone. Figure node AS-level Internet graph. Exploiting power-law connectivity in partitioning helps to reduce the frequency of message exchange between distributed machines, which reduces completion time due to reduced message exchange overhead. Considering locally star-like AS clusters, all network traffic generated by, or heading to, many low degree ASes should go through central high degree AS nodes. Hence, one locally star-like AS cluster is put into one partition group. This presents message exchanges within locally star-like AS clusters from being sent across distributed machines resulting in communication overhead. The partitioning routine takes a graph G = (V, E) and the number of machines k as input. It returns k partition groups as output. The entire procedure of partitioning consists of 4 steps sorting, phase 0, phase 1, and phase 2. In the sorting step, V, the set of nodes of the graph G, is sorted by the degree of each node. Phase 0 is a step that uses the power-law property for partitioning. During phase 0, the k largest locally star-like AS clusters are assigned to k separate partition groups. This is shown in Figure 5.9. Checking the adjacency list of each k highest degree nodes, degree-1 nodes are put into the same partition group as their adjacent highest degree node. This enables messages between degree-1 nodes connected by a central high-degree node to be localized within a partition group.