Clean Pipe Solution 2.0

Transcription

1 Clean Pipes 2.0 1

2 Clean Pipe Solution 2.0 Executive Summary...3 Best Current Practices...5 Network Infrastructure BCPs...5 Host Based BCPs...5 Dedicated DDoS BCPs...6 Cisco Clean Pipes Solution Overview...6 Evolution of Cisco Clean Pipes Solution...6 Protection Mechanism of Cisco Clean Pipes Solution...8 Cisco Clean Pipes 2.0 Components...10 Cisco Netflow Arbor Peakflow SP Arbor Peakflow SP Threat Management System (TMS) DDoS Protection Flow in Clean Pipes Baseline and Thresholds Detection Diversion Scrubbing Injection Migration to Cisco Clean Pipes Cisco Anomaly Guard and TMS Countermeasure Comparison...17 Clean Pipes 2.0 Deployment Considerations...19 Netflow considerations for Peakflow SP Deployment of Peakflow SP PI Deployment of Peakflow SP Collectors Deployment of Peakflow SP TMS Peakflow SP Communication Ports Data collection...24 Inter appliance communication (all appliances)...24 Peakflow SP PI and leader appliances...25 Scaling Clean Pipes Conclusion...28 Appendix...29 Appendix A: Peakflow SP System wide Enforced and Guideline Limits Appendix B: Peakflow SP PI 5500 Appliance Enforced and Guideline Limits Appendix C: Peakflow SP CP 5500 series Appliance Enforced and Guideline Limits Appendix D: Peakflow SP TMS Appliance Limits Appendix E: Six Phase Approach to Infrastructure Security

3 Executive Summary Distributed Denial-of-Service (DDoS) attacks are amongst the most prominent attacks targeting network infrastructures or computer services resources. The primary goal of DDoS attacks is to deny legitimate users access to a particular computer or network resources, which results in service degradation, loss of reputation and irretrievable data loss. DDoS attacks are aimed at businesses of any size and type. Businesses with on-line presence are all potential victims of DDoS attacks, including all vertical markets such as financial, retail, media and entertainment, manufacturing, services and the government. Even individuals are being attacked. Many enterprises are migrating to cloud computing models making use of centralized data centers and virtualization to reduce capital and operations expenses. The data centers that house these large virtualized data stores are particularly sensitive targets to DDoS attack as a single attack can produce considerable collateral damage beyond the direct victim. DDoS has evolved from random hacker exploits to organized criminal activities which often involve botnets, which are large groups of compromised host computers controlled by a central, coordinated commander. The size, complexity and sophistication of DDoS attacks are increasing at alarming rates making it more challenging to protect network resources. According to Arbor Networks Annual Worldwide Infrastructure Security Report (2008), Internet Service Providers have seen DDoS attacks as large as 40 Gbps that s an increase of 67% from 2007 and a 100 fold increase since An attack of over 80 Gbps was detected in the first half of 2009 showing that the growth trend is not slowing down. To address these inevitable, growing network threats, network operators are urged to employ the best current practices (BCP) for protecting networks. BCPs are pro-active methods that have been adopted in the industry to prepare networks against threats. BCPs include network infrastructure best practices, host best practices and deployment of dedicated DDoS detection and mitigation, such as Cisco Clean Pipes Solution. Cisco Clean Pipes Solution is the purpose-built architecture for dedicated DDoS detection and mitigation. As opposed to traditional DDoS defense techniques, the Cisco Clean Pipes solution can accurately distinguish legitimate traffic from malicious traffic destined for a mission-critical host or application. It precisely blocks the attack traffic while allowing legitimate traffic to pass through, which enables the maximum business and service continuity. The Clean Pipes Solution allows service providers to deliver in-cloud, managed anti-ddos services to their customers. It also provides enterprise customers with the ability to defeat DDoS attacks on their own premises with surgical DDoS attack detection and protection with finer granularity. The Cisco Clean Pipes solution has been a great success and widely adopted by many large service providers, hosting providers and large enterprise customers. Given the constant evolving nature of DDoS attacks, the Cisco Clean Pipes Solution also evolves rapidly. Starting with Cisco Anomaly Guard appliance which has 1Gbps mitigation capability, it now features Anomaly Guard Module (AGM) for up to 3Gbps mitigation per module. Multiple AGM modules can cluster offering 10+ Gbps protection. Moving forward, in an effort to achieve cost efficiency and high feature velocity, Cisco has decided to stop developing the Anomaly Guard Module and the Anomaly Detector Module, and 3

4 to partner with Arbor Networks to continue to provide a comprehensive and tightly integrated Anti-DDoS solution, which will evolve the Cisco Clean Pipes Solution to version 2.0. In Clean Pipes 2.0, Cisco and Arbor will closely collaborate in the integration of Cisco Netflow technology and Arbor DDoS detection and mitigation technology to provide more advanced and higher performance Anti-DDoS protection. Arbor Networks is well known for their security expertise and Peakflow SP solution which is deployed in many service provider networks where it provides comprehensive DDoS detection, surgical mitigation and reporting. The Arbor Networks and Cisco partnership is not new as Arbor s Peakflow solutions have leveraged Cisco s Netflow technology for a number of years. Also, Arbor s Peakflow SP product has been a supported option for attack detection in Clean Pipes Solution 1.0 and 1.5. For Clean Pipes 2.0, Arbor s Peakflow SP product will be used for anomaly detection, while the Peakflow SP Threat Management System (TMS) product will be used for surgical mitigation of DDoS attacks. For customers who re using Clean Pipes solution 1.0 or 1.5 today, Clean Pipes 2.0 is the migration path to achieve higher scalability and new functionalities in the future. Cisco and Arbor will continue the joint effort of creating an ever tighter integrated anti-ddos solution. 4

5 Best Current Practices There are a number of industry best current practices (BCPs) which should be proactively deployed by network operators responsible for Internet-facing infrastructure and properties. These practices were established in accordance with the Cisco-Arbor developed 6 phases of infrastructure security. Network Infrastructure BCPs Network infrastructure BCPS are pro active measures that are implemented directly on Cisco router and switching infrastructure along with other network devices. Interface ACLs (iacls) should be employed at the relevant network edges (peering/transit, customer aggregation edge, etc.) to protect the network infrastructure itself. Service-specific ACLs should be used on data center routers to restrict traffic destined for Internet-facing servers to the ports and protocols associated with the services and applications on those servers. Control and management plane protection mechanisms should be deployed per device, protocol and vendor recommendations All network infrastructure devices should be accessible only via designated management hosts, and this access should be facilitated via a dedicated out-of-band (OOB) management network. Flow telemetry using Cisco NetFlow should be enabled at all network edges, and exported into a collection/analysis system such as Peakflow SP Source-based remotely-triggered black holing (S/RTBH) is a powerful reaction technique that allows tens or even hundreds of thousands of attacking source IPs to be rapidly black holed based upon their source addresses. S/RTBH leverages BGP as a control-plane mechanism to instantaneously signal edge devices to start dropping attack traffic. Reverse-proxy caching in front of Internet-facing Web properties allows for scaling of capacity as well as a policy control point which enables filtering of layer-7 application protocol traffic. Host Based BCPs Host based BCPs are measures that are applied directly on hosts that may come under attack and provide a degree of initial protection. Pro active patching of the host 5

6 Server hardening including the shut down of any unnecessary services and host based ACLs restricting access to the server to only specific source hosts and on specific ports. Out of band management access to the device. Service specific configuration hardening including shut down of unused features and access mechanisms IP stack tuning Employ anti virus and anti spam mechanisms Dedicated DDoS BCPs The use of dedicated DDoS detection and mitigation infrastructure completes the net of a complete DDoS protection infrastructure. DDoS detection and mitigation devices are specifically designed to detect DDoS events as they occur, provide traceback and analysis to operators, and intelligently mitigate attacks by dropping malicious traffic while preserving legitimate traffic. Cisco Clean Pipes is the industry BCP for dedicated DDoS and mitigation. Cisco Clean Pipes Solution Overview The Cisco Clean Pipes Solution enables service providers to provide DDoS protection services to their customers and simultaneously harden and protect their own networks. Enterprise customers can also deploy Cisco Clean Pipes Solution on their own premises to protect their network infrastructure and server resources from DDoS attacks. Evolution of Cisco Clean Pipes Solution The essentials of Cisco Clean Pipes Solution are the DDoS attack detection and mitigation devices. Table 1 shows Anti-DDoS devices in Clean Pipes 1.x and 2.0: Table 1. Anti-DDoS Devices Cisco Clean Pipe Solution Clean Pipes 1.0 & 1.5 Clean Pipes 2.0 Detection Device Cisco Anomaly Detection Appliance; Cisco Anomaly Detection Module for Cisco Catalyst 6500/ Cisco 7600; Arbor Peakflow SP; Arbor Peakflow SP Mitigation Device Cisco Anomaly Guard Appliance; Cisco Anomaly Guard Module for Cisco Catalyst 6500/ Cisco 7600 Arbor Peakflow SP Threat Mitigation System (TMS) 6

7 In Clean Pipes 1.0 and 1.5, detection can be done by either Cisco Anomaly Detection Appliance/Anomaly Detection Module or Arbor Peakflow SP. Cisco Anomaly Guard Appliance/Anomaly Guard Module are the featured attack mitigation devices. Clean Pipes 2.0 will use Arbor Peakflow SP for detection and Arbor Peakflow Threat Management System for mitigation. Cisco routing and switching devices will provide Arbor Peakflow SP with netflow information which Peakflow SP will use to analyze and establish network traffic profile and detect traffic anomaly. Differences exist among Cisco and Arbor Anti-DDos devices in terms of how they function for DDoS attack detection and mitigation. Cisco Traffic Anomaly Detection vs. Arbor Peakflow SP: Both Cisco Traffic Anomaly Detection Appliance and Cisco Traffic Anomaly Detection Modules are packet-based anomaly detectors. They monitor a mirrored copy of selected inbound traffic flowing toward destinations under protection, building detailed profiles of normal behavior of each protected devices. Any activities deviating from these profiles can be potential attacks. If it senses abnormal or anomalous behavior, the Cisco Traffic Anomaly Detection device dynamically configures a set of dynamic filters to record the events and trigger an alarm to the network staff. It can also signal Cisco Anomaly Guard devices to activate the protection and mitigation if configured to do so. Arbor Peakflow SP is a netflow-based anomaly detector. It receives Netflow telemetry from Cisco routers and switches in the network. It continually models network behavior based on the netflow statistics creating baselines of expected traffic rates. Any events deviating from the established baseline model will be identified as an anomaly and trigger an alert which can lead to further actions, including informing the network staff, and/or activating the DDoS protection function on the mitigation device. Cisco Traffic Anomaly Guard vs. Arbor Threat Management System (TMS): Cisco Traffic Anomaly Guard XT and Cisco Traffic Anomaly Guard Modules are designed as central intelligence devices which are capable of both detecting and mitigating attacks, once they have been activated. When the Cisco Traffic Anomaly Detector Module (or any other Anomaly detection device) identifies a potential attack, it alerts the Cisco Anomaly Guard Module to begin dynamic diversion, which redirects traffic destined for the targeted resources-and only that traffic-for inspection and scrubbing. From point of diversion start, Cisco Guard operates independently from other devices. It applies blocking techniques, based on Cisco's unique multi-layer verification process architecture, which delivers multiple interactive layers of defense to identify and block all types of attacks. Arbor s Peakflow SP Threat Management System (TMS) is designed to provide centralized cleaning capacity in the network providing the active packet level processing needed to thwart complex attacks. TMS maintains an active communication with the Peakflow SP system for ongoing exchange of mitigation activities, health of the TMS scrubbing capacity, real-time data exchange for mitigation and supplemental application visibility. The TMS features a set of mitigation countermeasures that are designed to isolate and block malicious traffic while passing 7

8 desirable traffic. The countermeasures available currently include anti-spoofing, host authentication techniques, packet level threshold, application specific threshold, protocol verification, baseline enforcement, idle discovery, blacklist/whitelist and payload filtering techniques. Countermeasures are continuously added or updated as new threat vectors emerge. In addition, extensive real time and post mitigation reports are available on the Peakflow SP system allowing operators to make more informed decisions on how to adapt defenses during and between attacks. The Peakflow SP system provides a single pane of glass for command, control and reporting for one or more TMS devices. Together, the Peakflow SP system and the TMS devices provide a comprehensive threat management solution for the entire network. Protection Mechanism of Cisco Clean Pipes Solution DDoS attacks are among the most difficult network threats to defend against. They mimic valid requests, spoof source identification, and use armies of compromised zombie hosts to initiate attacks so that illegitimate packets are indistinguishable from legitimate packets. This makes detection more difficult and threat mitigation with business continuity more challenging. Network devices and traditional perimeter security technologies such as firewalls and intrusion detection systems (IDSs) do not by themselves provide comprehensive DDoS protection. To pick up where traditional DDoS defense techniques leave off, the Cisco Clean Pipes solution can accurately distinguish good traffic from bad traffic destined for a mission-critical host or application. It not only detects the presence of an attack, but also filters out only the bad traffic, allowing good traffic to pass through, enabling maximum business and service continuity. This solution offers three major functional elements that work towards protecting a network from DDoS attacks: Detection Identify and classify attacks based on anomaly characteristics Diversion/Injection Divert dirty traffic to the cleaning center to be scrubbed inject clean traffic back to the DDoS targeted host Mitigation Anti-spoofing, anomaly recognition and packet inspection cleaning(scrubbing) of dirty traffic Detection The fundamental premise of detecting attacks is to build a baseline of normal network traffic levels and then look for anomalies in traffic patterns compared with the baseline. A network traffic anomaly is an event or condition in the network characterized by a statistical abnormality compared to typical traffic patterns gleaned from previously collected profiles and baselines. Any difference in traffic patterns that are above a certain threshold will trigger an alarm. Traffic diversion and injection Traffic diversion is the mechanism used to instruct an upstream router in the core network to divert traffic of compromised servers to the mitigation devices for scrubbing the dirty traffic. After scrubbing off anomaly packets, the cleaned traffic is injected back to the normal data path to reach the destination in the network. There are multiple mechanisms for traffic diversion and 8

9 injection which will be discussed in later sections. Mitigation Mitigation in the Cisco Clean Pipes solution is the process in which attack traffic is scrubbed (i.e., checked via anti-spoofing, anomaly recognition, packet inspection, and cleaned to drop bad traffic and allow legitimate traffic to the same destination). The figure in below shows the typical Cisco Clean Pipes Solution Architecture: In general, Cisco Clean Pipes Solution provide four specific service deployment models, based on the common Clean Pipes architecture, along with design guidelines tailored for DDoS protection for different parts of the SP infrastructure and customer networks: Managed Network DDoS Protection Provides enterprise customers effective protection against DDoS attacks on their last-mile connections to SPs and internal infrastructures by subscribing to the Cisco Clean Pipes service offered by SPs. Managed Hosting DDoS Protection Enables hosting providers to protect their web and other hosting services from DDoS attacks. Peering Edge DDoS Protection Enables SPs to prevent bandwidth saturation by DDoS attacks against their peering points. On-premise DDoS Protection Enables enterprise customers to deploy anti-ddos detection and protection on their own premises with finer granularity for anomaly detection and protection. 9

10 Cisco Clean Pipes 2.0 Components Cisco Netflow Netflow, as a indispensable tool to provide the visibility into network traffic, has become a standard for acquiring IP operational data for many customers. Applications for NetFlow data are constantly being invented. One of them is Anti-DDoS protection. The highly scalable view of network traffic characteristics provided by netflow data makes netflow technology the most widely deployed DDoS identification technology for large scale IP networks. At the same time, the granular flow information enables netflow-based DDoS detection devices, such as Arbor Peakflow SP, to provide a surgical detection of traffic anomalies. Netflow classifies IP packets into flows and generates flow records which can be exported to a flow collector for further analysis. Each flow is defined by its unique seven-key characteristics: Ingress interface IP protocol type Type-of-service (ToS) byte Source IP address Destination IP address Source port number Destination port number Led by Cisco core routing and switching platforms (e.g CRS-1, Nexus 7000, ASR 1000 and Catalyst 6500), Cisco Netflow technology has been constantly enhanced and refreshed, such as: Supports multiple netflow formats (e.g. v5, v7. v8 and v9); Industry s premier platform to support v9; Highly scalable netflow table; Flexible netflow which allows users to select which key or non-key fields to define a flow. It grants users more flexibility, aggregation and scalability; Time-based sampled netflow; Packet-based sampled netflow; Netflow for both ingress and egress traffic; Netflow for MPLS & Multicast traffic; Netflow for bridged traffic (enables bump-in-the-wire deployments); When the network is operating under a normal situation, Netflow yields enough data to profile the network traffic and establish a baseline which is used for traffic anomaly detection. In the event of DDoS attacks, the statistic netflow information shows deviations from the traffic baseline which can be the first sign of the attacks. Further analysis of traffic pattern and behavior can be carried out with the detailed flow information. Once a traffic anomaly is identified, corresponding Anti-DDoS countermeasure can be initiated manually by the network operator or automatically by the Anti-DDoS protection system. When Netflow is used in Anti-DDoS protection, NetFlow is usually deployed across the edge of an SP or enterprise network to monitor inbound traffic on edge and peer interfaces, because these are the typical ingress points for most attacks. The router maintains a live NetFlow cache to track the current flows. IP flow information can be exported from the NetFlow cache to an external 10

11 collector for further analysis. In Clean Pipes 2.0, Arbor Peakflow SP is the flow collector. Flow data from multiple collectors can be mapped to identify the network nodes under DDoS attack and also to determine the attack characteristics. Cisco will continuously invest in high performance netflow technology and collaborate with Arbor Networks to ensure that the Clean Pipe Solution 2.0 get enhanced with the new Netflow features. This allows the maximum investment protection of Cisco network platforms for Clean Pipe Solution 2.0 customers. For more information about NetFlow, see the following URL: Arbor Peakflow SP Arbor Networks Peakflow SP is a scalable platform that provides a comprehensive solution delivering powerful DDoS as well as traffic and routing analysis to service providers and their customers. Peakflow SP Provides three leading solutions to the market place: managed security services enablement, infrastructure security, and traffic and routing visibility and analysis. The Peakflow SP solution scales with its multi-tier detection architecture of collectors: - Tier 1: Peakflow SP Portal Intelligence (PI), which provides a central point of command and control including event correlation and traceback. PI systems provide the leader and central command function for the deployment. - Tier 2: Peakflow SP Collector Platform (CP), which collects NetFlow statistics from multiple routers and acts as a correlation engine syncing data sets between all network collectors and the PI system. - Tier 3: Peakflow SP Flow Sensor (FS), that acts as additional layer of netflow and data collection designed to scale the Peakflow deployment to the largest worldwide networks. For Clean Pipes v1.5, the Peakflow SP solution works in conjunction with the Cisco Guard for DDoS protection. Upon receiving an anomaly fingerprint for a zone from a Peakflow SP CP collector, the Peakflow SP PI controller establishes a SSH connection to activate the Cisco Guard, putting the zone under attack in protection mode. For the Clean Pipes 2.0 Solution, Peakflow SP offers a streamlined approach to DDoS attack detection, traceback, and mitigation. Peakflow SP CP systems first build baselines of normal behavior, network-wide, leveraging flow data available from the routers already deployed on their network. In contrast to inline data collection methods, Peakflow SP collects Cisco NetFlow flow-based statistics from Cisco routers, which allows Peakflow SP to scale with the network. Alternatively, Peakflow SP TMS can use packet capture/ SPAN ports on routers in which NetFlow is not available. Neither NetFlow nor packet capture imposes a performance or reliability impact upon the network; the data collection is non-intrusive. The Peakflow SP network-wide anomaly detection identifies attacks using the two most effective methods available: signature analysis and dynamic profiling. Arbor s Active Threat Feed (ATF), a data feed of traffic signatures that pinpoint potential threats and concerns to network security can be used to match traffic reported via NetFlow. Alerts based on ATF matches can be reliably detected through netflow analysis. Peakflow also actively detects anomalies through misuse identification and dynamic profile detection. Netflow provides Peakflow SP the unique 11

12 perspective to run signature analysis pervasively with a high level of accuracy while augmenting that network wide visibility with targeted packet processing analysis of suspect traffic through Arbor s Peakflow SP TMS. The figure below shows the Clean Pipes 2.0 Solution Architecture: Arbor Peakflow SP Threat Management System (TMS) Arbor Network s Peakflow SP Threat Management System (TMS) provides surgical mitigation, service analysis and reporting. TMS provides scrubbing and application specific visibility to the Peakflow SP system. TMS can be deployed in centralized scrubbing locations, regional service POPs and IDCs for infrastructure protection and clean pipes. The TMS can also be deployed as dedicated solution for specific service protection and visibility. Arbor Networks TMS is a separate and purpose built hardware platform for advanced, highspeed traffic scrubbing and analysis. The TMS product family contains systems that provide throughput from 1.5 Gbps up to 10 Gbps throughput. The range of TMS models offered provides the correct performance package for each of the multiple use-cases of the Clean Pipes 2.0 Solution. Each system provides the same feature and functionality at different performance levels to meet the desired use case. Deployed in centralized locations for scrubbing center architecture, the 5 Gbps TMS-3050 and 10 Gbps TMS-3110 models provide very high speed performance with the ability to manage multiple events on the same platform. Multiple deployments of TMS distributed throughout the 12

13 network within the network can provide a distributed response to a coordinated DDoS event. Grouping multiple TMS systems into a single logical entity provides network operators the best solution to difficult trade-offs of backhauling attack traffic across the network or requiring each of the systems to be individually configured and managed. Grouping the TMS systems provides the ability to maintain a disturbed attack to a geographically limited set of locations and protects collateral damage of network assets. The figure below provides range of TMS models available in the Clean Pipes 2.0 Solution. TMS can also be offered as a dedicated solution for specific service protection or customers of a Clean Pipes service. As a dedicated solution, TMS provides specific mitigation actions to a specific customers leveraging the integration with Peakflow SP CP systems and the ability to model customer traffic through Netflow analysis saving these baselines in Managed Objects, each TMS can also employ a customer specific template of mitigation countermeasures to ensure custom handling of the event with respect to sensitive traffic. The TMS interacts with distributed Peakflow SP CP systems in the network to both baseline data 13

14 at Layer 7 as well as provide scrubbing statistics and forensic data gleaned from attack events. Advanced visibility within the attack event provides actionable data to the user while the mitigation is ongoing. This near real time interface provides best of breed management of DDoS events augmenting the operations tool kits with the ability to drill into attack packets, correlate data common to the attack traffic streams as well as measure the effects of countermeasure filters or REGEX expressions before they are employed into the configuration ensuring the least amount of negative impact to good traffic during the event. DDoS Protection Flow in Clean Pipes Baseline and Thresholds Collection of Netflow data from various router locations and correlating this data into a comprehensive model of the network is critical to development of a surgical response to threats. The Clean Pipes components provide this functionality natively in the solution and ensure that normal network traffic variability is accounted for in these measurements. Further developing granular models of network assets, customers, services, and infrastructure serves to provide accurate levels of granularity relative to the network scale for pinpoint detection. The Clean Pipes 2.0 solution provides the ability to accurately build thousands of models, each with relative baselines, thresholds, and traffic pattern reporting to scale to very largest service offerings and global networks. 2. Detection Data retrieved from Netflow updates provided by Cisco Router infrastructure is correlated to the baseline and threshold data held in Peakflow SP system. Clean Pipes 2.0 identifies threshold violations and provides actionable information to the operations teams as rapidly as possible. The system then provides options to the operator who has the options of auto mitigation, manual mitigation, use of layered mitigation techniques such as ACL or interface level filters on Cisco infrastructure, or black holing traffic are some choices to be made. If the attack is to be mitigated through the scrubbing technology such as TMS, then the operator will initiate a diversion event. 3. Diversion Surgically redirect (off ramp) traffic into scrubbing locations using the BGP control plane, remove traffic attacking the network and pass that attack traffic into locations of the network with the scale and the scrubbing systems deployed to mitigate the attack. Architecture goals of following best practices have greatly enhanced methods for diversion and distribution of diverted traffic to scrubbing systems. Multiple methods exist to accomplish this successfully including BGP Anycast, BGP route maps and BGP community use can all ensure that the network is resilient to the threat itself, maintaining a selfdefending network. 14

15 4. Scrubbing Identification of malicious and legitimate traffic through DPI Packet analysis, heuristics and validation methods called countermeasures. Each countermeasure can provide additional granular identification of traffic. Malicious traffic is removed from the traffic stream and legitimate traffic is placed back into the network. All actions taken by the TMS are reported in both the real time mitigation report and information is included in the after action reports for the event. 5. Injection Post Processed legitimate traffic routed through a unique path to the ultimate destination of the original traffic flow. Like diversion, architectures exist to optimize injection paths and traffic delivery to victims of attacks. These methods can leverage network capabilities enabled by Cisco routers as well as provide for additional opportunity to the Clean Pipes provider for additional dedicated, clean capacity into the destination network. The following diagram maps the DDoS Protection flow in Clean Pipes

16 Migration to Cisco Clean Pipes 2.0 As the Cisco Guard and Detector Modules approaching the end of their life cycle, the Arbor Peakflow SP Threat Management System (TMS) replaces the scrubbing technology in the clean pipes solution. The migration from the Cisco Guard to the Threat Management System (TMS) can be achieved through a mapping of protected resources configured in the Cisco Guards (Zones) to Peakflow SP (Managed Objects) as well as deployments of TMS in similar architectures as the Cisco Guard. Understanding the terminologies of Cisco Guard and Arbor PS/TMS and knowing how to map them will greatly increased knowledge transfer from the successful deployments of Clean Pipes 1.0/1.5 with Cisco Guard appliance/module for successful migration to Clean Pipes 2.0 with Arbor TMS for mitigation. The table below shows the key concepts mapping between these two technologies. Cisco Guard Arbor Peakflow SP Common Definition TMS Zone Managed Object The basic models that builds baseline, detection, mitigation and reporting. Model definitions are flexible and can be combined with many layers of data to specifically match critical areas of interest on the network. These models are used to monitor customer, peer, service, or profiled relationships. - Boundary A boundary demarcation point between administrative domains. Peakflow SP immediately builds a topological map of the monitored network using the network definition as the default global boundary. Boundaries are flexible and can be inherited such as a global boundary or specifically configured relative to the Managed Object monitored. The global boundary defines the point traffic enters or exists the monitored network. Baseline Baseline Collection of traffic behavior profiles building the expected traffic volume and anomaly detection thresholds. Protect mode / Zone Protection Enabled Mitigation Enabled Scrubbing configuration. This can include details about the destination, BGP prefix used to 16

17 change traffic path, active and passive filtering rules. Filter Countermeasure - Rule describing an evaluation of traffic to be scrubbed by the solution Template Mitigation Template - Preset configuration information used to protect a destination from specific vectors or to use specific filters in protection. Diversion Off-ramp - BGP Prefix announcement of destination with a change in Next-hop attribute to ensure inbound traffic will pass through the scrubbing solution. Re-injection On-ramp - Returning the cleaned traffic post processing from the scrubbing system to the network in a loop-free path to the protected destination Cisco Anomaly Guard and TMS Countermeasure Comparison Cisco Anomaly Guard and Arbor Peakflow SP TMS provide for significant mitigation capabilities based on available countermeasures. These countermeasures are designed to deal with the current common DDoS attack types as well as provide a capability to protect a target from a zero-day attack as well. Both solutions provide strong protection from spoofed source attacks, resource consumption attacks at the application layer, session layer or network layer. Both solutions provide solutions to brute force flooding attacks such as packet per second floods, TCP connection floods, UDP and ICMP floods to name just a few examples of common attacks. TMS provides for advanced capabilities to filter web-enabled (HTTP-based) services through authentication, validation, request tracking and limits as well as payload filtering. Each of these countermeasures, or any set of countermeasures can be brought into service across a group of TMS systems through the configuration of a single mitigation. This capability sets the TMS apart from previous mitigation solutions for large, distributed network deployments. Mitigation Countermeasure Function TMS Cisco Guard White list / black list filtering Per source IP rate thresholds TCP SYN Authentication with reset to Host TCP SYN Authentication with refresh sent to host TCP SYN Authentication with HTTP Authentication TCP SYN Authentication with safe reset to host 17

18 TCP SYN ACK Authentication TCP other flag authentication Basic/Default authentication of other protocols for client based on passed TCP authentication Strong Mode TCP Authentication using proxy * DNS Authentication through packet drop / re-transmission DNS Authentication By Reflexive-Redirection DNS Request Type Limiting By Source /32 DNS Cache Poison Defense * Target Release 5.1 Q Target Release 5.1 Q Target Release 5.1 Q DNS Authentication by converting to TCP Target Release 5.5 Q Strong Mode DNS Auth using TCP and TTL Regex based filtering DNS DPI REGEX Filtering HTTP Header REGEX Filtering Protocol baseline enforcement Source /24 based baseline enforcement Connection metrics based baseline enforcement Rate limiting Malformed HTTP Malformed SIP Malformed DNS SIP Authentication SIP source request thresholding Target Release 5.5 Q X Further Enhanced Target Release 5.1 Q

19 HTTP source IP rate thresholding HTTP source IP object get rate thresholding TCP Idle Timeout TCP multiple bad connection blacklisting * Requires the mitigation system to be in-line in both directions Despite the differences in specific filtering options available, each of the mitigation systems have demonstrated an ability to effectively mitigate the types of threats that are seen on the Internet today. Cisco and Arbor performed joint testing on the mitigation capabilities of the TMS system and the tests concluded that the TMS was able to mitigate the same types of attacks that the Guard was capable of mitigating. It sometimes used similar methods as the Guard solution and sometimes used different methods that were proven as effective. Clean Pipes 2.0 Deployment Considerations The Clean Pipes 2.0 solution provides a wide variety of protection values. - Managed DDoS Detection and Protection Services: Arbor Peakflow SP provides a complete solution for providers to offer a turnkey managed DDoS service. Managed objects monitoring each component of the subscribed customer network and services. - Managed Hosting DDoS Protection: Protection of critical services at the application layer with unique capabilities to further differentiate increasingly sophisticated attacks at the application layer itself. Managed services offerings can be offered granularly to customers per application or critical service that may be protected or protection of the customer site entirely. - Peering Point Bandwidth Protection: Detection and mitigation capabilities at the peering points can protect against collateral damage from transit of DDoS attacks. - Network Service Protection: Arbor Networks TMS can provide service specific application reporting as well as dedicate DDoS protection to critical network services. Netflow considerations for Peakflow SP Netflow telemetry is an intrinsic part of the Peakflow SP solution. Peakflow SP leverages Netflow data to provide operators complete visibility into network traffic characteristics and rates, create baselines of normal traffic, detect deviations from these baselines that may be due to threats, characterize the threats and trace them back to network borders. In short, Netflow provides the basic building blocks for each of the 6 phases of infrastructure security. 19

20 Pervasive monitoring of the network is a necessary component of the Clean Pipes solution so Netflow telemetry should be exported from all Cisco routers where critical data may traverse including those in the network peering, core, distribution and data center. It is recommended that ingress Netflow be enabled on all logical interfaces for each router being monitored. This provides the Peakflow solution with a full picture of what traffic is going through the router regardless of direction. Peakflow has the ability to focus on desired traffic within each Netflow data stream so it is not necessary to filter what traffic is sent to the Peakflow SP collectors. Unsampled netflow provides accurate flow information on the network traffic which can be leveraged by features such as ATF and fingerprints that rely on matching certain behaviors for more effective triggers. However, when the network traffic load is high, unsampled netflow can limit the scalability of anomaly detection due to the excessive processing work load on the flow reporting devices and the Peakflow appliance. In this case, sampled netflow can be implemented as a way of facilitating a better scalability. The appropriate sampling rate to apply is a function of what type of router it is, how much traffic is going through the router and what line cards are available in the router. Sampling rates can go over 1000:1 in large hardware accelerated platforms (CRS 1, Catalyst 6500). In general the more traffic going through, the higher the sampling rate that would be applied. When exporting Netflow from routers that are carrying IPv6 or MPLS traffic, it is necessary to utilize Netflow v9 with IPv6 and MPLS explicitly enabled to get visibility into this traffic. The following are specific considerations when enabling Netflow on Cisco devices: Netflow versions 5, 7 and 9 are supported on Peakflow devices. Netflow export should always be set to a 1 minute active flow timeout ensuring real time analysis can be done on longer lived connections. On Catalyst switches, full interface flow mask should be used to ensure that all Netflow fields available will be populated Peakflow SP supports a single sampling rate per network device so do not configure multiple sampling rates on a single router. Deployment of Peakflow SP PI The Peakflow PI appliances provide users direct access to the data stored and distributed in the collector devices in the network. The PI functions as a leader to coordinate all data reports, alerts and system health, data from the individual collectors and present that data in a unified view. The PI appliance provides for secure access to user, administrators and service customers groups. Each PI deployed provides access to the deployment through the GUI or API. PI appliances support hot/hot active redundancy of up to 10 devices and together act as a unified access point into the deployment. Administrative, DDoS Alert and API access is synced between PI systems 20

21 to ensure each system provides equal level of access based on centralized access, accounting and user privileges. For the hardware enforced scalability limits and the guidelines for scaling the Peakflow SP PI deployment, refer to Appendix-B. Deployment of Peakflow SP Collectors Peakflow SP collectors provide distributed data collection and detection in the network deployment. Detection for specific customer or services on the network can be done through the processing and correlation of netflow information from any point that traffic passes through the network. This provides the Peakflow system the ability to scale to the network cloud for detection and reporting. By intelligently matching traffic to the object definition within the system each configured managed object baseline is developed constantly and reported virtually. Placement of the collector is critical to ensure that detection and reporting visibility are optimized for the protection and reporting desired. Pervasively monitoring peering capacity, external border connectivity and long haul capacity typically provides value in increased transit optimization. Protecting the network from off-net or transiting attack traffic is often the first level of protection successful Managed DDoS services as well as Peering point protection. Deployments of Collector platforms at the aggregation or within the core of the network ensure that on-net traffic does not become a threat customers or network assets from internal attack. This layer of detection can be critical in isolation of internal network attacks, customer-tocustomer attacks, capacity issues, and critical network service monitoring. Setting network Netflow settings for sampling rate, export timing, Netflow export locations and pervasive enablement all affect the detection and reporting of data on the network. Peakflow SP can provide for very accurate detection and reporting through sampled Netflow processing. Evaluating the correct level of sampling ratio relies on both the ability of the router platform and the software version its running. Recent versions of IOS and Cisco router infrastructure provide extremely accurate and high performance platforms for Netflow export to Peakflow SP Collectors. Peakflow SP Collectors also manage downstream systems such as TMS. TMS reports health and mitigation statistics as well as additional Netflow data back into the Peakflow SP collector. All inter-device communications are carried out over SSL. Through this secure connection, the collector and TMS have a dedicated link to exchange detection data, baselines, thresholds and mitigation configurations. This link is secure and can be maintained over geographically separate devices. The following diagram illustrates deployment concepts of Peakflow SP CP and TMS 21

22 For the hardware enforced scalability limits and the guidelines for scaling the Peakflow SP CP deployment, refer to Appendix-C. Deployment of Peakflow SP TMS Deployment of the mitigation component of the Clean Pipes solution can be optimized to fit your specific network architecture. Considerations of the specific protection values will influence the deployment of the TMS into the network. The TMS can be deployed within the Peering layer of the network or adjacent to the network border to ensure that off-net attacks are mitigated directly at the network edge. This provides for an advantage of a direct solution to peering point threats and the ability to maintain distributed attack traffic separated to ensure that it does not threaten the network through an aggregation of this traffic. Central scrubbing capacity can be deployed in centralized locations to provide a consistent experience to protection services customers. Regional mitigation or scrubbing centers provide dedicated locations where diverted attack traffic can be scrubbed and returned to the ultimate destination of the threat traffic. Providing regionally or geographically based mitigation capacity can provide protection services to regional customers avoiding excessive backhaul of traffic. Dedicated TMS deployments can provide advanced value for specific infrastructure, customers or services. Dedicated systems can be deployed directly in data centers adjacent to the resource 22

23 can ensure that application reporting, service specific reports, performance data, change alerts and packet level forensics that can aid in both validation of normal operation but also improve troubleshooting and application visibility. Data center placement of TMS appliances also provides protection for critical resources from sources within the network that may not be protected by peering and central scrubbing locations. Broadband consumers, infected hosts within the network and customers networks present a potential threat to network resources and must be considered when building defenses. The following diagram depicts TMS network deployment locations: Diversion of traffic into the TMS systems is most often triggered by an IP traffic routing change such as a BGP announcement changing the attack target destination route to a next-hop of the mitigation systems available. BGP route maps, anycast route announcement, community attribute setting can all be used to design a specific solution to the problem of diverting traffic into central or distributed TMS systems within the network for mitigation. TMS provides the ability to group devices into a single event to gain efficiency of central management as well as provide scale to the mitigation capabilities. Policy based routing as well as static routes may also provide local traffic diversion into the system. In some cases diversion can also be accomplished through local techniques at layer 2 to ensure that traffic is passed through the TMS. Policy based routing architectures, static ARP table entries and VLAN mappings all can provide for layer 2 diversions into the TMS systems. Reinjection of traffic post processing through the TMS must avoid the diversion method used to ensure that a routing loop does not occur. This reinjection is typically accomplished through GRE encapsulation from the TMS to the provide edge or CPE device. Another well known method is to configure a MPLS VRF instance to separate the forwarding data from the diversion 23

24 segment and the reinjection segment. Last, VLANs can be used to separate the forwarding path if the destination is reachable through a layer 2 domain. Successful reinjection of cleaned traffic into the network establishes an alternative traffic flow of the attack traffic from the normal data path to a clean virtual pipe anywhere within the network. For the hardware enforced scalability limits and the guidelines for scaling the Peakflow SP TMS deployment, refer to Appendix-D. Peakflow SP Communication Ports For Peakflow SP to function properly, the appropriate communication ports must be allowed through the network devices and firewalls. For all appliances, the following ports must be allowed: NTP SNMP management (recommended) Management console < > CP/FS/PI/TMS Data collection The following ports must be allowed through the firewalls so that the Peakflow SP CP, FS, and TMS appliances can collect data: Port BGP 179 TCP UDP flows any (default) Traffic flow appliance < > routers router > CP router > FS SNMP CP > router FS > router Inter appliance communication (all appliances) The following ports must be allowed through the firewalls so that the CP, FS and TMS appliances can communicate with other appliances: Port ArborFlow (31373 UDP) Traffic flow FS > CP appliance TMS > SP appliance (if you have ArborFlow enabled) 24

25 HTTPS 443 SSL (configurable) CP browser > SP leader appliance CP/FS/PI > CP/FS/PI Peakflow SP PI and leader appliances The following ports must be allowed through the following firewalls for the PI and leader appliances: Port HTTPS 443 Remote Arbor services Traffic flow CP browser > SP leader appliance remote services HTTPS/443 routeviews (off by default) ATF anonymous statistics fingerprint sharing Local services DNS NTP AAA/TACACS Scaling Clean Pipes 2.0 Peakflow SP provides the largest network scale available today. Using a many to one monitoring model made possible by using Netflow, the system can scale to monitor over 2200 routers within the network. As more and more collectors are deployed for visibility each collector increases the total number of routers monitored. As the number of collectors increases another layer of the detection hierarchy can be added to the solution to further increase its ability to provide detection across very large networks of routers. The Flow Sensor provides collector level functions at the aggregation level of the network. This extension of the collector platform further increases the ability of Peakflow SP to provide 25

26 pervasive coverage of both the external border of the network as well as the internal aggregation edge for detection. TMS provides a range of capacity in the mitigation portion of the solution ranging from 1.5 Gbps to 10 Gbps throughput. Collectors can manage up to 50 TMS systems within a single deployment. Peakflow SP allows for up to 1000 native managed objects on a system for baseline, threshold and reporting data. This number can be scaled to a total of 10,000 managed objects monitored on a single deployment with the addition of Business Intelligence appliances. This additional component provides scale to the number of managed objects in incremental blocks of 500 to grow the monitoring and detection capabilities to keep pace with network and service growth. Appendix A through D provides the scalability numbers and guidelines for the Peakflow SP system and each individual appliance. Best Practices in Cisco Clean Pipes 2.0 General deployments of Clean Pipes Solutions follow a set of known best practices to ensure the successful delivery of the service. Operational experience and problems have improved on the best practices employed in dealing with attacks over the course of the last decade the Internet has been experiencing both a growing frequency and severity of the DDoS attacks. - Operationalize the Six Phases of Security Best Practice: Jointly developed by Arbor Networks and Cisco Systems, the six phases of security is a framework for continuous assessment, action and improvement. Following this process of preparedness and improvement is a critical step to deploying any security solution, including a Clean Pipes solution. - Maintain a separate and secure management network for the Clean Pipes solution. The communication between detection and mitigation components in the solution is most critical during an attack so a separate, isolated management network is required to ensure that communication is maintained during adverse conditions. - Out of band management to the devices in the Clean Pipes solution must be employed to ensure that network operators will have full access to the systems at all time and can perform necessary maintenance without significant outage. - Build sufficient mitigation capacity: Minimum capacity requirements are typically equal to the amount of available bandwidth at the network border. Typically providers target enough capacity to protect critical infrastructure and match that capacity to ensure that peering capacity can be cleaned. This ensures that attacks that overwhelm the capacity of a given resource can still be effectively mitigated. - Ensure comprehensive path detection for the network resources protected: Ensure that any critical path of traffic to or from critical network resources have a level of detection that will measure traffic threshold violations. 26

27 - Minimize traffic backhaul and maintain distribution of the attack traffic: Traffic backhaul and aggregation of the attack traffic may actually result in a portion of the network being overwhelmed. Managing the attack traffic in a distributed manor will limit the possibility that the traffic will overwhelm network resources. - Manage attacks as near the source of the attack as possible: Trace-back traffic to its source and limit the network s exposure to that traffic will limit collateral damage and impact of the event on the network. - Build redundancy into the mitigation solution such that the network will remain protected even if one or more mitigation devices become unreachable - Drop known threat traffic as quickly as possible: Bringing operational and historical knowledge of attack vectors and their sources can aid in the amount of traffic that must be analyzed when protecting a network service such as DNS, VoIP or Web Services. Being able to run a course-grained filter before a more surgical filter is employed will limit the amount of advanced scrubbing capacity required to mitigate attacks. - Automate response and traffic redirection when ever possible: Automation will increase response time to network events and attacks. Where possible, course filters, traffic diversion and traffic scrubbing can be automated to ensure rapid response to threats. - Maintain reports and data from the alert through the mitigation for analysis: Reporting and comprehensive history of mitigation actions provide learning opportunities for future events. 27

28 Conclusion Clean Pipes solutions have been rolled out throughout the world as an answer to established and evolving DDoS threats. These solutions have been a result of the close relationship between Arbor and Cisco Systems to both develop as well as leverage each company s technology to bring about a better combined solution. Clean Pipes 2.0 continues to provide the best available solution to emerging DDoS threats. Clean Pipes 2.0 now focuses technology from each of the providing partners into their core competency and furthers the development of these technologies by focused and expertise of each vendor. The partnership between Cisco Systems and Arbor Networks continues to provide best of breed solutions to customer problems and evolve existing solutions to the next phase of capabilities. 28

29 Appendix Appendix A: Peakflow SP System wide Enforced and Guideline Limits The following table includes the system wide enforced limits: Type Limit CP appliances 35 Important: You must assign a PI appliance as the leader if you have five or more CP, TMS, or BI appliances in your deployment. Up to 30 CP appliances are supported in a single deployment. FS appliances TMS appliances 150 (maximum of 5 per CP appliance) 50 (maximum of 5 per CP appliance) Note: Up to 3 stacked TMS 2700 appliances count as 1 appliance toward this limit. PI appliances 10 Third Party Firewalls Monitored routers 10 (Cisco Guard) 2,250 (maximum of 5 per CP appliance or 15 per FS appliance) Monitored interfaces 100,000 Managed objects 10,000 Note: If you have more than 1,000 managed objects, then you must add Business Intelligence (BI) appliances for additional managed object storage. Mitigation templates 1,000 Fingerprints (2.0) 100 Applications 500 Note: These are also bound by the managed object limit. 29

30 The following table includes the system wide guideline limits: Type Concurrent logins per PI appliance on a 10 PI load. Concurrent logins to the leader appliance Limit 125 (requires multiple PI appliances) 10 Configured users 500 SOAP queries per minute, per appliance 200 Simultaneously active DoS alerts 1,000 BGP traps 100 Active fingerprints 20 Managed objects with filters 20 Reports 500 completed or up to 20 GB of disk space Report templates 500 Active mitigation actions 1,000 Note: This includes blackhole offramps, TMS mitigations, Cisco Guard offramps, and Flowspec mitigations. CIDR group prefixes 50,000 Note: This limit applies to CIDR entries across all CIDR groups, including duplicates. Unique CIDR blocks across all CIDR groups for all managed objects 2,000 30

31 Number of prefixes per CIDR group entry 100 Example: If you have a CIDR group called Datacenter that has three prefixes ( /24, /24, /24), then you can add prefixes, but you cannot exceed 100 total. Auto configuration rules 5,000 Archived alerts 100,000 Alert deletions per day 2,000 Multisite members 15 Services 50 CIDRs defined per service

32 Appendix B: Peakflow SP PI 5500 Appliance Enforced and Guideline Limits The following table includes the enforced and guideline limits for the PI 5500 appliance: Type Enforced Guidelines Limit Non-leader PI supports all mitigation configuration except thirdparty Leader PI supports all mitigation configuration, including thirdparty Supports software upgrades from CP 5000 but may require new hardware 200 automated SOAP queries per minute 250 managed services systems (system-wide) 50 simultaneous users per PI appliance 120 simultaneous users (system-wide) 100 Web 2.0 API objects Appendix C: Peakflow SP CP 5500 series Appliance Enforced and Guideline Limits The following table includes the enforced limits for the CP 5500 series of appliances: Type Model Limit Data sources (routers or appliances) CP Note: Up to 3 stacked TMS 2700 appliances count as 1 appliance toward this limit. Interfaces seen All 20,000 Interfaces monitored All 10,000 Mitigation slots CP CP Note: Up to 3 stacked TMS 2700 appliances count as 1 appliance toward this limit. 32

33 CP OSPF area All 1 Simultaneous DoS alerts All 300 The following table includes the guideline limits for the CP 5500-series appliances: Type Limit Input: flows per second 50,000 (supports peak rates of up to 100,000 fps) Note: Flow throughput depends on how many managed objects match as in/out per flow. The current supported flow limit assumes that no more than five managed objects match per flow. Input: ArborFlows per second from the TMS and FS appliance Forensic flows Steady state BGP routes Local managed objects per CP appliance 50,000 (supports peak rates of up to 50,000 fps) Up to four days Note: This is constrained by the system wide limit. 2 million (300,000 per every router) 500 Note: This limit is constrained by the system limit. Ongoing DoS alerts 300 BGP flaps per second 1,000 33

34 Appendix D: Peakflow SP TMS Appliance Limits TMS 3100 and 3110 appliance limits The following table includes the application monitoring and mitigation limits for the TMS 3100 and 3110 appliances: Type Offramping or inline traffic Offramping or inline traffic Limit 10 Gbps 8 Mpps Ongoing mitigations per appliance 50 Interfaces (physical, logics, sub-interface / VLAN) 1,000 TMS 3050 appliance limits The following table includes the application monitoring and mitigation limits for the TMS 3050 appliance: Type Offramping or inline traffic Offramping or inline traffic Limit 5 Gbps 3.5 Mpps Ongoing mitigations per appliance 50 Interfaces (physical, logics, sub-interface / VLAN) 1,000 34

35 Appendix E: Six Phase Approach to Infrastructure Security Cisco and Arbor advocate a six-phase framework for deploying security systems. The six phases are: Preparation Detection Classification Traceback Reaction Post mortem While the six-phase approach was designed primarily to counter DDoS attacks, this framework provides a good overall approach to securing service provider environments. Preparation Phase Preparation is probably the most important of the six phases. This phase includes setting up both technical and nontechnical processes, tools, and organizational structure that constitute the security system. The tasks in the preparation phase include: Select, develop, install, and test the security tools and techniques you will use. Define and agree upon security policy and incident response procedures. Set up communications channels with service provider peers and customers, and establish equipment vendor incident response teams. Identification Phase In the identification phase, you detect unusual activity or behavior and activate appropriate measures after an alert is raised. You can use many tools and data sources to identify these issues, including NetFlow information, SNMP information about the CPU, and interface 35