Experiences in Building a 100 Gbps (D)DoS Traffic Generator

Transcription

1 Experiences in Building a 100 Gbps (D)DoS Traffic Generator DIY with a Single Commodity-off-the-shelf (COTS) Server March 31, 2018 Umeda Sky Building Escalators Surasak Sanguanpong Surasak.S@ku.ac.th

2 About me University Computer Engineering Head of Applied Network Research Lab Chairman of UNINET Network Monitoring Working Group Electronics Transactions Committee (DE Ministry) Interesting Areas Internet System Security Traffic Analysis and Measurements ISP-Application Collaboration 2

3 About This Talk How to DIY a 100 Gb/s (D)DoS traffic generator? HW and SW solutions What are the underlying technology and techniques? Theory and Tools What are lessons learned from the deployment? Experiences and Outcomes

4 Goal and Constraints Full 100 Gb/s [~100 Mpps] Capability Running on a single COTS server Running on a single 100 GigE NIC Closed Network Deployment and Testing with Synthetic Traffic

5 Outline PART I: Introduction DDoS Understanding Ethernet Revisiting & Update PART II: HW and SW Solution Hardware Components OS and Software Tools PART III: Testbed and Performance Results Throughput CPU Utilization PART IV: Lesson Learned Experiences Outcomes Related Projects

6 PART I Introduction Understanding DDoS

7 2018: Welcome to the New Tb/s DDoS Era! Biggest-Ever 1.35 Tb/s DDoS Attack Hits Github Feb 28, 2018 Misconfigured Memcached servers to amplify DDoS Memcached Amplification Attack Breaks New 1.7 Tb/s DDoS Arbor confirms a 1.7 Tb/s attack targeted at a customer of a U.S. based ISP Mar 5, 2018 Source: ~91,500 Simultaneous HD TV channels Source:

8 DoS Single Source

9 DDoS Simulating this!

10 Broadly types of DDoS Volume Based Attacks To saturate the bandwidth of the attacked site Measured in bits per second (bps) Protocol Attacks To consumes target resources, or intermediate communication equipment (firewalls, IPS, Load balancers, etc.) Measured in packets per second (pps) Application Layer Attacks Mostly low-and-slow attacks to crash targets Measured in requests per second (rps)

11 PART I Introduction Ethernet Revisiting & Update Understanding Ethernet Wire Speed and Throughput Calculations

12 Evolution of Ethernet Capacity and speed requirements on data links keep increasing 40,000X in 34 yrs 40,100 Gb/s 25 Gb/s IEEE Std 802.3bs 200, 400 Gb/s Servers have begun to be capable of sustaining 100 Gb/s to memory 10 Mb/s 100 Mb/s 1 Gb/s 10 Gb/s

13 Theoritical 100 GigE Characteristics (Wire Speed) Frame Type Frame Size Max Packets Max Bandwidth Frame Duration Minimum 64 bytes Mpps Gb/s 6.72 ns Maximum 1518 bytes 8.1 Mpps Gb/s ns

14 The Frame sizes matter 1 Smallest : Minimum Frame Size S S S S S S S 1 second (High Rate, Low Volume) 2 Largest: Maximum Frame Size L L L 1 second (Low Rate, High Volume)

15 Ethernet frame by frame delivery 84 to 1,538 bytes 64 to 1,518 bytes to 1, PA SFD DA SA Type Payload FCS IFG PA SFD Frame Frame 64 bytes* Minimum Frame Size 7+1+( )+12 = 84 bytes (672 bits) 1,518 bytes* Maximum Frame Size 7+1+( ,500+4)+12 = 1,538 bytes (12,304 bits) * Excluded 20 bytes :- PA:7+SFD:1+IFG:12)

16 Maximum Frame Rate for 100 GigE Max bytes M = Speed/Size = 100x10 9 / (84*8) = 148,809,523 pps Maximum throughput T = M*64*8 = Gbps Max bytes M = Speed/Size = 100x10 9 / (1,538*8) = 8,127,438 pps Maximum throughput T = M*1,518*8 = Gbps

17 Theoritical 100 GigE performance Maximum Bandwidth Gb/s #Frame #Frame (1,518B) Mb/s 987 Mb/s Gb/s 9.87 Gb/s Gb/s 98.7 Gb/s Maximum Frame Rate Gb/s #Frame (@64B) #Frame (1,518B) M 81 K M 812 K M 8.1 M Frame Duration 1/(148.8x10 6 ) = 6.72 ns Frame Duration 1/(8.1x10 6 ) = ns

18 Timing and CPU budget in 100 GigE Time (ns) ,518 1,518 3 GHz Clock 30th cycles 60th cycles 90th cycles 330th cycles

19 PART II HW and SW Investigation: A COTS Server with Multicores CPU is it capable?

20 To Delivery 100 GigE with COTS 100 GbE

21 Performance Characteristics of Buses GbE CPU 3 4 Four Crucial components 1 CPU Multicores, Multithread High Clock Speed 3 PCI Bus PCIe 3.0 Gb/s PCIe 4.0 Gb/s 2 Interconnection 4 Gb/s Gb/s Memory Bus DDR4-2400MHz Quad Gb/s DDR4-2666MHz Six Gb/s

22 Yes!, the hardware is capable. Next : SW investigation, focusing on OS Kernel & Network Stack

23 OS s obstacle Traditional OS network stacks is problematic Not design with this speed in mind Many features essential for networking filtering, connection tracking, memory management, VLANs, overlay, and process isolation Not scalable even many CPU cores these days

24 Overhead in Linux kernel Socket based system calls Context switching and blocking I/O Data copying from kernel to userspace Interrupts Handling High latency! Linux stack designed as control plane not data plane NOT SCALE! Linux Network Stack Walkthrough (2.4.20)

25 How to solve this obstacle? Solution: Kernel Bypass

26 Conventional Stack V.S. Kernel bypass Let s bypass kernel and work directly with NICs Allows access to the hardware directly from applications Using a set of libraries for fast packet processing Reduces latency with more packets to be processed Handles packets within minimum number of CPU cycles But Provides only very basic set of functions (memory management, ring buffers, poll-mode drivers) Require reimplementation of others IP stack features Conventional (Sockets based) User Application Sockets Kernel TCP/IP Stack Network Driver Hardware Kernel Bypass (RDMA based) User Application Packets Library Kernel TCP/IP Stack Network Driver Hardware

27 Zero Copying (ZC) with RDMA Conventional (Sockets based) Kernel Bypass (RDMA based) User User Application Application App buffer Shared buffer Data copy Packet Libraries Kernel Sockets TCP/IP Stack Network Driver Sockets buffer Data copy Device buffer Data copy Kernel TCP/IP Stack Network Driver ZC with Remote Direct Memory Access Hardware Hardware

28 Fast (Userspace) Packet Processing DPDK Netmap PF Ring OS Linux, FreeBSD FreeBSD,Linux Linux License BSD BSD LGPL + paid Language C C C Use Case Appliances, NFV NFV, Router Packet Capture, IDS/IPS NIC vendors Several Intel Intel Supports Community Community Company Kernel bypass also known as Fast Packet Processing High-Performance Packet IO Data Plane Processing Acceleration Framework

29 DPDK (Data Plane Development Kit) A set of libraries and drivers for fast packet processing Main Libraries multicore framework huge page memory ring buffers poll-mode drivers Originally developed by Intel Currently managed as an open-source project under the Linux Foundation

30 DPDK Architecture DPDK Programmable Packet Processing Pipelines

31 DPDK based Open Source Projects Virtual multilayer switch integrated into various cloud platform Carrier-grade, integrated, open source platform to accelerate Network Function Virtualization (NFV) SPDK pktgen-dpdk Libraries for high performance, scalable, user-mode storage applications Original DPDK traffic generator Packet-journey Linux router IO services framework for the network and storage software with Vector Packet Processing Linux scalable software routers, proved with 500k routes The Stateful traffic generator for L1-L7 Flexible stateless/stateful traffic generator for L4-L7

32 TRex DPDK based stateful/stateless traffic generator (L4-L7) Replay of real traffic (pcap), scalable to 10K parallel streams Supports about mpps per core, scalable with the number of cores High scale benchmarks for stateful networking gear (Firewall/NAT/DPI) Generating high scale DDOS attacks High scale, flexible testing for switches Scale to 200 Gb/s for one COTS Scale tests for huge numbers of clients/servers

33 PART III Testbed and Performance Measurements

34 Testbed HW: Two Rack Servers Xeon GHz, 10-cores 64 GB RAM (4x16 GB DDR GHz) 1.5 TB NL-SCSI PCIe Gen3x16 2 ports 100 GigE NIC Sender 100 GigE Receiver OS&SW CentOS 7.3 Kernel 3.10 DPDK TRex 2.29

35 TRex sample configuration file 65,535 clients talking to 255 servers trex: ~/trex-core/scripts# cat cap2/imix64.yaml - duration : 1.0 generator : distribution : "seq" clients_start : " " clients_end : " " servers_start : " " servers_end : " " clients_per_gb : 201 min_clients : 101 dual_port_mask : " " tcp_aging : 0 udp_aging : 0 cap_info : - name: cap2/udp_64b.pcap cps : ipg : rtt : w : 1

36 Trex Console

37 Testbed bytes UDP packets with random 65,535 source IP address to 255 destination IP address Throughput V.S. #CPU Cores Throughput V.S. #CPU Cores CPU Utilization V.S. #CPU Cores CPU Utilization V.S. #CPU Cores

38 Throughput bytes Theoretical Max: 76.2 Gb/s Theoretical Max: pps

39 CPU bytes

40 PART IV Lesson Learned and Related Projects

41 Why DDoS traffic generator? DDoS Detection Traffic Analytics Traffic Profile Usage Behavior 6 Projects in 4 Groups to be Introduced Router IDS, IPS Test Tools Packet Processing Core Traffic Log Law Enforcement Accounting Firewall Load Balancer Data Exfiltration Deep Packet Inspection Quota Control IoT Discovery Protocol Discovery

42 (1) DDoS Detection/Mitigation Model Packet Guardian In progress R&D Inline 100 GigE Stateless DDoS Detection/Mitigation Internet Gateway Router 100 GigE Experiments SYN Flooding and simple P2P Detection Results: 90 Mpps Detection Capability Research Tasks: Investigation of Efficient Detection/Mitigation Methodology HW/SW optimization techniques Core Router Internal network

43 (2) HTTP Flood Detection (1x100 GigE) PCAP traffic replaying Pure HTTP-GET flood attacks with NO background packets Detection against 86K signatures Gen 100 GigE Detector Gb/s Gb/s 8.3 Mpps E5-2640v4 10 GHz Preliminary Results: 31.1 Gb/s 86K Signatures

44 (3) HTTP Logger (10x10 GigE) PCAP traffic replaying HTTP packets with background packets Inspection and log only HTTP 31.1 Mpps 99.5 Gb/s Gen #1 2x10 GigE Gen #2 6x10 GigE Logger GHz Gen #3 2x10 GigE

45 (4) Traffic Logger Performance Real Deployment in 10 Gb/s Campus Network Real-time HTTP and Packet Header Log Repository for Data Analytics Peak 2,100 req/s (33GB/day) Data Lake Statistics Peak 380,000 req/s (330 GB/day) Sample HTTP Log format X X TCP GET /index.html Sample Packet Header Log format :53: X X 1514 TCP x :53: X X 90 UDP Billion records (Total 2.57 TB) 3.27 Trillion records (Total TB) ELK Stack as Indexing Platform with 80K/s/machine Indexing Rate

46 (5) Traffic Analytics

47 (6) Traffic Accounting/Control Ads Track sessions and flow for counting BW usage once login Login Sessions IPv4 and IPv6 # of Active Sessions 65X,XXX Concurrent Flows Dual Authen Max Burst Today s Usage One Click Session Termination All Active Address

48 Lessons Learned Server is really faster than you think! Faster, Better Use latest PCIe Gen3x16 slots Faster CPU clock speed is rather more preferences than number of cores Reducing inter-processor communication cost is a key Required in-depth understanding of packet I/O C code implementation

49 Summary Generic OS with default network stack: Incapability of handling 100 GigE saturated with smallest frame Proved Solution: Data Plane Fast Packet Framework COTS Server is capable for 100 GigE Rising trend SW based appliances for high speed network COTS Security Appliance based Fast Packet Framework

50 Thank you for your attention Collaboration and Students Recruitment Welcome! Q&A Q & A Time Sunset at Narita Airport