Jaal: Towards Network Intrusion Detection at ISP Scale A. Aqil, K. Khalil, A. Atya, E. Paplexakis, S. Krishnamurthy, KK. Ramakrishnan University of California Riverside T. Jaeger Penn State University P. Yu, A. Swami US Army Research Lab 1
Is IDS Needed at ISP Scale? Increasing number of network attacks Distributed, span entire WANs Unnoticed until too late Mirai botnet Mirai exploits vulnerable devices spread across the internet to launch DDoS Sep 2016: Krebs on Security (620 Gbit/s), OVH (1Tbit/s) Oct 2016: multiple attacks on Dyn, affected Twitter, Github, Airbnb, Netflix, others Nov 2016: Liberia s internet infrastructure 2
Is IDS Needed at ISP Scale? Simple two step attack: scan then flood Hardcoded default passwords control vulnerable devices (scanning a large set of IP addresses) Compromised devices also repeat the scan Launch coordinated attack on targets at the bot master signal Inherently difficult to detect Scanning activity observable only at ISP level DDoS prevention works best deep in the network, where the pipes are the largest and the capability to identify and block the attacks is the most evident Bruce Schneier, security expert 3
ISP Scale IDS is Challenging State of the art NIDS (e.g., Snort, Bro) are effective But expect to inspect all packets Works only at enterprise scale Problematic at ISP scale: Multiple ingress/egress points To create global view required for analysis, information collected from multiple vantage points needs to be aggregated Challenge: how to aggregate? 4
Aggregation Approach I Copy and forward to central engine Simple, but lead to performance degradation Percentage decrease 100 50 Performance degradation as traffic replication increases Avg decrease in throughput Worst decrease in throughput Drop in accuracy 70% Tput loss 0 0 10 20 30 40 50 60 70 80 90 100 Percentage of traffic replicated 5
Aggregation Approach II Sample and forward to central engine Already used by ISPs for heavy-hitter detection Efficient but achieves poor detection accuracy for general attacks Attack Reservoir Sampling Distributed SYN Flood 54% Sock Stress 60% SSH Brute Force 42% 6
Aggregation Approach III Create sketches and forward to central engine Targeted measurement approach Strong resource/accuracy guarantees Lacks generality: need one sketch for every measurement task For TCP/IP header, need 2 18 different sketches to capture all possible measurements 7
Jaal Design Goals Design an ISP-scale NIDS that: Can detect wide array of attacks requiring global view, using signatures similar to Snort s Focus on TCP/IP header-based attacks Does not require copying and forwarding raw packets (minimizes bandwidth overhead) 8
Jaal Overview III- Flow assignment: Assigns flows to monitors Load balancing I- Monitors: Filter target flows Process packet batches, create summaries Packet Filtering Summatization Monitor Load Info. Summaries Assignments Flow Assign. Inference Assignments Load Info. Summaries Packet Filtering Summatization Monitor II- Inference engine: Collects summaries Performs pattern matching NIDS Rules Decision 9
Summarization Goal: produce a representative summary of packets Enables high accuracy detection of attacks using general signature Light weight, low BW overhead all flows all flows assigned flows packets batch batch summary packets mode fields mode 10
Summarization (cont.) Two step summarization SVD to reduce fields mode Clustering to reduce packets are mode the left sin batch X = U V T eliminate small singular values packets mode n SVD k-means summary k centroids counts fields mode p r 11
Inference Summary Individual S m 1 or Sm 2 Aggregator S a Config. summaries d, c question vectors NIDS q Translator Similarity Rules Estimator Q Config. h, v Postprocessor Inference Engine Alert, Q Feedback Alert Collect individual summaries (push or pull) Transform NIDS rules (normalization, marking irrelevant fields) Estimate similarity Feedback: request finer grained summary to improve performance Estimate variance (e.g. port scans, DDoS) 12
Flow Assignment Requirements: Cover all flows Each flow is processed by exactly one monitor (for correct operation) Balance load to the extent possible Simple/Fast algorithm Challenge: Flows can start/terminate at any time, vary in packet rate Packet rates unknown a priori Solution: Model as constrained online load balancing problem Simple greedy algorithm, (empirically) close to optimal 13
Evaluation Implemented on in-house high performance SDN-testbed Two Realistic RocketFuel topologies (~350 routers) Complex topologies created by instantiating Open vswitches instances connected via virtual links Two ISP backbone traces from MAWI group as background traffic + inject malicious traffic Five different attacks: DoS: SYN flood DDoS: distributed SYN flood Port scans: distributed port scans Brute forcing: distributed SSH brute forcing Sockstress most common attack classes 14
Evaluation (cont.) 98% average TPR @ 9% FPR and only 35% BW overhead (with feedback) Summarization parameters n, k, r set by studying ROC curves Each point has a different n: batch size, r: rank, k: centroids r = 14 retains most information in fields mode n 600, k 0.2n, r 12 enables high detection accuracy 15
Evaluation (cont.) Simulating Mirai progression Scanning on ports 23, 2323 Randomly select a source node + 150 vulnerable nodes Jaal detects the scan with 95% accuracy Number of Infected devices vs time Number of infected devices 150 100 50 Unchecked infections Remaining infected devices after Jaal 0 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 Time (s) 16
Conclusion ISP scale NIDS is needed in the face of large scale attacks State of the art NIDS inadequate at ISP scale Jaal presents a major step forward in developing ISP scale NIDS Uses dimensionality reduction and clustering Centralized pattern matching on packet summaries Achieves high detection accuracy at low bandwidth overhead 17
Thanks! 18