Dept. of Electrical and Computer Eng. : Trie Bitmap Content Analyzer for High-Speed Network Intrusion Detection N. Sertac Artan and Jonathan H. Chao 8 May 27 26th Annual IEEE Conference on Computer Communications
Outline of the Talk Introduction Data Structure Introduction to Algorithms for for Intrusion Detection and Prevention Implementation and Performance Conclusion 5/8/7 INFOCOM 27 2
Introduction
Network Intrusion Detection and Prevention NIDS NIPS Like a security camera Like a security guard On-the-side Inline Only monitors the traffic Both monitors and police the traffic Cannot modify the traffic Internet Internet IPS IDS Protected Network 5/8/7 Protected Network INFOCOM 27
The attack is known and defined by a signature Assumptions A signature is a payload excerpt to identify a particular attack No zero-day attacks considered Signatures are contained within a single packet 5/8/7 Our previous work deals with detecting signatures spanning multiple packets INFOCOM 27
Deep Packet Inspection Slide a window on a packet payload Advance the window one byte at a time Compare the window content against all signatures Per-byte processing has stringent time-constraint: 2 ps for 4-Gbps line Packet Hdr Attack signatures database Payload Attack { Signature SQL Slammer 726e5686f756e MyDoom 57696e646f7773 5/8/7 INFOCOM 27
Motivation: The memory bottleneck Goal: Small and constant worst case time Some options: Bottleneck External Memory 5/8/7 On-chip memory is limited Signatures Finite Automata: Too big TCAM: Too expensive Around Mbits for stateof-the-art FPGAs Signatures stored in slow external memory INFOCOM 27 7
Trie Bitmap for Content Analysis
Our Contribution : Trie Bitmap Content Analyzer FPGA Signatures External Memory 5/8/7 Space-efficient and fast Signatures fit into a fraction of an FPGA No external memory access No memory bottleneck Guaranteeing one operation/byte Allows multiple parallel engines on a single chip to boost performance No reconfiguration. Easy updates through on-chip memory updates Uses minimal perfect hashing INFOCOM 27 9
Ordinary Hashing vs. Minimal Perfect Hashing U U S Hash Function I S S S Collision: Unpredictable query time Memory > S 5/8/7 I = S U >> I = S = S ) U >> I O( S ) S Hash Function S INFOCOM 27 S Collision Free: Predictable query time Memory: Same as S
Binary Trie as an Address Decoder b b b2 b3 b4 b5 b6 b7 b4 b5 b6 b7 b b b2 b3 log(n) levels b7 b6 b7 b5 b6 b5 b4 b3 b2 b4 b3 b2 b b Addr= 5/8/7 b b INFOCOM 27
Node Structure b b b2 b3 b4 b5 b6 b7 Hash Function 5 Data Bitmap (DB) Group to left child 5/8/7 Next Node Bitmap (NB) Group to right child INFOCOM 27 2
: Successful Query b6 b5 b4 b2 b b3 b b7 b7 b6 H b7 H2 b5 b3 b2 b4 Compare b5 b7 b6 b b3 b2 Match b b4 b5 b4 b3 b2 b b b b H H2 b b Addr= 5/8/7 b b b6 b H INFOCOM 27 3
: Successful No-Match Query b6 b5 b4 b9 b3 b b7 H b7 b6 b9 H b7 H2 b2 b b5 b3 b2 b4 Discard b5 b6 b b b3 b2 b4 b7 b6 b5 b4 b3 b2 b b Addr= 5/8/7 b b INFOCOM 27 4
Equal-Partitioning
Offline Data Structure: The Occupancy Table The Occupancy Table Bin No. 4 Items 2 3 Data Bitmap (DB) Next Node Bitmap (NB) A Node Occupancy Item List - - 2 2,4 3-4 - 5 3 6-7 2 Constraint: All items hashed to the same bin must go to the same next node 5/8/7 INFOCOM 27 6
Cast into Number Partitioning Problem The Occupancy Table Bin No. 5/8/7 INFOCOM 27 Occupancy Item List - - 2 2,4 3-4 - 5 3 6-7 2 7
Easy or Hard? As, Item values decrease Number of items increases Chance of equal-partitioning increases Hayes formulated hardness as, > n/2 = 8 6 Total w/q w : The number of bits required to represent the largest number q : is the count of items Total Since for26 hashing, q is expected to be higher at high levels w increases slower compared to n Equal-partitioning at high levels is easier than lower levels 26 Total 5/8/7 INFOCOM 27 8
Easy or Hard? b b b2 b3 b4 b5 b6 b7 b5 b4 Items / node decreases b b b2 b3 b4 b5 b6 b7 b7 b6 Node count/level increases b3 b2 Max. occupancy/n increases b b Possible partitionings decrease Harder to solve Use naïve algorithms for high-levels of Use Brute-force for low-levels 5/8/7 INFOCOM 27 9
Partitioning High-levels: Blackjack Algorithm Partition 8 items into two 4 item groups Start Sum: 2 3 Occupancies Next Node Bitmap (NB) 3 6 3 4 If fails, change starting point If all starting points exhausted, change hash function 5/8/7 INFOCOM 27 2
Partitioning Low-levels: Greedy Algorithm Let s go back b b b b2 b b3 b b2 b3 b b2 b3 b b2 b3 b Collision No partitioning Higher a node in the trie, higher the expected number of different equal-partitionings Expected possible equal partitionings for a node with 6 items and load factor.5 is over 8 5/8/7 INFOCOM 27 2
Partitioning Low-levels: Greedy Algorithm For each node, 2 2 3 3 q q2 q3 Occupancies q = Q = 5, qmax = max (n, m) q4 q5 Try all 2n 2 subsets of Q for sum = n/2 {q}, {q2}, {q3}, {q4}, {q5} {q, q2}, {q, q3}, {q, q4}, {q, q5}, {q2, q3} {q, q2, q3}, {q, q2, q4}, {q, q2, q5}, 5/8/7 INFOCOM 27 22
Success probability using Greedy Algorithm for the last 4 nodes 5% chance of equal-partitioning with a single set of random hash functions for a set of, signatures 5/8/7 INFOCOM 27 23
for NIDPS
Snort Signatures Signature length between to 22 bytes Total of 655 unique signatures 5/8/7 INFOCOM 27 25
Preparation Chop signatures into fixed-length chunks and ID these chunks For c = 4, Signature ABCDEFGHIJKLMN ABCD ID 5/8/7 EFGH ID2 IJKL ID3 INFOCOM 27 MN Note the short suffix ID4 26
Detecting Long Signatures Sig: ABCDEFGHIJKL, c = 4 S Detector ABCD ID EFGH ID2 IJKL ID3 ABCD S Detector 2 S, ID S S, ID2 S2 S2, ID3 S3, Match EFGH S2 Match / No Match IJKL Input S3 ABCDEFGHIJKL Detector S, ID ID Concatenate S,ID S Match / No Match Detector 2 State S 5/8/7 INFOCOM 27 27
Detecting Long Signatures Sig: ABCDEFGHIJKL, c = 4 S Detector ABCD ID EFGH ID2 IJKL ID3 ABCD S Detector 2 S, ID S S, ID2 S2 S2, ID3 S3, Match EFGH S2 Match / No Match IJKL Input S3 ABCDEFGHIJKL Detector S, ID2 ID2 Concatenate S,ID S Note other offsets (e.g. BCDE) have their own current state 5/8/7 INFOCOM 27 Match / No Match Detector 2 State S2 28
Detecting Long Signatures Sig: ABCDEFGHIJKL, c = 4 S Detector ABCD ID EFGH ID2 IJKL ID3 ABCD S Detector 2 S, ID S S, ID2 S2 S2, ID3 S3, Match EFGH S2 MATCH! Match / No Match IJKL Input S3 ABCDEFGHIJKL Detector S2, ID3 ID3 Concatenate S,ID S2 Match / No Match Detector 2 State S3 5/8/7 INFOCOM 27 29
Implementation and Performance
Case Study Snort Signature detection using 2 stages achieved using 5/8/7 29 kbits for high-levels 36. kbits for low-levels and hash keys 65. kbits in total Construction time: 8 seconds on a 2.8 GHz Pentium-4 3 MHz clock speed on a Xilinx Virtex2Pro FPGA Gbps using a Xilinx Virtex2Pro FPGA INFOCOM 27
5/8/7 Rehash Operations Required INFOCOM 27
Demonstration Setup IDS Sensor GigE User Browsing Monitor s Web Server UDP Alert Messages...2..3.7 Internet Traffic Generator 5/8/7 IDS Monitor/ Web Server INFOCOM 27 33
Demonstration Setup Test Setup IDS Monitor/ Web Server Intrusion Detection And Prevention System (IDPS) Traffic Generator INFOCOM 27 34
Conclusion 5/8/7 A high-speed low-cost Deep Packet Inspection method for NIDS is proposed Over Gbps throughput with single Virtex2Pro FPGA Gbps proof-of-concept design is validated on hardware 4 Gbps is believed to be achievable using today s state-of-the-art FPGAs No external memory needed INFOCOM 27 35
5/8/7 Q&A INFOCOM 27 36