Project Proposal. ECE 526 Spring Modified Data Structure of Aho-Corasick. Benfano Soewito, Ed Flanigan and John Pangrazio

Transcription

1 Project Proposal ECE 526 Spring 2006 Modified Data Structure of Aho-Corasick Benfano Soewito, Ed Flanigan and John Pangrazio 1. Introduction The internet becomes the most important tool in this decade that impact to all of the aspect from to the internet banking. In today s internet, worms and viruses cause service disruptions with enormous economic impact. Current attack prevention mechanisms rely on enduser cooperation to install new system patches (Windows Update) or upgrade security software (anti virus update), yielding slow reaction time. However, malicious attacks spread much faster than users can respond, making effective attack prevention difficult. Network-based mechanisms, by avoiding end-user coordination, can respond rapidly to new attacks. Such mechanisms require the network to inspect the packet payload at line rates to detect and filter those packets containing worm signatures. Network Intrusion Detection System is well suited for this purpose. Today Intrusion Detection System (IDS) techniques are usually classified as either signature detection or anomaly detection. In this project we only focus on signature detection. Signature detection is based on matching events to signatures of known attacks. IDS can monitor packet in the network traffic for security purposes by scanning the packet payload to detect malicious intrusions or attack signatures. Currently, most IDSs are software based running on a general purpose processor. SNORT is a popular open source IDS has thousands of rules in which the rules refer to the header as well as to the packet payload. A sample Snort rule is shown below, Alert tcp $BAD 80 -> $GOOD 90 (content: perl.exe ; msg: detected perl.exe ;) The rule examines the protocol, source and destination IP address, source and destination TCP port. The content option indicates that the packet payload is to be matched against the string enclosed in double quotes, in this example is perl.exe. The continued growth in both network traffic and intrusion signature database makes the IDS increasingly difficult for software running on general purpose processor to keep up with increasing network speeds that approach 10Gbps. There needs to be a new form of intrusion detection system that can handle this type of speed and load. In this project we will study and implement Aho-Corasick, the famous string matching algorithm for searching multiple strings in one pass using a trie with backpointers. This algorithm needs a lot of space in memory to hold the trie data structure that contains the rule and also the back pointers. We will modify the data structure of the algorithm using the bitmap in order to reduce the space for this algorithm and finally this system can put in the faster memory (instead of a general purpose processor) that can make the searching become faster. 2. Previous work String matching algorithm is one of the well studied classical problems and has been extensively studied for many years. Most known IDS implementations use a general purpose string matching algorithms, such as Boyer-Moore, Aho-Corasick and Bloom Filter. In this chapter we would like to explain the basic idea of several string matching algorithms.

2 2.1. Boyer-Moore The algorithm of Boyer and Moore [2] is widely used algorithm for string matching. The algorithm compares the pattern with the text from right to left. The idea is reduce the large number of comparison the string. It has two heuristic are triggered when mismatching occurred. The basic idea as follow, if the text symbol that is compared with the rightmost pattern symbol does not occur in the pattern at all, then the pattern can be shifted by m positions behind this text symbol where m is the length of the pattern. The following example illustrates this situation. a b b a d a b a c b a b a b a c b a b a c The first comparison d-c at position 4 produces a mismatch. The text symbol d does not occur in the pattern. Therefore, the pattern cannot match at any of the positions 0,..., 4, since all corresponding windows contain a d. The pattern can be shifted to position 5. The best case for the Boyer-Moore algorithm is attained if at each attempt the first compared text symbol does not occur in the pattern. Then the algorithm requires only O(n/m) comparisons. Bad character heuristics This method is called bad character heuristics. It can also be applied if the bad character, i.e. the text symbol that causes a mismatch, occurs somewhere else in the pattern. Then the pattern can be shifted so that it is aligned to this text symbol. The next example illustrates this situation. a b b a b a b a c b a b a b a c b a b a c Comparison b-c causes a mismatch. Text symbol b occurs in the pattern at positions 0 and 2. The pattern can be shifted so that the rightmost b in the pattern is aligned to text symbol b. Good suffix heuristics Sometimes the bad character heuristics fails. In the following situation the comparison a-b causes a mismatch. An alignment of the rightmost occurrence of the pattern symbol a with the text symbol a would produce a negative shift. Instead, a shift by 1 would be possible. However, in this case it is better to derive the maximum possible shift distance from the structure of the pattern. This method is called good suffix heuristics.

3 a b a a b a b a c b a c a b a b c a b a b The suffix ab has matched. The pattern can be shifted until the next occurrence of ab in the pattern is aligned to the text symbols ab, i.e. to position 2. In the following situation the suffix ab has matched. There is no other occurrence of ab in the pattern. Therefore, the pattern can be shifted behind ab, i.e. to position 5. a b c a b a b a c b a c b a a b c b a a b In the following situation the suffix bab has matched. There is no other occurrence of bab in the pattern. But in this case the pattern cannot be shifted to position 5 as before, but only to position 3, since a prefix of the pattern (ab) matches the end of bab. We refer to this situation as case 2 of the good suffix heuristics. a a b a b a b a c b a a b b a b a b b a b The pattern is shifted by the longer of the two distances that are given by the bad character and the good suffix heuristics Aho-Corasick Aho-Corasick (AC) [1] is a multi-string matching algorithm, meaning it matches the input against multiple strings at the same time. Multi-string matching algorithms generally preprocess the set of strings, and then search all of them together over the input text. The algorithm consists of two parts. The first part is the building of the tree from keywords you want to search for, and the second part is searching the text for the keywords using the previously built tree (state machine). Searching for a keyword is very efficient, because it only moves through the states in the state machine. If a character is matching, it follows goto function otherwise it follows fail function. The root node is used only as a place holder and contains links to other letters. Links created in this first step represents the goto function, which returns the next state when a character is matching. To construct goto function, we shall construct goto graph. We begin with a graph consisting of one vertex which represents the state 0. We then enter each keyword y into the graph, by adding a directed path to the graph that begins at the start state. New vertices and edges are added to the graph so that there will be, starting at the start state, a path in the graph that spells out the keyword y. The keyword y is added to the output function of the state at which the path terminates. We add new edges to the graph only when necessary. For example, suppose {he, she,

4 his, hers} is the set of keywords. Adding the first keyword to the graph and continue will obtain the graph like shown below in the keyword tree. During the second phase, the fail and output functions are found. The fail function is used when a character is not matching and the output function returns the found keywords for each reached state. For example, in the text "SHIS", the failure function is used to exit from the "SHE" branch to "HIS" branch after the first two characters (because the third character is not matching). Figure 1. The keyword tree Figure 2. The tree with the fail function 2.3. Bloom Filter Dharmapurikar et al. proposed a multiple-pattern matching solution using parallel bloom filters [3]. Their approach can handle thousands of patterns. The proposed scheme builds a bloom filter for each possible pattern length. A Bloom filter is a data structure that stores a set of signatures compactly by computing multiple hash functions on each member of the set. This technique queries a database of strings to check for the membership of a particular string. The answer to this query can be false positive but never a false negative. An important property of this data structure is that the computation time involved in performing the query is independent of the number of strings in the database provided the memory used by the data structure scales linearly with the number of strings stored in it. Furthermore, the amount of storage required by the Bloom filter for each string is independent of its length. Each Bloom filter scans the streaming data and checks the strings of corresponding length. Whenever a Bloom filter detects a suspicious string, an analyzer probes this string to decide whether it indeed belongs to the given set of strings or is a false positive. Based on the analyzer s determination, the system can take appropriate action (either drop, forward, or log) for the string s associated packet. Let the signature lengths range from Lmin to Lmax. The Bloom filter engine reads as input a data stream that arrives at the rate of one byte per clock cycle. It monitors a window of Lmax bytes, as shown in Figure 3. When this window is full, it contains Lmax Lmin substrings, which are potential matches for signatures. The system verifies the membership of each substring, using the appropriate Bloom filter. Each hardware Bloom filter gives one query result per clock cycle. In this way, the system can verify the memberships of all the Lmax Lmin strings in a single clock cycle. If none of the substrings match a signature, the data stream can advance by a byte. Monitoring a window in this way eventually scans all the possible strings of length from Lmin bytes to Lmax bytes in every packet. In the case of multiple substrings matching within a single window, the longest substring becomes the string of interest, a policy called longest substring first (LSF). Thus, in the case of multiple matches at the same time in the array of Bloom filters, the analyzer probes the substrings, from longest to shortest. The search stops as soon as the analyzer

5 first confirms the match of a substring. After the search is over, the window advances by a byte, and the system repeats the same procedure. Figure 3. Window of streaming data containing strings of lengths from Lmin=3 to Lmax=W. [4] From those three string matching algorithm we can summarize as follow [5] Idea Computation Storage Problem Boyer-Moore Skip O(m*n) worst 0.1 MB (10K Rules) Aho-Corasick Trie O(n) worst 50 MB (1500 Rules) Bloom Filter Approximate O(n) 0.1 MB searching (10K Rules) Shift table needed Storage False Positive Base on this study, the Aho-Corasick has a good performance in the worst case but the problem is the storage, it needs 50 MB to hold 1500 rules. So in this project we would like to decrease the storage for Aho-Corasick using methodology that will explain in the next chapter. 3. Methodology The previous section discussed the previous approaches to string matching algorithms. We will now discuss the Deterministic Memory-Efficient String Matching Algorithm for Intrusion Detection [4]. First we will build the tree data structure that will create the state machine; this will be done with next pointer function or goto function. This is the most important part in this project. We will modified the next state pointer that has 256 next state pointer in Aho-Corasick algorithm become only use one pointer and encoded the 256 next state pointers using a bitmap compression

6 scheme. This will greatly reduce the memory space but the on other hand will a produce a slight increase the execution time. The pseudo code Bitmap Data Structure is shown below, struct bitmap_state { struct bitmap_state * next_state; bitmap next_state_valid : 256; struct bitmap_state * failure_state; struct rule * rule_list; }; Figure 4. Pseudo code bitmap data structure The diagram of the state node is shown in figure 5. The next pointer points to the bar of the children node. In this figure we have an example if the next character id D, then we transition from the current state by first checking to see if the fourth bit from the left in the bitmap is set and we assume an alphabet where A=1, B=2,... Z=26. Finding that it is, we know that there is a valid transition at some offset from the next pointer. We then count all the set bits prior to bit four in the bitmap, and find that there is only one of them and therefore our offset from the next pointer is one. We add the size of one node to our next state pointer, jump to that data structure which is the correct node for our D transition, and examine the next character in our packet. If on the other hand, our next character was C, we would look in the bitmap and see that C was disabled. We would then follow the failure pointer and repeat the check with C on whatever node it pointed to. Figure 5. Diagram Bitmap Data Structure We also construct the rule pointer that will point to the rule file that content the string that would to match. This rule pointer will construct from each final state of the state engine. The last pointer that we construct is the fail pointer. To construct fail pointer we will using pseudo code in figure 6 below. The failure function is constructed from the next pointer function. Let us define the depth of a state s in the next pointer graph as the length of the shortest path from the start state to s. Thus in Figure 7, the start state is of depth 0, states 1 and 3 are of depth 1, and states 2, 4, and 6 are of depth 2, and so on. We shall compute the failure function for all states of depth 1, then for all states of depth 2, and so on, until the failure function has been computed for all states (except state 0 for which the failure function is not defined).

7 begin queue ~ empty for each a such that g(o, a) = s ;~ 0 do begin queue ~ queue LI {s } f(s) ~ 0 end while queue ~ empty do begin let r be the next state in queue queue ~-- queue - {r} for each a such that g(r, a) = s fail do begin queue ~ queue t2 {s } state ~ f(r) while g (state, a) = fail do state ~ f ( s t a t e ) f(s) ~ g(state, a) output(s) ~ output(s) U o u t p u t ( f ( s ) ) end end end Figure 6. Failure function Figure 7. Pattern matching machine for the set of keyword {he, she, his, hers} The algorithm to compute the failure function f at a state is conceptually quite simple. We make f(s) --0 for all states s of depth 1. Now suppose f has been computed for all states of depth less than d. The failure function for the states of depth d is computed from the failure function for the states of depth less than d. The states of depth d can be determined from the non fail values of the next pointer function of the states of depth d- 1. Specifically, to compute the failure function for the states of depth d, we consider each state r of depth d 1 and perform the following actions. 1. l f g ( r, a) =failfor all a, do nothing. 2. Otherwise, for each symbol a such that g(r, a) -- s, do the following: (a) Set state = f(r).

8 (b) Execute the statement s t a t e ' - f ( s t a t e ) zero or more times, until a value for state is obtained such that g ( s t a t e, a ) # f a i l. (Note that since g(o, a) # fail for all a, such a state will always be found.) (c) Setf(s) --g(state, a). For example, to compute the failure function from Figure 7, we would first set f(1 ) = f(3) = 0 since 1 and 3 are the states of depth 1. We then compute the failure function for 2, 6, and 4, the states of depth 2. To compute f(2), we set state = f(1 ) = 0; and since g(0, e) = 0, we find that f(2) = 0. To compute f(6), we set state =f(1 ) = 0; and since g(0, i) = 0, we find that f(6) = 0. To compute f(4), we set state = f(3) = 0; and since g(0, h) = 1, we find that f(4) = 1. During the computation of the failure function we also update the output function. When we determine f(s) = s', we merge the outputs of state s with the outputs of state s'. For example, from Figure 7. we determine f(5) = 2. At this point we merge the output set of state 2, namely {he}, with the output set of state 5 to derive the new output set {he, she}. 4. Expected Results With the implementation of the Aho-Corasick algorithm the size of the rule set for the Snort Sensor IDS is expected to decrease significantly. According to the preliminary results shown in the material the memory should be reduced by a factor on the order of 20. This reduction will depend upon the rule set implemented. Our goal is to implement the Snort IDS with the Aho-Corasick algorithm as well as the modified Aho-Corasick algorithm and test the memory changes with multiple rule set sizes. A plot of the number of rules to the size of the rule set for the modified and unmodified Aho-Corasick algorithm will be created. In addition to the graphic results, the speed of the modified algorithm will be tested and compared to the base Aho-Corasick algorithm. These results will be plotted against the size of the rule set and the amount of traffic processed by each algorithm. 5. Project Summary The first task is the implementation of the modified Aho-Corasick algorithm in the Snort Sensor software. After this programming is complete the benchmarking of the software can begin. Ideally the results will show that the rule set can be reduce to a size that would be small enough to implement on a network based IDS. This would require the rule set to be very small. The rule set will need to be on the order of 1MB so that the rules can be stored in a fast SRAM. The implementation of the system on a generic system will allow ease of implementation without having to re-write the entire Snort package for a network processor. This will also allow the Snort package to be tested very ruinously without worrying about the SRAM vs. DRAM and having to simulate multiple memory configurations. The goal is to just test the changes to the Aho-Corasick algorithm. The implementation of the Snort sensor on a generic processor will allow testing of the speed of the algorithm in addition to its memory usage. This data will allow predictions as to the amount of additional overhead that is necessary for the bitmap calculations. These results can be further used to make predictions to the fesablility as well as the areas of the Aho-Corasick algorithm that need further improvement.

9 6. References [1] A.V. Aho and M. J. Corasick. Efficient string matching: An aid to bibliographic search. [2] R. S. Boyer and J. S. Moore. A fast string searching algorithm. [3] S. Dharmapurikar, et al., Implementation of a Deep Packet Inspection Circuit using Parallel Bloom Filters in Reconfigurable Hardware. [4] G. Varghese, T. Sherwood, N. Tuck and Brad Calder. "Deterministic Memory-Efficient String Matching Algorithms for Intrusion Detection [5] Class Notes