VARIABLE-LENGTH HASHING FOR EXACT PATTERN MATCHING

Transcription

1 VARIABLE-LENGTH HASHING FOR EXACT PATTERN MATCHING Dionisios Pnevmatikatos Foundation for Research and Technology-Hellas Institute of Computer Science Vasilika Vouton, Heraklion, Greece Aggelos Arelakis ECE Department Technical University of Crete Chania, Greece {pnevmati, ABSTRACT HashMem is a memory based, exact pattern ing architecture for SNORT-like intrusion detection. It uses style functions to determine a unique location for a possible and then es the input against the pattern stored in the specified memory location. This approach achieves is a very low logic and a reasonable memory cost. In this paper we extend the HashMem architecture to allow storing of variable-length patterns in a single memory structure, reducing the number of required memory structures and comparators. In this way, we improve the density of the memories and reduce the necessary logic for functions and comparators. These improvements allow V- HashMem to accommodate the newest SNORT rule-set with modest memory and very low logic cost of about 0.06 logic cells per search pattern character. This logic cost is almost an order of magnitude smaller compared to other research. Variable length HashMem uses single-ported memories, thus allowing the simultaneous processing of two characters per cycle using the FPGA dual ported memories and additional logic. We also extend the V-HashMem architecture to include a Header-ID field and support header ing information, a feature missing both from our own earlier work and many related systems. 1. INTRODUCTION Research in Network Intrusion Detection System acceleration has flourished in the last years, and many papers have been published on hardware and reconfigurable acceleration of SNORT [1] pattern ing systems. The importance of inspection and early warning systems is easily justified. In order to be successful, proposed acceleration techniques have to achieve low logic and memory cost, high processing throughput, and ensure scalability. Scalability is the ability to accommodate of future search pattern set expansion while maintaining constant processing throughput, and is also a cost indicator, since better scalability allows a fixed hardware device to last longer before becoming obsolete. The HashMem pattern ing architecture [2], is a SNORT accelerator architecture based on the idea that a simple hash function of the input can generate a set of sparse but distinct addresses for the search patterns. We use this address as a pointer to a possible ing string, and compare that to the input to determine the final signal. The HashMem architecture was successful in achieving both reasonable memory use (in the order of Xilinx memory blocks depending on the configuration), and low logic cost (in the order of 0.15 LC/character). One limitation of HashMem is that a separate memory structure is needed for each distinct pattern length leading to significant memory and logic cost. Also, the memory utilization is heavily dependent on the number of search patterns of that length. To address these two limitations and further improve the already low logic and memory cost of HashMem we introduce Variable-Length HashMem or V-HashMem. The goal is to allow the storage of several close string lengths (say for example 5-7 characters) to be stored in a single memory structure thus sharing memory space. In addition to reducing the memory cost, V-HashMem also reduces the required logic since each such structure uses a single -generator and comparator logic. Furthermore, we extend our architecture to accommodate header-ing information and report es only when the packet header es a desired set of conditions, a limitation shared by both our own earlier work as well as several other previously published works. The rest of the paper is organized as follows. Section 2 gives a short overview of the basic HashMem architecture. Next in section 3 we present our V-HashMem extensions, and discuss the implementation details. Section 4 presents our implementations results comparing them to other published results. Finally, section 5 gives an overview of related work and section 6 offers our conclusions. 2. THE HASHMEM ARCHITECTURE The basic idea of HashMem, is to use a simple -type hash function on the input pattern to generate a unique candidate pattern address. For example, lets assume we want to a set of patterns of length L characters. We feed L input bytes into a hashing module to generate the unique address of the candidate pattern. We then read the candidate X/06/$20.00 c 2006 IEEE.

2 input input L=4 Index Mem Index MEM Pattern Mem String Memory L=4 Compare L=4 L=3.. 4 Chars 10 Chars Index Pattern Mem Index MEM Mem L=4 L=10 Variable 3-4 Compare Variable 5-10 Compare L= Fig. 1. The HashMem Architecture. For each pattern length N, N input characters are hashed to produce the memory address of a single possible candidate pattern. The Index memory redirects the sparse hash addresses into a compact (but wider) Pattern Memory. If the stored pattern es the input pattern then we have a. pattern from the memory and compare it with the (delayed) input to verify the. This operation is depicted in Figure 1. This process operates only for patterns of length L. Since any given character of the input stream can be the last character of a pattern of arbitrary length, we use the simple approach of replicating the entire structure once for each of the different pattern lengths. The HashMem architecture has several advantages. First, it relies on memories to hold the search patterns, exploiting the FPGA dense and fast memory blocks. This leads to improved speed, but more importantly decouples the size of the logic from the size of the search pattern set. HashMem is not the only memory based ing architecture; proposals such as Cho s [3], Sourdis s [4] and Attig et al. [5] also suggest storing the patterns in memories. The second advantage of HashMem is the low logic cost and stems primarily from the simple hashing functions that it uses. We have used simple, XOR-based -type hashing functions. This approach is simple and has small implementation cost. It is also very flexible: depending on the polynomial used, each function will produce a different mapping of patterns to locations. This gives us a systematic approach of adapting to different pattern sets, that is, given a set of patterns, we can select a polynomial that produces distinct values for every search pattern. To facilitate the search for an appropriate hash function, we allow a sparse address space (experimentally we found that 12 bits are sufficient for Snort patterns). The index memories (shown in Figure 1) point to compact pattern memories so that the sparseness is limited to the (few) index bits and not to the wide pattern storage. 3. VARIABLE-LENGTH HASHMEM: ARCHITECTURE AND IMPLEMENTATION Variable-Length HashMem (or V-HashMem) is based on the observation that for efficiency purposes, it is advantageous to store the search patterns of different lengths in a single memory structure. If this is possible, then we need fewer Fig. 2. V-HashMem showing two structures of sizes 3-4 and 5-10 characters. The generation is performed at the smaller length, while the comparator takes into account the don t care bits for the variable portion (character 4 for the first structure, and 6-10 for the second). memory structures, i.e. less memory, as well as fewer generators and comparators, i.e. less logic. The V-HashMem targets exactly these factors. However, this extension raises the following issues that have to be addressed. Hash function generation: HashMem uses all L characters of the input to hash the address of the possible. If we allow for several lengths, how do we know how many characters to use for the hash calculation? Comparators: Similar to the previous issue, how do we know how many characters to between the input and the stored pattern? The first issue is addressed by observing that the hashing to produce an address is actually a hint, rather than a certain fact. The hashed value is a pointer to the possible, and has to be unique for each stored pattern in the set. Other than being unique, no other property of the hashed value is used: the actual is determined by the comparators. Therefore, we do not necessarily need to use all the L input characters to produce the hashed value, merely enough to guarantee a unique value. In the V-HashMem environment, that means that it is sufficient to use only the minimum length of the stored patterns for hashing purposes, without knowing the exact length of the candidate pattern. For example, assume we have a structure that holds patterns of lengths 3, 4 and 5. We can use a function of 3 characters, as long as we can ensure the the hashed values for the first 3 characters of all the stored patterns (whether they are 3, 4 or 5 characters long) are unique. This approach if successful, is also beneficial in terms of cost: not only there is the potential of having a single generator instead of three in our example, but the generator will also be the narrower of the three (3 versus 4 or 5 character). The second issue is easier to tackle: while we cannot know in advance which input characters may which

3 pattern length, we do know in advance the length of each of the search patterns we are looking for! So it is straightforward to encode this information in the memory and subsequently use it during the comparison. The simplest way of encoding the pattern length is to add don t care bits in the variable portion of the stored characters. In our example, the first 3 stored characters are ed always and do not need marking, while the next two are optional and need a don t care bit each. This extension leads to a small increase in the stored memory bits, and also to a small increase in the cost of the comparators that now have to consider the don t care bits. This overall structure is similar to a ternary CAM, but at the granularity of a character. Figure 2 shows the overall V-HashMem architecture for two example structures that can store patterns of 3-4, and 5-10 characters each Discussion Storing variable-length patterns in a memory structure is clearly beneficial, but is not always possible. As we described earlier in our example, to store multiple patterns in the single structure we have to ensure the the hash values for the first 3 characters of all the stored patterns (whether they are 3, 4 or 5 characters long) are unique. A prerequisite for this condition is that the 3 character prefixofall patterns stored in the structure are distinct, because otherwise they would result in the same hash value. To put in another way, we could not store in this structure the patterns abc, abcd nor the patterns abcd, abce. All these patterns have a common prefix that would result in a hash value conflict. While the restriction in the first set of patterns may be acceptable (any input ing abcd also es abc so there will be a reported ), the second conflict set must be supported somehow. How common is this situation in the SNORT rule-set? Common prefixes do exist, but as we will see soon in the experimental evaluation section, it is possible to deal with common prefixes with small cost overhead. One solution for this problem is to make the memory structure hold overlapping length ranges. The idea is that if we have two structures, one for 3-5 characters and another for 5-7 characters, in case of a conflict between a 3- and a 5-character pattern, we can move the 5 character pattern in the other structure (5-7) and remove the conflict. Another technique could be to provide multiple copies of the same range, for example, having two copies of the 3-5 structure. This approach offers an easy solution to handling conflicts, but at the cost of duplicating the entire structure. These two techniques can be used in conjunction, giving considerable flexibility in dealing with these potential conflicts. In the evaluation section we will see that in practice removing all the conflicts is possible with reasonable cost. Yet another approach to handle conflicting patterns is to remove them from the memory structures and them with external logic-based circuitry (for example DCAM [6]). Input Index Port A Port B Pattern Port A Port B == == Fig. 3. Doubling the processing throughput requires only the replication of the generators and comparators. This solution is guaranteed to work always but at the expense the logic area cost, and was not adopted in this work. Finally, another independent parameter in deciding the length of the structures is the memory utilization. Since the pattern length distribution is known, one can size the various memory structures so that the utilization of each memory structure is satisfactory. Such an approach does reduce the memory usage but might introduce conflicts when the number of structures decreases Supporting Header Match information A complete SNORT-like NIDS system combines two pieces of information: header ing with payload scan. It is easy to see that payload scan is by far the most difficult and complex task, since header ing generally involves merely equality or range ing on fixed numeric fields. However, payload scanning should consider the header ing information and report es only when the combined check of header and pattern are found. To include this functionality, we can add a Header Group ID field along with the search pattern in the pattern memories. This ID will determine the set of search patterns that are compatible with the current packet header. Our earlier research [7] indicates that the snort rules can be classified into 300 groups, which can be encoded with 9 bits. Upon arrival of a new input packet, the header ing circuitry performs its tests and provides the Header-ID to the V-HashMem sub-system. The pattern checking occurs as described earlier, but to report a the comparators also test the Header-ID field for equality with that of the incoming packet. This test suppresses false positive answers when the search pattern is not compatible with the header of the packet. The additional cost of this feature is the memory bits to store the Header-ID and the additional comparator logic, and as we will see is affordable in modern FPGA devices Processing Two Characters per Cycle One advantage of V-HashMem compared to our earlier Hash- Mem, is that it achieves a more compact memory representation. In HashMem to achieve better memory utilization we resorted to advanced sharing techniques; to achieve this sharing, HashMem used both read ports of the Xilinx mem-

4 ory blocks. V-HashMem merges multiple lengths in the same memory structure and treats them as a single length, therefore using only a single memory port. The unused second memory read port can be used to increase the throughput of the architecture, processing two patterns at offsets 0 and 1 per cycle. This improvement comes at the cost of doubling the logic for generation and comparators. The overall structure for a given pattern length is shown in Figure 3 and is described in detail in our earlier work [2] Reusing Logic and Memory for Wide Patterns The SNORT rules contain some very wide search patterns, that present additional inefficiencies in their implementation. Even using the widest Xilinx memory blocks (512x36 bits), storing a few wide patterns is very inefficient. Furthermore, most of the HashMem logic cost comes from generators and pattern comparators. To address this inefficiency, we used a partial ing technique where a wide pattern is broken down into a series of more manageable partial patterns or fragments. The fragments are ed in the regular memory structures, and custom glue logic combines the partial signals to determine whether the entire wide pattern was actually found in the input. Determining the actual involves delaying the partial signals appropriately to indicate the actual position on the partial pattern in the overall input. This technique has been used in our earlier work [2], while a similar technique has been also adopted by Song and Lockwood [8]. Given the number of wide patterns (624 patterns wider than 18 characters), using a per fragment solution as in Hash- Mem is inefficient. Instead we used an addressing convention that places fragments in consecutive locations, so that when the first fragment of a wide string is located say at location x, the expected location of the subsequent fragment is already known to be at location x +1. Then we compare the delayed expected address to the address of the second fragment and determine the. Since this rule holds for all fragments of all patterns, the cost of the approach is to add one to the address and broadcast it (appropriately delayed) to all memory structures for future inspection. There are however rules that when broken down to fragments create tree-like structures. Consider the case of two patterns abcdef and abcxyz. A of abc followed by a of either def or xyz is a full. This fan-out structure deviates form the +1 rule. This situation is relatively rare (178 patterns that form 41 trees in our rule set), and we solved it by adding a small exception memory in parallel to each pattern memory. In our example we would place the abc and def fragments in locations x and x +1. Then we would place the xyz fragment at another location y and place the entry x +1in location y in the exception memory, indicating that a at location y after ing location x +1is a full. Finally we stored 5 control bits along with the actual pat- Table 1. V-HashMem Memory structure configuration. Width Num. of Width (Chars) Mem Blocks Range Patterns Comp. Index Data Header-ID A B Total HashMem tern to differentiate between first, middle, and last fragments in the sequence, and to indicate the source structure of the previous fragment. 4. EXPERIMENTAL EVALUATION RESULTS We evaluate the V-HashMem architecture and our implementations using the official Snort rule-set [1] dated May 5th 2005 that consists of 2187 patterns and a total of 33,613 characters. We implemented the V-HashMem sub-systems in VHDL using the Xilinx ISE tools. We also used automated tools that given a set of patterns generated the VHDL code for the generators and the corresponding mapping of patterns to memories. In our results, we report the number of Logic Cells (i.e. one LUT plus one Flip-Flop). The first and foremost V-HashMem design decision is the number and configuration of the memory structures. After experimenting with the rule-set, we arrived at the final configuration shown in Table 1. Several factors influenced our decisions. First was the matter of conflicts: in the heavily populated length ranges (for example 11-14) there were several conflicting cases. We dealt with conflicts allowing for overlapping length ranges, which offer several placement possibilities for the conflicting sets. It turns out that we did not need more than two alternatives to remove all conflicts. Examples are the 5-10 and 8-13 structures that overlap for lengths Similarly structures 8-13 and overlap for lengths A additional reason forced us to use two structures for lengths Besides dealing with conflicts, this structure was very heavily populated having around 800 patterns. This increased the density of the indirection memory and forced us to use 13 bit function and consequently more (5) memory blocks. Splitting the structure in two halves actually reduced the required memory blocks and allowed us to use 12 bit functions. It should be noted that the distribution of patterns into lengths is vastly different from the SNORT rule-set statistics, since we use the partial ing technique. Patterns wider than 17 characters are broken down to a series of subpatterns populating structures up to 17 characters wide. The resulting structures use a total of 74 character comparators, compared to 150 in HashMem, a 50% improvement, and require 6 generators with a total of 55 char-

5 Table 2. Logic Cost of V-HashMem Components. Component Required Logic Cells FIFOs 250 Generators 362 Comparators 444 Glue Logic 948 Match & Address Report 80 Total 2084 acter input, compared to 150 character input for HashMem. Given the observation that in HashMem generation and comparators account for 32% and 42% of the logic respectively, there is the potential for considerable logic savings (in the order of 40%), despite using the newer SNORT ruleset with 70% more characters. On the other hand, the newer rule-set also includes many more wide strings that have glue logic overhead, reducing the potential improvement. The memory usage of our V-HashMem configuration is also depicted in table 1. We can see that 23 memory blocks are needed for pattern storage and 14 for index memories. In addition to these memories we need another 2 memory blocks to the very narrow 1 and 2 character patterns. This leads to a total of 39 memory blocks. The best Hash- Mem configuration used 31 memory blocks but to store significantly fewer patterns and using both memory read ports. Table 2 shows the area cost of each sub-system of our V-HashMem architecture. We break down the cost into (a) the character FIFOs that accumulate the characters for the generators and the comparators, (b) generators, (c) comparators, (d) glue logic for partial es, and (e) final address and reporting circuitry. The total calculated logic cost is 2086 Logic cells, or a cost of 0.06 LCs/character. This corresponds to a per character improvement of 50% over the unoptimized HashMem. The operating frequencies are 306 and 334MHz for a XC2VP30 Virtex2Pro and a XC4VSX25 Virtex4 device, that correspond to processing bandwidths of 2.45 and 2.76 Gbps respectively. These bandwidths can be doubled if we process two characters per cycle as described in section 3.3. Another metric used to compare hardware NIDS designs is the Performance Efficiency Metric (PEM), that is the ratio of throughput (in Gbps) to the logic cost per pattern character. Our original HashMem had a very high PEM (14 and 19 for Spartan3 and Virtex2Pro devices respectively). V- HashMem increases this to a PEM rating about 41 for Virtex2Pro and 46 for Virtex4 devices, in excess of a twofold improvement! This increase is due to the reduction in logic area cost, but also due to the increased rule-set size that fits in the now fewer V-HashMem memory structures. Finally, an important finding in our design is that the necessary circuitry to support Header-ID ing is very affordable. Using 10 bits to encode the header group increases the cost comparators by only 80 logic cells. How- Table 3. V-HashMem Compared to related works. LCs per Mem Approach Device Char Kbits #chars PEM V-HashMem V2Pro , PH-Mem [4] V2Pro ,911 7,03 ROM-Based [14] Spart ,800 4 DCAM [6] V2Pro ,036 4,05 Tree-based [15] V2Pro ,584 5,86 ever storing the Header-ID along with the patterns makes the memories wider. Some memories have spare bits and can accept this extension while other do not; overall, storing the Header-ID along with the patterns requires an additional 3 memory blocks, a small price for the increased functionality. Actually with the additional memories we can accommodate a Header-ID field up to 15 bits, although a wider field incurs increased logic cost for ing. 5. RELATED WORK In the recent years many pattern ing architectures have been proposed specifically for accelerating a SNORT-like NIDS using FPGAs. The architectures differ in the approach (finite automata or CAM-like), internal organizations, and of course in their cost-performance tradeoffs [9, 10, 11, 5, 12, 6, 13, 14, 15, 16, 4]. All these works strive for lower cost, at the same or better performance. HashMem is based on two ideas: (i) use of simple hashing to summarize the multiple input bits (also used in Bloom filters [11, 5]), and (ii) on the use of memories to provide exact with fewer gates (alsousedbychoet al. [14, 13], Sourdis et al. [4]). The use of Bloom filters for pattern ing has also been proposed for low cost pattern ing [11, 5, 8]. Bloom filters are very elegant in representing set membership, but suffer two potential drawbacks: (i) they require multiple hash functions and memories, and (ii) they give an approximate answer since they allow false positives. Solutions to these limitations exist but at additional cost. Cho and Magnione-Smith used a CAM to short patterns and to unique prefixes of longer search patterns [14, 13]. They choose the CAM width so as to provide unique prefix signals for each possible. The signals for all prefixes are then encoded to provide a memory address where the candidate suffixes are stored. The remaining input is compared against the expected suffix, and the result is the overall for the pattern. Their approach offers very good memory density and low gate count. The cost of this approach however increases if the patterns have many and long common prefixes. Sourdis et al. [4] have used a centralized memory based pattern ing, where the memory location is selected using a perfect hashing of selected input bits. This approach shares many of the advantages of HashMem, and achieves even better memory usage but at a higher logic cost.

6 Table 3 offers a brief comparison with other published results. V-HashMem used significantly less logic at the expense of relatively modest memory requirements: even with Header-ID support, the 42 memory blocks are available in all but the two smaller Virtex2Pro devices, while the largest one (XC2VP100) offers more than ten times so many. However the memory efficiency of V-HashMem is lower compared to other memory-based approaches. This difference is due to the use of index memories. 6. CONCLUSIONS The goal of this work was to extend the HashMem architecture and achieve even better efficiency. To this end we have achieved savings in the logic cost, while retaining the memory use at levels comparable to that of HashMem despite the 70% larger rule-set. This is a clear indication of the scaling abilities of the overall HashMem approach. The efficiency of V-HashMem is also evident in the logic area cost per search pattern character ( 0.06 LCs/char) and in the expected PEM rating, which about doubled compared to our earlier work, and is the highest that we are aware of. Another important discovery was that as the SNORT ruleset progresses, new and more difficult rules are included. In our case, the difficulty stems from very wide patterns that increased the cost of glue logic considerably. Other potential challenges for the future include regular expressions, specified offset ranges for a search string, and other advanced features that may be used more frequently in the future. We also investigated the support for header ing information and found that it is possible at the cost of three extra memory blocks and about 80 logic cells for the necessary logic. This is a very small cost compared to the increased accuracy of the pattern ing subsystem. Of course, a complete system would also add the cost of header classification into the header group identifiers, which we did not include in our measurements. These improvements lead closer to an efficient FPGA implementation of a NIDS system. We believe that a successful such system will rely on memory to store the patterns, and that the HashMem architecture with the proposed variable length extensions is a very competitive approach. 7. ACKNOWLEDGMENTS This work was partially supported by European Union FET project SARC under contract REFERENCES [1] M. Roesch, Snort - lightweight intrusion detection for networks, in Proceedings of LISA 99: 13th Administration Conference, November , seattle Washington, USA. [2] G. Papadopoulos and D. Pnevmatikatos, Hashing + memory = low cost, exact pattern ing, in Proceedings of the 15th International Conference on Field Programmable Logic and Applications, [3] Y. H. Cho, S. Navab, and W. Mangione-Smith, Specialized hardware for deep network packet filtering, in Proceedings of the 12th International Conference on Field Programmable Logic and Applications, [4] I. Sourdis, D. Pnevmatikatos, S. Wong, and S. Vassiliadis, A reconfigurable perfect-hashing scheme for packet inspection, in Proceedings of the 15th International Conference on Field Programmable Logic and Applications, [5] M. Attig, S. Dharmapurikar, and J. Lockwood, Implementation results of bloom filters for string ing, in Proceedings of the 12th IEEE Symposium on Field-Programmable Custom Computing Machines, April [6] I. Sourdis and D. Pnevmatikatos, Pre-decoded CAMs for efficient and high-speed nids pattern ing, in Proceedings of the 12th IEEE Symposium on Field-Programmable Custom Computing Machines, April [7] V. Dimopoulos, G. Papadopoulos, and D. Pnevmatikatos, On the importance of header classification in hw/sw network intrusion detection systems, in Proceedings of the 10th Panhellenic Conference on Informatics (PCI), November 11-13, [8] H. Song and J. Lockwood, Multi-pattern signature ing for hardware network intrusion detection systems, in Proceedings of IEEE Globecom 2005, November 28 - December2, [9] M. Gokhale, D. Dubois, A. Dubois, M. Boorman, S. Poole, and V. Hogsett, Granidt: Towards gigabit rate network intrusion detection technology, in Proceedings of the 12th International Conference on Field Programmable Logic and Applications, [10] J. Moscola, J. Lockwood, R. P. Loui, and M. Pachos, Implementation of a content-scanning module for an internet firewall, in Proceedings of the 11th IEEE Symposium on Field- Programmable Custom Computing Machines, April [11] S. Dharmapurikar, P. Krishnamurthy, T. Spoull, and J. Lockwood, Deep Packet Inspection using Bloom Filters, in Hot Interconnects, August 2003, stanford, CA. [12] C. R. Clark and D. E. Schimmel, Scalable parallel patterning on high-speed networks, in Proceedings of the 12th IEEE Symposium on Field-Programmable Custom Computing Machines, April [13] Y. H. Cho and W. H. Mangione-Smith, Deep packet filter with dedicated logic and read only memories, in Proceedings of the 12th IEEE Symposium on Field-Programmable Custom Computing Machines, April [14], Programmable hardware for deep packet filtering on a large signature set, in First Watson Conference on Interaction between Architecture, Circuits, and Compilers(P=ac2), [15] Z. K. Baker and V. K. Prasanna, Automatic synthesis of efficien intrusion detection systems on FPGAs, in Proceedings of the 14th International Conference on Field Programmable Logic and Applications, August [16], Time and area efficient reconfigurable pattern ing on FPGAs, in Proceedings of FPGA 04, 2004.