Application Protocol Breakdown

Transcription

1 Snort 2.0: Protocol Flow Analyzer Authors: Daniel Roelker Sourcefire Inc. Marc Norton Sourcefire Inc. Abstract The Snort 2.0 Protocol Flow Analyzer classifies network application protocols into client and server data flows. Indepth analysis of these protocol data flows allows the Fusion Detection Engine to make intelligent decisions about protocol inspection, greatly enhances performance and efficiency, and helps to reduce false positives. Currently, the Fusion Detection Engine has an HTTP flow analyzer that is user-configurable and significantly reduces the Fusion Detection Engine s HTTP processing time. Protocol Flow Analyzer What is a Protocol Flow? A protocol flow refers to the client or server communication in an application protocol. For example, HTTP clientto-server communication is considered a flow and HTTP server-to-client communication is considered a separate flow. This allows the Fusion Detection Engine to break down a particular application protocol into two distinct flows, a client flow and a server flow. Application Protocol Breakdown Client To Server CLIENT FLOW SERVER FLOW Server To Client Protocol Flow Analysis Protocol flow analysis is performed at a high level and is usually only concerned with a few important aspects of a particular protocol flow, such as a server response code or a client request type. Flow analysis does not replace other Snort 2.0 protocol inspection technologies; instead it compliments them. It is rather a generic analysis that allows an application protocol to be classified as a client or server flow. Once an application protocol is classified into a client flow and a server flow, it gives the Fusion Detection Engine useful knowledge as to the type of inspection and the regions of the protocol flow to inspect. Benefits of Protocol Flow Analysis Protocol flow analysis gives the Fusion Detection Engine rudimentary knowledge of a particular application protocol. With this knowledge Fusion Detection Engine decides what part of a protocol to inspect, if any part at all. This significantly reduces processing time, since the Fusion Detection Engine would usually inspect the ignored protocol data with all the applicable rules. Flow analysis also reduces false positives by limiting the amount of inspection that the Fusion Detection Engine does, so it is less likely to alert on unrelated rule content matches. Copyright 2003 Daniel Roelker, Marc Norton Sept

2 HTTP Flow Analyzer Overview The HTTP flow analyzer, now available in the Fusion Detection Engine, illustrates the benefits of flow analysis. It attempts to reduce the amount of network traffic analyzed by the Fusion Detection Engine at both the packet level and within the TCP stream reassembler. This technique reduces the amount of data the must be inspected for content matches. The idea for the HTTP flow analyzer was adapted from Web Protocols and Practice [1]. The following assumptions about HTTP traffic and the Fusion State Engine are made: 1. Due to the request-response model of HTTP, 95+% of HTTP traffic for a given web transfer is the response header and data, of which only a fraction (typically 3% - 5% or a few hundred bytes) is the response header. 2. Traffic from a protected server is usually trusted traffic and generally well conditioned. 3. The Fusion TCP State Engine knows which side of a connection is server-side based in the TCP 3-way handshake. So even if both source and destination ports are port 80, it is known whether a packet is from the server or client. 4. IP fragment reassembly is transparent to the HTTP Flow Analyzer, and is handled prior to flow analysis. Also, only fully reassembled IP packets are passed down to flow analyzer. The essence of the HTTP Flow Analyzer is to know what part of HTTP traffic should be inspected and what part should not be inspected. This eliminates a lot of useless pattern matching, as this paper will show. HTTP Traffic Profile In the HTTP protocol, the client request usually consists of a few hundred bytes on average. Fusion Detection Engine has 500+ rules, which it applies to client-side HTTP requests. This is a case where lots of rules are matched against a small amount of text. However, the HTTP server response contains much more data, about 5,000 to 15,000 bytes on average. But the server response header only constitutes about 300 bytes of the total server response. The rest of the server response is the response payload, and is generally not considered malicious traffic. This is due to the fact that servers in the protected network are trusted. The HTTP server response flow makes up 95% of the total HTTP traffic. And the response header is typically less than 5% of the total HTTP traffic. It is this 5% that needs to be inspected. The server response payload, which is 95% or more of the server response, can be eliminated. In total, this means that only 10% (5% client-side data and 5% server-side data) of all web traffic needs to be inspected. For instance, in an enterprise network, HTTP traffic can amount to ~75% of all network traffic, of which only 10% of HTTP traffic need be processed by the detection engine. This reduces the HTTP processing from 75% of total network traffic to 7.5%. Work Flow Analysis The total rule processing work of an application protocol consists of both the server protocol flow and the client protocol flow. This can be defined as: Total Work = Server Flow Work + Client Flow Work Using this definition, the processing time that the detection engine saves from protocol flow analysis can be determined. This is accomplished by finding the Work Flow Ratio. The Work Flow Ratio is computed by comparing the Total Work it takes for the detection engine to process server and client flows without using flow analysis, and the Total Work it takes for the detection engine to process server and client flows using flow analysis. In Appendix A, we calculate that after protocol flow analysis, the Work Flow Ratio reduces rule processing time for HTTP traffic by 89%. Reducing the rule processing time by 89% reduces the total Snort processing time by 44%. Copyright 2003 Daniel Roelker, Marc Norton Sept

3 This is because the rule processing time for HTTP traffic is about 50% of the total Snort processing. Appendix A shows the breakdown of total Snort processing and an in-depth look at the Work Flow Ratio. HTTP Flow Analyzer Processing Methods There are two methods of processing HTTP packets for performance purposes: packet-based and session-based. Packet-Based Processing When processing HTTP requests and responses by packet, the only processing information available is that which is available in the packet. The only required information is a user-defined variable, called HTTP Maximum Response Bytes. This variable specifies the number of HTTP response bytes to process. The process is as follows: 1. Check transport protocol for TCP. 2. Check source port = web ports and destination port!= web ports. 3. Check that the first four bytes of payload are HTTP, if this is the case then the packet is an HTTP response header and we process up to the HTTP Maximum Response Bytes limit, if not then it is an HTTP response payload, which is not processed. Arguably, this method is naive and the HTTP response header inspection could be evaded through breaking up the header from the initial HTTP response, but the HTTP Flow Analyzer assumes that server flows from the protected net are trusted. This is a primitive inspection, but it is still very effective for enhancing performance. Session-based Processing In session-based processing, HTTP requests and responses can be intelligently handled since TCP state is handled before sessions are processed. The session-based processing assumes that the client flow of a session will be rebuilt and that the server flow also be rebuilt. These characteristics are defined as: The session-based processing flow follows: 1. Check for a known HTTP session. If there isn t a known HTTP session, then this payload is not processed since the TCP state has not been verified. 2. If this is a known HTTP session and has been verified by TCP state, then the first four bytes of the incoming payload are inspected for HTTP. If HTTP is found in this packet, a new HTTP server response header inspection is performed, until it is terminated by the standard HTTP end-of-header delimiter (\r\n\r\n). HTTP Connection: Keepalive Both packet-based and session-based processing approach deal with the problem of the HTTP Connection: Keepalive command. In practice, the HTTP responses are fragmented at the application level and not the byte level, so new HTTP responses will begin in a new packet that begins with HTTP. Both packet-based and sessionbased procedures handle these responses without any tweak to the current procedures. However, the theoretical scenario allows multiple HTTP responses to be embedded in the same packet. While multiple HTTP responses in a single packet will confuse the current packet-based procedure, it will not confuse the session-based procedure. Session-based processing can deal with multiple HTTP responses in a single packet by implementing a quick protocol scanner or decoding the protocol fully. This technique is possible since it is session oriented and TCP payloads are reconstructed. In practice, multiple HTTP responses in the same packet are unlikely to occur from a trusted server. In conclusion, the HTTP performance technique that has been discussed here is essential for high-performance inspection. It is just another example of how protocol analysis is incorporated into the Fusion Detection Engine. By using this technique, the detection engine effectively drops about 90% of HTTP traffic. In a typical 100 Mbits/sec Copyright 2003 Daniel Roelker, Marc Norton Sept

4 link, this eliminates ~67.5 Mbits/sec per link, given that HTTP traffic constitutes ~75 Mbits/sec in a 100 Mbits/sec link. This is a significant speed up for such a simple modification. Bibliography [1] B. Krishnamurthy and J. Rexford. Web Protocols and Practice. Addison-Wesley: 2001, AT&T Corp. Page 520. Copyright 2003 Daniel Roelker, Marc Norton Sept

5 Appendix A - Work Flow Ratio Explained In order to explain how the HTTP Flow Analyzer affects Snort performance, the total Snort performance needs to be summarized. The total Snort performance breakdown has been accomplished through in-depth testing of Snort using real HTTP traffic on the wire. The following is the total Snort Processing Time that we have measured through testing: Snort Processing Time 50% Sniffing, Packet Decoders, Preprocessors 50% Rule Inspection By using this information, an example of speed improvement during rule inspection can be modeled. For example, if the rule inspection processing time is reduced by 50% (improving the rule inspection processing time down to 25%), the total Snort processing is only increased by 25%. This is shown as follows: Total Snort Processing = 50% (Sniffing, Decoders, Preprocessors) + 25% (decreased rule inspection time) Total Snort Processing = 75% So, the total Snort processing time is increased by 25%. Now that there is an example of how total Snort processing time can be measured, the rule processing work flow ratio can be calculated. Work Flow Ratio Before the Work Flow Ratio is calculated, it is important to note that many of the numbers used in calculating the Work Flow Ratio were discovered through in-depth testing. As usual, testing results may differ depending on the type of hardware and the compiler used to compile Snort. The following calculation of the work flow ratio is taken from the example of HTTP rule processing. By analyzing the HTTP protocol in real traffic, it appears that the client request is on average 300 bytes, and the server response on average is between 5,000 and 15,000 bytes. We ll use 10,000 bytes for the average server response. For the particular protocol flows, assume Snort uses 500 rules for the HTTP client flow and 50 rules for the HTTP server flow. The actual rule processing difference between 500 and 50 rules has been calculated for a specific P4 configuration. It is important to note these tests were performed on a particular hardware platform using a benchmark test and pattern files. Individual test results may vary. We ve found that processing 500 rules results in a rate of 22 Mbytes/second. Processing 50 rules results in a rate of 62 Mbytes/second. This shows a work flow ratio of about 1/3. The work ratio is calculated using the resulting Mbytes/second: 22/62 ~ 1/3 With this ratio, we can now set up the following example. HTTP Flow Analyzer Example The following equation is used to calculate the total work it takes Snort to process HTTP traffic: Total Work = Server-Side Work + Client-Side Work Before HTTP flow analysis is used, we see the following results: Server Client Data Bytes 10, # Of Rules Copyright 2003 Daniel Roelker, Marc Norton Sept

6 This shows that the server-side does 33x (10,000 / 300) the work in processing data bytes that the client does. This means that we assume that the client does 1 unit of work in processing data bytes. And the work associated with processing 50 and 500 rules has already been established as 1/3. Again we assume that the client takes 1 unit of work to process rules. In order to find the total work of both the client and server flow processing, we multiply the work units it takes to process data bytes and the work units to process the number of rules. We multiply both of these parameters since the total work involved in processing data bytes is proportional linked to the number of rules. So we use the following equation for server-side and client-side work: Client/Server-side Work = Byte processing work * Rule processing work When the both the client-side work and the server-side work are added together, we get: Total Work = (33 * 1/3) + (1 * 1) = 37/3 ~ 12 So, the total work is approximately 12 work units. Please note that the majority of this work is due to the server flow. After HTTP flow analysis is used, the server data bytes are reduced to 300 bytes. So, we now see the following results: Server Client Data Bytes # Of Rules This shows that the server-side does the same work in processing data bytes as the client, so they both get 1 unit of work. And the difference between processing 50 and 500 rules is still the same, so the work is 1/3 for the server and 1 for the client. When these are added together, the total work units become: Total Work = (1 * 1/3) + (1 * 1) = 4/3 So, the total work before flow analysis is 12 units and the total work units after flow analysis is 4/3. By dividing the total work units after flow analysis by the total work units before flow analysis, we find the relative percentage that flow analysis saves us during rule processing. (4/3) / 12 =.11 or 11% This tells us that after flow analysis, we are only doing 11% of the work we were doing before analysis. So, flow analysis for the HTTP protocol saves Snort approximately 89% of rule processing time. Total Snort Performance Effects Let s look at how this affects total Snort performance. As stated earlier, rule processing is only 50% of the total Snort processing time. With flow analysis, the rule processing has been reduced by 89%, so we calculated the total rule processing work in Snort as:.5 *.11 =.06 This means that the rule processing time has been reduced to 6% processing, instead of the previous 50% processing. So, the total performance of Snort is reduced from 100% processing to 50% + 6% (rule processing), which equals 56% processing. HTTP protocol flow analysis saves Snort 44% in total processing time of HTTP traffic. Copyright 2003 Daniel Roelker, Marc Norton Sept