TEST METHODOLOGY. Next Generation Intrusion Prevention System (NGIPS) v2.0

Transcription

1 TEST METHODOLOGY Next Generation Intrusion Prevention System (NGIPS) v2.0

2 Table of Contents 1 Introduction The Need for Next Generation Intrusion Prevention Systems About This Test Methodology Inclusion Criteria Security Effectiveness Exploits Library False Positive Testing Coverage by Attack Vector Coverage by Impact Type Coverage by Date Coverage by Vendor Coverage by Impact Target Type Live Drive-by Exploits Application ID Evasion IP Packet Fragmentation Stream Segmentation RPC Fragmentation SMB and NetBIOS Evasions URL Obfuscation HTML Obfuscation FTP Evasion Payload Encoding Layered Evasions TCP Split Handshake Spoof Performance Raw Packet Processing Performance (UDP Traffic) Byte Packets Byte Packets Byte Packets Byte Packets Byte Packets Byte Packets Latency Byte Frames Byte Frames Byte Packets

3 Byte Packets Byte Packets Byte Packets Maximum Capacity Theoretical Maximum Concurrent TCP Connections Theoretical Maximum Concurrent TCP Connections with Data Theoretical Maximum Data Throughput on a Single Connection Maximum TCP Connections per Second Maximum HTTP Connections per Second Maximum HTTP Transactions per Second HTTP Capacity with No Transaction Delays KB HTTP Response Size 2,500 Connections per Second KB HTTP Response Size 5,000 Connections per Second KB HTTP Response Size 10,000 Connections per Second KB HTTP Response Size 20,000 Connections per Second KB HTTP Response Size 40,000 Connections per Second HTTP Connections per Second and Capacity with Delays KB HTTP Response with Delay KB HTTP Response with Delay Application Average Response Time: HTTP Real-World Traffic Real-World Protocol Mix (Enterprise Perimeter) Real-World Protocol Mix (Financial) Real-World Protocol Mix (Education) Stability and Reliability Blocking under Extended Attack Passing Legitimate Traffic under Extended Attack Behavior of the State Engine under Load State Preservation Normal Load State Preservation Maximum Exceeded Protocol Fuzzing and Mutation Power Fail Persistence of Data Total Cost of Ownership and Value Appendix A: Change Log Contact Information

4 1 Introduction 1.1 The Need for Next Generation Intrusion Prevention Systems Introduced over a decade ago, the first network intrusion prevention systems (IPS) were built on generic Intel servers with the purpose of blocking exploits that target vulnerable servers. Soon after, attacks against desktop clients emerged and the first generation of intrusion prevention struggled to maintain performance and security. This led to a new hardware-accelerated generation of IPS that were capable of inspecting much more traffic and at higher speeds than software-only solutions. Over the past few years, several trends have emerged, each posing challenges for intrusion prevention: social media, remote workers, wireless, bring your own device (BYOD), and the explosion of business/personal web applications all have led to the near-disintegration of the network perimeter. Simultaneously, cybercriminals have grown more aggressive, increasingly targeting corporate assets including clients, browsers, and plug-ins. The growing number of vulnerability disclosures in widely deployed operating systems and applications is a multifaceted problem. As a result, a new generation of intrusion prevention is required to meet the challenges of organizations without clearly defined perimeters. These next generation intrusion prevention systems (NGIPS) must provide organizations with the ability to identify both the applications and the users on their internal networks. As with their predecessors, NGIPS must protect the enterprise user against threats/exploits. Designed to identify and block attacks against internal computing assets, a good NGIPS can provide temporary protection and relief from the immediate need to patch affected systems. The NGIPS must catch sophisticated attacks while producing as few false positives as possible. Since the role of an NGIPS is to protect users and provide granular visibility into network traffic, the NGIPS should be considered a perimeter device and should be deployed behind a firewall to supplement the overall security of the enterprise. 1.2 About This Test Methodology NSS Labs test reports are designed to address the challenges faced by IT professionals in selecting and managing security products. The scope of this particular methodology includes: Security effectiveness Performance Stability and reliability Total cost of ownership (TCO) NGIPS products are deployed at critical points in the network, and their stability and reliability are imperative. Regardless of any technical features, an NGIPS must be as stable, reliable, fast, and flexible as the infrastructure it protects, and it must improve the security posture of an enterprise. It should also be possible to incorporate an NGIPS into an existing security architecture without requiring a network redesign. The following capabilities are considered essential as part of an NGIPS product: Intrusion prevention Application identification 4

5 Resistance to known evasion techniques Reputation awareness Highly resilient and stable Operation at Layer 2 (network transparency) 1.3 Inclusion Criteria In order to encourage the greatest participation, and allay any potential concerns of bias, NSS invites all security vendors claiming NGIPS capabilities to submit their products at no cost. Vendors with major market share, as well as challengers with new technology, will be included. The NGIPS should be implemented as an inline device operating at Layer 2; that is, it should be transparent so that it can be added to an existing network infrastructure without altering the network. The NGIPS should be supplied as a single appliance where possible. Cluster controller solutions are also acceptable. The device should be equipped with an appropriate number of physical interfaces, with a minimum of one inline Gigabit Ethernet (GbE) port pair per gigabit of throughput or one inline 10GbE port pair per 10 Gbps of throughput. For example, an 8 Gbps NGIPS with only four gigabit port pairs will be limited to 4 Gbps. The minimum number of port pairs will be connected to support the claimed maximum bandwidth of the DUT. For example, an 8 Gbps NGIPS with ten port pairs will be tested with eight 1-GbE connections. Once installed in the test lab, the DUT will be configured for its most common use case: a corporate network perimeter. The device will also be configured to block all traffic when resources are exhausted or when traffic cannot be analyzed for any reason. 5

6 2 Security Effectiveness Vendors are permitted and encouraged to assist NSS engineers in tuning the DUT to eliminate false positives. Some vendors provide customers with factory-tuned policies instead of manually tuning their products. These policies usually contain all the signatures that can be safely deployed without triggering false positives. This approach allows customers to quickly deploy a more aggressive security policy with relatively little effort or risk and will be reflected positively in the time and cost sections of this test. Performance and security effectiveness testing are always measured with both the vendor recommended policy and the tuned policy. Once testing begins, the product version and configuration will be frozen to preserve the integrity of the test. 2.1 Exploits Library The latest signature pack is acquired from the vendor, and the DUT is deployed with the out-of-the-box vendor recommended security policy. All signatures used must be available to the general public at the time of testing. NSS security effectiveness testing leverages the deep expertise of our engineers who utilize multiple commercial, open source, and proprietary tools, including NSS network live stack test environment 1 as appropriate. This test goes far beyond replaying packet captures or pressing the button on a test tool. In short, NSS engineers trigger vulnerabilities for the purpose of validating that an exploit was able to pass through the DUT False Positive Testing The ability of the DUT to identify and allow legitimate traffic is of equal importance to providing protection against malicious content. This test will include a varied sample of legitimate application traffic that should properly be identified and allowed. After completion of the false positive testing, and prior to the mitigation testing being performed, those signatures that were deemed to cause the false positive alerts will be disabled within the security policy Coverage by Attack Vector Threats and exploits can be initiated by either the target or the attacker targeting either local or remote vulnerabilities Attacker Initiated The threat/exploit is executed by the attacker remotely against a vulnerable application and/or operating system Target Initiated The threat/exploit is initiated by the vulnerable target. The attacker has little or no control as to when the target user or application will execute the threat. 1 For more information on the NSS Live Testing harness and methodology, please refer to the latest Security Stack (Network): Test Methodology located at 6

7 Network Threats/exploits that are initiated as a result of network communication Local Local execution that requires access to the target host Coverage by Impact Type The NSS threat and attack suite contains thousands of publically available exploits (including multiple variants of each exploit), from which groups of exploits are carefully selected to test based on appropriate usage. Each exploit has been validated to impact the target vulnerable host(s). Based on the impact of the threat against the target, the following metrics are reported: System Exposure Attacks resulting in remote system compromise and the ability of the attacker to execute arbitrary system-level commands. Most weaponized exploits in this class will provide the attacker with a fully interactive remote shell on the target client or server Service Exposure Attacks resulting in an individual service compromise but not arbitrary system-level command execution. Typical attacks in this category include service-specific attacks, such as SQL injection, that enable the attacker to execute arbitrary SQL commands within the database service. These attacks are somewhat isolated to the service and do not immediately result in full system-level access to the operating system and all services. However, by using additional localized system attacks, it may be possible for the attacker to go from the service level to the system level System or Service Fault Attacks resulting in a system-level or service-level fault that crashes the targeted service or application and requires administrative action to restart the service or reboot the system. These attacks do not enable the attacker to execute arbitrary commands. However, the resulting impact to the business could be severe given that the attacker could crash the protected system or service Coverage by Date The typical enterprise will run a mix of both old and new applications, and NSS research shows that crimeware kits will frequently include exploits that date back several years. Therefore, NSS security effectiveness testing will include exploits current at the time of the test, as well as targeting vulnerabilities covering multiple years, dating backwards from the time of the test Coverage by Vendor Exploits will target both client and server-side applications and services Coverage by Impact Exploit severity will vary. 7

8 Arbitrary Code Execution This describes a software bug that allows an attacker to execute any commands of the attacker's choice on a target machine or in a target process Buffer Overflow The exploitation of a software bug due to improperly established memory bounds, which allows an attacker to overwrite adjacent memory and execute a command Code Injection The exploitation of a software bug that allows for the processing of invalid data within a program. Code injection can be used by an attacker to introduce code into a computer program in order to change the course of execution Cross-Site Script The exploitation of a web application that enables attackers to insert a malicious script into web pages, which can then be executed by other users Directory Traversal The exploitation of a lack of security in an application (as opposed to exploiting a bug in the code) that allows usersupplied input with characters representing traverse to parent directory to be passed through to the file APIs. The goal of this attack is to order an application to access a file or executable that is not intended to be accessible Privilege Escalation This exploit type allows an attacker to gain access to resources that would not normally have been available Target Type The following list of web target types is represented in the NSS live exploit test. This list is not intended to be allinclusive, and additional target types may be added or removed. Browser plug-ins/add-ons ActiveX Web browser Java 8

9 2.2 Live Drive-by Exploits Protection from web-based exploits targeting client applications, also known as drive-by downloads, can be effectively measured in the NSS unique test harness through a series of procedures that measure the stages of protection. Unlike traditional malware that is downloaded and installed, drive-by attacks first exploit a vulnerable application and then silently download and install malware. This means that there are three opportunities to break the chain of events leading to a successful compromise: 1. URL access (Reputation) 2. Exploit trigger 3. Malware drop or execution Success or failure will be determined based upon blocking the attack. Attacks that are not successfully blocked will be measured as a failure. 2.3 Application ID Complex outbound and inbound policies consisting of many rules, objects, and applications verify that the DUT is capable of correctly determining the correct application from deep packet inspection (regardless of port/protocol used) and taking the appropriate action. The following list contains examples of network traffic that may be addressed by this test. It is not intended to be all-inclusive, and additional applications may be added or removed. For an IPS, application identification is for high-level visibility, not granular control. Popular social networking web sites (web applications) Instant messaging (IM) Skype and other VoIP Streaming Peer-to-peer Online gaming 9

10 2.4 Evasion Attackers can modify basic attacks to evade detection in a number of ways. If a DUT fails to detect a single form of evasion, any exploit can pass through the device, rendering it ineffective. NSS verifies that the DUT is capable of detecting and blocking basic exploits when subjected to varying common evasion techniques. Wherever possible, the DUT is expected to successfully decode the obfuscated traffic to provide an accurate alert relating to the original exploit, rather than alerting purely on anomalous traffic detected as a result of the evasion technique itself. Providing exploit protection results without fully factoring in evasion can be misleading, since the more different types of evasion that are missed IP fragmentation, TCP segmentation, RPC fragmentation, URL obfuscation, and FTP evasion the worse the situation. For example, it is better to miss all techniques in one evasion category (e.g., FTP evasion) than one technique in each category. Furthermore, evasions operating at the lower layers of the network stack, such as IP fragmentation or TCP segmentation, will have a more negative impact on security effectiveness than those operating at the upper layers (HTTP or FTP obfuscation), since lower-level evasions will potentially impact a wider number of exploits. Thus, missing TCP segmentation is a much more serious issue than missing FTP obfuscation. A number of common exploits are executed across the DUT to ensure that they are detected in their unmodified state. These will be chosen from a suite of older/common basic exploits for which NSS is certain that all vendors will have signatures. None of the exploits that were used in section 2.1 will be used as evasion baselines. This ensures that vendors are not provided with any information on the content of any part of the main NSS exploit library in advance of the test. It is a requirement of the test that the DUT submitted for testing should have all evasion detection options enabled IP Packet Fragmentation These tests determine the effectiveness of the fragment reassembly mechanism of the DUT: Fragments of various sizes Ordered, out-of-order, or reverse order fragments Fragment overlap, favoring new and favoring old data Interleaved, duplicate, duplicate with or without incrementing DWORD, duplicate packets with random payload, or duplicate packets scheduled for later delivery Random chaff Any combination of the above methods Stream Segmentation These tests determine the effectiveness of the stream reassembly mechanism of the DUT: Segments of various sizes Ordered, reverse ordered, or out-of-order segments, with favor old or favor new Duplicate, duplicate interleaved, duplicate last packet, or overlapping segments Invalid or NULL TCP control flags 10

11 Sequence resync requests, random initial sequence number, or out-of-window sequence numbers Faked retransmits, PAWS elimination, or segments containing random data Endianness interchanged Random chaff Any combination of the above methods RPC Fragmentation Both Sun/ONC RPC and MS-RPC allow a sending application to fragment requests, and all MS-RPC services have a built-in fragmentation reassembly mechanism. An attacker can transmit a BIND followed by a single request fragmented over a hundred actual requests with small fragments of the malicious payload. Alternatively, the attacker could transmit both the BIND and request fragments in one large TCP segment, thus foiling any signatures which use a simple size check. These tests determine the effectiveness of the RPC reassembly mechanism of the DUT: Sun/Open Network Computing (ONC) RPC byte fragmentation Fragments sent in one or more TCP segments, with or without the last fragment RPC fragmentation may or may not be performed in a single segment MSRPC messages are sent with interchangeable endianness Various byte-size MSRPC fragments are sent in the same lower layer message MSRPC requests are fragmented to contain at most 2048 bytes of payload Any combination of the above methods SMB and NetBIOS Evasions Upper-layer protocols such as SMB and NetBIOS can be manipulated in various ways, including segmentation of SMB read/write operations, or inserting fake SMB requests and chaffed NetBIOS messages: SMB or NetBIOS chaff before the real message Chaff may contain MSRPC, HTTP POST, HTTP GET, or random payload SMB filename obfuscation using case changes and dummy paths SMB writes segmented to contain at most 2048 bytes of payload Various byte sizes of random padding inserted into WriteAndX messages between an SMB header and payload Any combination of the above methods URL Obfuscation Random URL encoding techniques are employed to transform simple URLs that are often used in pattern-matching signatures to apparently meaningless strings of escape sequences and expanded path characters using a combination of the following techniques: Escape encoding (% encoding) Microsoft %u encoding Path character transformations and expansions ( /./, //, \ ) 11

12 These techniques are combined in various ways for each URL tested, ranging from minimal to extreme transformation (every character transformed). All transformed URLs are verified to ensure they still function as expected after transformation. URL encoding Levels 1 8 (minimal to extreme) Premature URL ending Long URL Fake parameter TAB separation Case sensitivity Windows\delimiter Session splicing Any combination of the above methods HTML Obfuscation Recognizing malicious HTML documents is becoming increasingly important when protecting the enterprise. Malicious HTML documents exploit flaws in common web browsers, browser plug-ins, and add-ons to gain control of the client system and silently install malware such as Trojans, rootkits, and key loggers. Therefore, it is becoming increasingly important that security products charged with protecting end systems must correctly interpret HTML documents. Many security products use simple pattern matching systems with very little semantic or syntactic understanding of the data they are analyzing. This leaves them vulnerable to evasion through use of redundant, but equivalent, alternative representations of malicious documents. This test suite uses a number of malicious HTML documents that are transferred from server to client through the DUT. Each malicious HTML document is served with a different form of obfuscation, as follows: UTF-16 and UTF-32 character set encoding (big-endian) UTF-16 and UTF-32 character set encoding (little-endian) UTF-7 character set encoding Chunked encoding (random chunk size, fixed 8-byte chunk size, or chaffing/arbitrary numbers inserted between chunks) Compression (deflate) Compression (gzip) Base-64 character set encoding with or without bit shifting or chaffing Any combination of the above methods The UTF-16 character set specifies a 2-byte sequence for most characters and a 4-byte sequence for the others (a small percentage). Recoding an HTML document in UTF-16 significantly changes its appearance. A document that contains just the ASCII subset of characters will appear to have a null byte between every one of the original characters. There are also two different forms of the UTF-16 encoding, depending on whether the null high byte comes first (big-endian) or second (little-endian) this test uses big-endian byte ordering. 12

13 The UTF-32 character set specifies a 4-byte sequence. Similar to the UTF-16 character set encoding, there are two variations big-endian and little-endian and this test case uses big-endian byte ordering. The UTF-7 character set encodes most ASCII characters as themselves. However, in addition to recoding non- English characters as other encodings do, it also recodes many punctuation symbols, including many of the symbols that are important to the HTML specification. Therefore, recoding an HTML document in UTF-7 significantly changes its appearance Chunked encoding allows the server to break a document into smaller chunks and transmit them individually. The server needs only to specify the size of each chunk before it is transmitted and then indicate when the last chunk has been transmitted. Since chunked encoding intersperses arbitrary numbers (chunk sizes) with the elements of the original document, it can be used to greatly change the appearance of the original document as observed on the wire. In addition, the server can choose to break the document into chunks at arbitrary points. This makes it difficult for simple pattern matching systems to reliably identify the original HTML document from the raw data on the network. Per RFC 2616, the HTTP protocol allows the client to request, and the server to use, several compression methods. These compression methods not only improve performance in many circumstances, they completely change the characteristic size and appearance of HTML documents. Furthermore, small changes in the original document can greatly change the final appearance of the compressed document. This property of these algorithms could be used to obfuscate hostile content for the purpose of evading detection. The deflate compression method is a Lempel-Ziv coding (LZ77), specified in RFC The gzip compression method is specified in RFC For each of the above, it is verified that a standard web browser (such as Internet Explorer) is capable of rendering the results of the evasion FTP Evasion When attempting FTP exploits, it is possible to evade some products by inserting additional spaces and telnet control sequences in FTP commands: Inserting spaces in FTP command lines Inserting non-text Telnet opcodes Levels 1 8 (minimal to extreme) Any combination of the above methods Payload Encoding This test attempts to confuse the DUT into allowing an otherwise blocked exploit to pass using various encoding options: x86/call4_dword_xor This encoder implements a Call+4 Dword XOR Encoder x86/countdown This encoder uses the length of the payload as a position-dependent encoder key to produce a small decoder stub. x86/fnstenv_mov This encoder uses a variable-length mov equivalent instruction with fnstenv for getip. x86/jmp_call_additive This encoder implements a Jump/Call XOR Additive Feedback Encoder x86/shikata_ga_nai This encoder implements a polymorphic XOR additive feedback encoder. The decoder stub is generated based on dynamic instruction substitution and dynamic block ordering. Registers are also selected dynamically. 13

14 2.4.9 Layered Evasions This test attempts to bypass the DUT by performing any legitimate combination of the evasion techniques specified in section TCP Split Handshake Spoof The TCP split handshake blends features of both the three-way handshake and the simultaneous-open connection. The result is a TCP spoof that allows an attacker to bypass the DUT by having the attacker instruct the target to initiate the session back with the attacker. Popular TCP/IP networking stacks respect this handshaking method, including Microsoft, Apple, and Linux stacks, with no modification. 2 2 The TCP Split Handshake: Practical Effects on Modern Network Equipment, Tod Beardsley & Jin Qian, 14

15 3 Performance This section measures the performance of a device using various traffic conditions that provide metrics for realworld performance. Individual implementations will vary based on usage; however, these quantitative metrics provide a gauge as to whether a particular DUT is appropriate for a given environment. Security devices exhibit an inverse correlation between security effectiveness and performance. An increased level of deep packet inspection results in an increase in the amount of time a device requires to forward packets. Furthermore, it is important to consider a real-world mix of traffic that a device will encounter. NSS utilizes a range of traffic types and mixes. Performance and security effectiveness testing are always measured with both the vendor recommended policy and the tuned policy. Once testing begins, the product version and configuration will be frozen to preserve the integrity of the test. 3.1 Raw Packet Processing Performance (UDP Traffic) This test uses UDP packets of specific sizes, generated by a traffic generation tool, to validate the packet performance of a DUT. A constant stream of packets of a specified size, with varying source and destination IP addresses, transmitting from varying source ports to a fixed destination port, is transmitted bi-directionally through each tested port pair of the DUT. Each packet contains dummy data, and is targeted at a valid port on a valid IP address on the target subnet. The percentage load and frames per second (fps) figures across each inline port pair are verified by network monitoring tools before each test begins. Vendors will be expected to create an ad hoc signature to detect random payload data prior to running the first test to demonstrate that deep packet inspection is occurring on this traffic. Multiple tests are run and averages taken where necessary. This traffic does not attempt to simulate any form of real-world network use case. No TCP sessions are created during this test, and there is very little for the detection engine to do. The goal of this test is to determine the raw packet processing capability of each inline port pair of the DUT and its effectiveness at forwarding packets quickly, in order to provide the highest level of network performance and lowest latency Byte Packets Maximum 1,488,000 frames per second per gigabit of traffic. This test determines the ability of a device to process packets from the wire under the most challenging packet volume processing conditions Byte Packets Maximum 844,000 frames per second per gigabit of traffic Byte Packets Maximum 452,000 frames per second per gigabit of traffic Byte Packets Maximum 234,000 frames per second per gigabit of traffic. This test provides a reasonable indication of the ability of a device to process packets from the wire on an average network. 15

16 Byte Packets Maximum 119,000 frames per second per gigabit of traffic Byte Packets Maximum 81,000 frames per second per gigabit of traffic. This test assesses a device s maximum achievable bit rate, with a minimum of influence by its packet forwarding capacity. 3.2 Latency The purpose of the latency test is to determine the amount of time it takes for network traffic to pass through a DUT under a range of load conditions. The average latency (µs) is recorded for each specified packet size at a load level of 90% of the maximum throughput with zero packet loss, as previously determined in section Byte Frames Maximum 1,488,000 frames per second per gigabit of traffic Byte Frames Maximum 844,000 frames per second per gigabit of traffic Byte Packets Maximum 452,000 frames per second per gigabit of traffic Byte Packets Maximum 234,000 frames per second per gigabit of traffic Byte Packets Maximum 119,000 frames per second per gigabit of traffic Byte Packets Maximum 81,000 frames per second per gigabit of traffic 3.3 Maximum Capacity The purpose of these tests is to stress the inspection engine and determine how it handles high volumes of TCP connections per second, application layer transactions per second, and concurrent open connections. All packets contain valid payload and address data, and these tests provide an excellent representation of a live network at various connection/transaction rates. The following behaviors are used as failure criteria for each of these tests: Excessive concurrent TCP connections Excessive response time for HTTP transactions Unsuccessful HTTP transactions 16

17 3.3.1 Theoretical Maximum Concurrent TCP Connections This test is designed to determine the maximum concurrent TCP connection capacity of the DUT, with no data passing across the connections. This type of traffic would not be typical of a normal network, but it provides a means to assess a device s connection capacity, unencumbered by traffic payloads. An increasing number of TCP sessions are opened through the DUT. Each session is established and then held open for the duration of the test as additional sessions are added, up to the maximum possible. The test traffic load is increased until no further connections can be established. The maximum number of established connections is recorded Theoretical Maximum Concurrent TCP Connections with Data This test is identical to the test in section 3.3.1, but with the addition of 21 KB of data, which is transmitted in 1 KB segments during the session. This ensures that the DUT is capable of passing data between the connections once they have been established Theoretical Maximum Data Throughput on a Single Connection Concurrent connections are opened to 10% of the level determined in section 3.3.1, after which an increasing amount of data is transmitted across a single connection. This provides a comparison between the amount of data that can be transmitted across a single consumer versus multiple connections Maximum TCP Connections per Second This test is designed to determine the maximum TCP connection rate of the DUT with one byte of data passing across the connections. This type of traffic is atypical of a normal network, but it provides a means to determine the maximum possible TCP connection rate of the DUT. An increasing number of new sessions are established through the DUT and ramped slowly to determine the exact point of failure. Each session is opened normally, one byte of data is passed to the host, and then the session is closed immediately. The test traffic load is increased until one or more of the points of failure defined in section 3.3 is reached Maximum HTTP Connections per Second This test is designed to determine the maximum TCP connection rate of the DUT with a 1-byte HTTP response size. The response size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 1-byte response size is designed to provide a theoretical maximum HTTP connections per second rate. The test uses HTTP 1.0 without persistent connections (HTTP keep-alive). The client opens a TCP connection, sends one HTTP request, and once the request is fulfilled, promptly closes the connection. This ensures that any concurrent TCP connections that occur are a result of the latency induced by the DUT. The test traffic load is increased until one or more of the points of failure defined in section 3.3 is reached Maximum HTTP Transactions per Second This test is designed to determine the maximum HTTP transaction rate of the DUT with a 1-byte HTTP response size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 1-byte response size is designed to provide a theoretical maximum connections per second rate. 17

18 Client and server are using HTTP 1.1 with keep-alive, and the client will open a TCP connection, send 10 HTTP requests, and close the connection. This ensures that TCP connections remain open until all 10 HTTP transactions are complete, thus eliminating the maximum connection per second rate as a bottleneck (1 TCP connection = 10 HTTP transactions). Load is increased until one or more of the breaking points defined earlier is reached. 3.4 HTTP Capacity with No Transaction Delays The aim of these tests is to stress the HTTP detection engine and determine how the DUT copes with network loads of varying average packet size and varying connections per second. By creating genuine session-based traffic with varying session lengths, the DUT is forced to track valid TCP sessions, thus resulting in a higher workload than for simple packet-based background traffic. This provides a test environment that simulates real-world HTTP transactions in the lab while ensuring absolute accuracy and repeatability. Each transaction consists of a single HTTP GET request and there are no transaction delays (i.e., the web server responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and address data, and this test provides an excellent representation of a live network (albeit one biased towards HTTP traffic) at various network loads. Connections Per Second 44Kbyte Response 21Kbyte Response 10Kbyte Response 4.5Kbyte Response 1.7Kbyte Response CPS 2,500 5,000 10,000 20,000 40,000 Mbps 1,000 1,000 1,000 1,000 1,000 Mbps KB HTTP Response Size 2,500 Connections per Second Maximum 2,500 new connections per second per gigabit of traffic with a 44 KB HTTP response size average packet size 900 bytes maximum 140,000 packets per second per gigabit of traffic. With relatively low connection rates and large packet sizes, all DUTs should be capable of performing well throughout this test KB HTTP Response Size 5,000 Connections per Second Maximum 5,000 new connections per second per gigabit of traffic with a 21 KB HTTP response size average packet size 670 bytes maximum 185,000 packets per second per gigabit of traffic. With average connection rates and average packet sizes, this is a good approximation of a real-world production network, and all DUTs should be capable of performing well throughout this test KB HTTP Response Size 10,000 Connections per Second Maximum 10,000 new connections per second per gigabit of traffic with a 10 KB HTTP response size average packet size 550 bytes maximum 225,000 packets per second per gigabit of traffic. With smaller packet sizes coupled with high connection rates, this represents a very heavily used production network. 18

19 KB HTTP Response Size 20,000 Connections per Second Maximum 20,000 new connections per second per gigabit of traffic with a 4.5 KB HTTP response size average packet size 420 bytes maximum 300,000 packets per second per gigabit of traffic. With small packet sizes and extremely high connection rates, this is an extreme test for any DUT KB HTTP Response Size 40,000 Connections per Second Maximum 40,000 new connections per second per gigabit of traffic with a 1.7 KB HTTP response size average packet size 270 bytes maximum 445,000 packets per second per gigabit of traffic. With small packet sizes and extremely high connection rates, this is an extreme test for any DUT. 3.5 HTTP Connections per Second and Capacity with Delays Typical user behavior introduces delays between requests and reponses; for example, think time, as users read web pages and decide which links to click next. This group of tests is identical to the previous group except that these include a 5-second delay in the client request for each transaction. This has the effect of maintaining a high number of open connections throughout the test, thus forcing the DUT to utilize additional resources to track those concurrent connections KB HTTP Response with Delay Maximum 5,000 new connections per second per gigabit of traffic with a 21 KB HTTP response size average packet size 670 bytes maximum 185,000 packets per second per gigabit of traffic. There is a 5-second transaction delay, resulting in an additional 50,000 open connections per gigabit over the test described in section With average connection rates and average packet sizes, this is a good approximation of a real-world production network, and all DUTs should be capable of performing well throughout this test KB HTTP Response with Delay Maximum 10,000 new connections per second per gigabit of traffic with a 10 KB HTTP response size average packet size 550 bytes maximum 225,000 packets per second per gigabit of traffic, with a 5-second transaction delay resulting in additional open connections over the test described in section With large average packet sizes coupled with very high connection rates, this represents a very heavily used production network and is a strenuous test for any DUT. 3.6 Application Average Response Time: HTTP The results are recorded at each response size: 44 KB, 21 KB, 10 KB, 4.5 KB, and 1.7 KB, at a load level of 90% of the maximum throughput, with zero packet loss, as previously determined in section 3.4 and section Real-World Traffic Where previous tests provide a pure HTTP environment with varying connection rates and average packet sizes, the goal of this test is to simulate a real-world environment by introducing additional protocols and real content while still maintaining a precisely repeatable and consistent background traffic load. The result is a background traffic load that is closer to what may be found on a heavily utilized normal production network. 19

20 3.7.1 Real-World Protocol Mix (Enterprise Perimeter) Traffic is generated across the DUT comprising a protocol mix typically seen in an enterprise perimeter Real-World Protocol Mix (Financial) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large financial institution Real-World Protocol Mix (Education) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large educational environment. 20

21 4 Stability and Reliability Long-term stability is particularly important for an inline device, where failures can result in network outages. In addition, a device that cannot be managed or monitored can be a security risk. These tests assess the ability of a DUT to maintain security effectiveness while under normal and excessive utilization and while managing malicious traffic. The failure criteria for this category of tests are as follows: An inability to pass legitimate traffic while under attack Allowing prohibited, malicious traffic to pass through The DUT becomes unreachable or unusable during or after the test interval The DUT is required to remain operational, manageable, and capable of passing non-malicious traffic throughout these tests, and it is required to continue managing all previously identified malicious traffic. 4.1 Blocking under Extended Attack The DUT is exposed to a constant stream of security policy violations over an extended period of time. The device is configured to block and alert, and thus this test provides an indication of the effectiveness of both the security event management and alert handling mechanisms. A continuous stream of security policy violations mixed with legitimate traffic is transmitted through the DUT at a rate not to exceed 80% of the device s stated capacity for 8 hours with a steady stream of legitimate traffic mixed in. This test is not intended as a stress test in terms of traffic load. It is merely a reliability test in terms of consistency of its blocking performance. Any leak of exploit traffic will be considered a fail for this test. 4.2 Passing Legitimate Traffic under Extended Attack A continuous stream of legitimate traffic is transmitted through the DUT at a rate not to exceed 80% of the device s stated capacity for 8 hours with a steady stream of exploits mixed in. This test is not intended as a stress test in terms of traffic load. It is merely a reliability test in terms of consistency of passing legitimate traffic. The DUT is expected to remain operational and stable throughout this test and have no failure to pass legitimate traffic. The connection rate at the point of failure will be recorded. 4.3 Behavior of the State Engine under Load This test determines whether the DUT is capable of preserving state across a large number of open connections over an extended time period. At various points throughout the test (including after the maximum has been reached), it is confirmed that the DUT is still capable of inspecting and blocking traffic that violates the currently applied security policy, whilst confirming that legitimate traffic is not blocked (perhaps as a result of exhaustion of the resources allocated to state tables). The DUT must be able to apply policy decisions effectively based on inspected traffic at all load levels. 21

22 4.3.1 State Preservation Normal Load This test determines whether the sensor maintains the state of pre-existing sessions as the number of open sessions reaches 75% of the maximum, as determined in section A legitimate HTTP session is opened and the first packet of a two-packet exploit is transmitted. As the number of open connections approaches the maximum, the initial HTTP session is completed with the second half of the exploit, and the session is closed. If the sensor is still maintaining state on the original session, the exploit will be recorded. If the state tables have been exhausted, the exploit string will be seen as a non-stateful attack and will thus be ignored. Both halves of the exploit are required to trigger an alert. A product will fail the test if it fails to generate an alert after the second packet is transmitted, or if it raises an alert on either half of the exploit on its own State Preservation Maximum Exceeded This test is identical to except that the second half of the exploit is transmitted as the number of open sessions exceeds the maximum as determined in section This will determine whether the DUT is configured to drop new connections or age out old connections as state tables are exhausted. 4.4 Protocol Fuzzing and Mutation This test stresses the protocol stacks of the DUT by exposing it to traffic from various protocol randomizer and mutation tools. Several of the tools in this category are based on the ISIC test suite and other well-known test tools/suites. 4.5 Power Fail Power to the DUT is removed whilst passing a mixture of legitimate and disallowed traffic. The DUT should always be configured to fail closed by default no traffic should be passed once power has been cut. 4.6 Persistence of Data The DUT should retain all configuration data, policy data, and locally logged data once restored to operation following power failure. 22

23 5 Total Cost of Ownership and Value Organizations should be concerned with the ongoing, amortized cost of operating security products. This section evaluates the costs associated with the purchase, installation, and ongoing management of the DUT, including: Product Purchase the cost of acquisition Product Maintenance the fees paid to the vendor (including software and hardware support, maintenance, and updates) Installation the time required to take the device out of the box, configure it, put it into the network, apply updates and patches, perform initial tuning, and set up desired logging and reporting Upkeep the time required to apply periodic updates and patches from vendors, including hardware, software, and firmware updates 23

24 Appendix A: Change Log Version 1.0 March 7, 2014 Version 2.0 September

25 Contact Information NSS Labs, Inc. 206 Wild Basin Rd, Building A, Suite 200 Austin, TX USA This and other related documents available at: To receive a licensed copy or report misuse, please contact NSS Labs NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a retrieval system, ed or otherwise disseminated or transmitted without the express written consent of NSS Labs, Inc. ( us or we ). Please read the disclaimer in this box because it contains important information that binds you. If you do not agree to these conditions, you should not read the rest of this report but should instead return the report immediately to us. You or your means the person who accesses this report and any entity on whose behalf he/she has obtained this report. 1. The information in this report is subject to change by us without notice, and we disclaim any obligation to update it. 2. The information in this report is believed by us to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at your sole risk. We are not liable or responsible for any damages, losses, or expenses of any nature whatsoever arising from any error or omission in this report. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND EXCLUDED BY US. IN NO EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY, OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and/or software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet your expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners. 25