Web Gateway Security Appliances for the Enterprise: Comparison of Malware Blocking Rates A test commissioned by McAfee, Inc. and performed by AV-Test GmbH Date of the report: December 7 th, 2010 (last update: February 22 nd, 2011) Executive Summary In November 2010, AV-Test performed a comparative review of four web gateway security solutions for the enterprise to determine their malware detection and blocking capabilities. McAfee commissioned AV-Test to run an independent test of McAfee Web Gateway, Blue Coat, Websense and Cisco Ironport. In order to ensure a fair review, the sponsor has not supplied any samples or had any influence or any prior knowledge regarding the samples being tested. The following test scenarios are standard tests that AV Test does on a regular basis for gateway antimalware solutions. There are two ways to stop malware at the gateway level today. These are signature-based detection options that protect against known vulnerabilities and malware, and heuristic/generic detection options which work on both known and unknown vulnerabilities and malware, including Zero-Day. These tests cover both. 1. Zero-Day Testing: Testing of the effectiveness of dynamic URL filtering capabilities and protection against zero-day malware by accessing real URLs that host malicious downloads, 2. PE Malware Test: Detection of relevant current malicious Win32 portable executable (PE) files, also referred as Zoo viruses, which are not older than 3 months at the start of the review, 3. Non-PE Malware Test: Detection of current malicious non-pe files, such as PDF exploits, as well as files including malicious scripts and macros for Microsoft Office and other applications, which are also not older than 3 months. Breaking out the data by test shows big variation between vendors for zero day and Non PE threats: Zero Day Test 181 147 106 50 200 PE 203,410 191,303 185,012 173,043 203,412 Non PE 5,562 5,555 880 3,991 5,619 Detect Rate 99.96% 94.16% 88.90% 84.64% 209,231 Figure 1: Summary of the appliance test results The best overall malware detection results were achieved by McAfee Web Gateway. It reached the highest overall score and the highest individual scores for the three distinct malware test cases. A summary of the results can be found in the following chart: 1
Detection Rate 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 99.99% 98,86% 94,05% 98,99% 90,50% 90,95% 85,07% 73.5% 71,03% 53% 25% 15,66% Blue Coat Cisco McAfee Websense 0-day protection rate PE malware detection Non-PE malware detection Figure 2: Detection Rates Overview With the increasing volume of malware, targeted attacks and advanced persistent threats spreading through the Internet these days, the danger of getting infected has increased as well. In the year 2000, AV-Test received more than 170,000 new unique samples, and in 2009, the number of new samples grew to over 12,000,000 and reached 18,000,000 in 2010. The growth of these numbers is displayed in Figure 3. 20.000.000 18.000.000 16.000.000 14.000.000 12.000.000 10.000.000 8.000.000 6.000.000 4.000.000 2.000.000 0 New unique samples added to AV-Test's malware repository (2000-2010) 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 Dec Nov Oct Sep Aug Jul Jun May Apr Mar Feb Jan Figure 3: New malware samples per year 2
To protect the enterprise network against the growing number of threats a multilayered security setup is recommended. The layers at a minimum should include the enterprise firewall, a web- and content-filter for every kind of traffic, which is the topic of this document, and an endpoint protection product as the last barrier for the malware. A clever combination of those layers makes it hard for the attacking site to infiltrate the enterprise network. Products Tested The following four products were tested, using the latest signature updates available at the beginning of the test: Product Software Version AV-Engine Blue Coat SG510 with ProxyAV 6.1.1.1 Kaspersky Cisco IronPort S160 6.3.3-015 McAfee Anti-Malware Engine, Webroot McAfee WG-5000 7.0.1.5.0 (8505) McAfee Gateway Anti-Malware Engine (1) Websense V10000 7.5 Websense (1) McAfee Gateway Anti-Malware Engine includes both signature based and behavioral components Figure 4: List of tested products Methodology and Scoring Platform All tests have been performed with identical PCs equipped with the following hardware which was used to fetch content from the internet and feed the malware samples to each gateway solution: Intel Xeon Quad-Core X3360 CPU 4 GB Ram 500 GB HDD (Western Digital) Intel Pro/1000 PL (Gigabit Ethernet) NIC The operating system was Windows XP Service Pack 3 with all updates applied, running another Windows XP Service Pack 3 with an Apache 2 webserver within a VirtualBox VM, as shown in figure 5. 3
Figure 5: Test platform overview Testing methodology In the interest of fairness, testing was performed with each solution configured for the best possible security level they offer, whether delivered from the box itself, the cloud or both. The purpose of the test was to assess malware detection and not throughput, hence no attempt was made to use appliances of equal throughput and the test results are not affected by the size of the appliance. The test has been performed according to the following methodology: 1. Internet Access. The appliances had access to the Internet at all times in order to use any inthe-cloud queries that each of the solution may offer. 2. Product Configuration. All products were run with their best possible configuration by AV- Test s knowledge. The last signature update was executed before the beginning of the testing. 3. Testing. All files, except for the malicious URLs, were downloaded via http from the virtual webserver to the host system using wget with http proxy set to one of the appliances. For the URL testing, an additional client has been used without proxy configuration to download the reference samples from the Internet. 4. Analysis. The downloaded files were compared with the original files (reference files at URL testing) by MD5 hash. For verifying the results, the wget and appliance report files were analyzed. The static set of files consisted of 203,412 malicious PE files (Zoo malware) and 5,619 non-pe files. The dynamic tests were performed using 200 working malicious URLs. To insure fairness in the 4
dynamic tests, five clients requested and downloaded the malicious URLs simultaneously, with four clients using one of the appliances as http proxy and one client using no http proxy as reference. Test Results Test #1: Zero-Day protection rate The following table summarizes the effectiveness of the solutions in identifying zero day threats. Zero-day threats are typically identified through the gateway s ability to open up content for inspection coupled with whatever proactive scanning abilities and cloud intelligence a vendor may provide. A block can be based on URL filtering or Web Reputation services, by signatures or heuristic scanning of the provided content, and other inspection and filtering technologies. In the case of the blocking of malicious URLs McAfee scored highest in detection of all products tested with a 90.5% protection rate (see Figure 6): Zero Day Rate 181 147 106 50 200 Zero Day Detection % 90.50% 73.50% 53.00% 25.00% Test #2: PE malware detection Figure 6 The total number of malicious samples tested was 203,412. This includes the following number of samples: 19,641 Backdoors, 3951Bots, 11787 Viruses, 18362 Worms, 130,853 Trojan Horses, 6,948 potentially unwanted applications (PUA) as well as 11,870 rogue applications (e.g. Fake AV). In case of the Trojan Horses category, the following threats have been used: 14,094 downloader, 6,989 dropper, 102,462 generic malicious programs, and 7,308 password-stealing programs (PWS). This test focuses on the generic malware detection and blocking capabilities, especially on the signature-based detection as well as generic and heuristic technologies. McAfee achieved the best score for the malware testing, detecting almost all Zoo samples (see figure 7): PE Detection Rate 203,410 191,303 185,012 173,043 203,412 PE Detection % 99.99% 94.05% 90.95% 85.07% Test #3: Non-PE malware detection Figure 7 While many companies incorporate complete blocking of PE files for security reasons, so a product would filter out 100% of the test cases shown in the Test #2 (when this option is enabled), many other file formats like DOC or PDF exist which can also be very dangerous due to exploits, scripts or macros, but blocking would not be an option. 5
The following test set includes 5,619 samples with the 27 generic file format exploits, 4,416 PDF exploits and 16 Shell-based exploits, 5 current Excel and 108 Word macro viruses, as well as 10 Linux, 1,035 script and 1 Symbian malware. The detection of Blue Coat and McAfee were very high against the tested malicious non-pe files. On the other hand, the Websense detection rate was very low in this test. For example, no malicious PDF files were identified at all and only one macro virus was blocked (see figure 8). Non PE 5,562 5,555 880 3,991 5,619 Detect Rate 98.99% 98.86% 15.66% 71.03% Figure 8 Conclusion With the given configuration, McAfee's WG-5000 scored best in all three reviewed malware detection categories. However, the results can vary over time and may vary with other configurations as the vendors offer several options for AV scan engines and web filter modules. It should also be noted that only the malware detection and blocking was reviewed, not the user interface (e.g. configuration options), false positives or scan performance (speed). Copyright 2010-2011 by AV-Test GmbH, Klewitzstr. 7, 39112 Magdeburg, Germany Phone +49 (0) 391 60754-60, Fax +49 (0) 391 60754-69, Web http://www.av-test.org 6