CONSUMER EPP COMPARATIVE ANALYSIS

Transcription

1 CONSUMER EPP COMPARATIVE ANALYSIS Socially Engineered Malware Randy Abrams, Jayendra Pathak, Mohamed Saher, Ahmed Garhy Tested Vendors AVG, F- Secure, Kaspersky, McAfee, Microsoft, Symantec, Trend Micro Tested Products AVG Internet Security Version F- Secure Internet Security 2014 Version 1.89 build 205 Kaspersky Internet Security (c) McAfee Security Center Version 12.8 Microsoft Security Essentials Antimalware Client Version Norton Internet Security Version Trend Micro Titanium Maximum Security Version Environment Operating System: Windows 7 Enterprise SP1 32 bit Windows Defender disabled Browser: Internet Explorer Version Smart Screen disabled

2 Overview For several years, the use of social engineering has accounted for the bulk of the attacks used by cyber criminals against consumers. Whether used in conjunction with phishing, malware, or websites booby- trapped with exploits, consumers are falling prey to deceptive tactics, and are therefore increasingly relying upon browsers and endpoint protection products (EPP) to shield them from harm. EPP products include multiple protection technologies that include malware protection, firewalls, whitelisting, blacklisting, cloud- based content agnostic malware protection (CAMP) systems, and other security mechanisms to secure computers and other devices. NSS Labs performs comparative tests of EPP products for three of the most relevant type of attacks. Socially engineered malware (SEM) protection Exploit protection Phishing protection This comparative tests EPP products for their ability to block socially engineered malware attacks. Many consumers choose an EPP product based on a variety of test results. Tests of a product s ability to block malware are of greatest concern to most consumers. Consumers will consider factors such as performance and the quality of technical support after selecting a product. The perception of unacceptable performance impact or poor technical support often results in the selection of a different solution, regardless of relative quality. The quality of technical support and the determination of acceptable system impact are subjective decisions that each consumer decides for himself or herself. Product protection metrics are measured objectively in NSS testing. SmartScreen Application Reputation in Internet Explorer (IE) must be disabled in this test because, as demonstrated in the 2013 browser security comparative analysis for socially engineered malware, 1 Microsoft s SmartScreen Application Reputation technology blocks virtually all SEM. With virtually all SEM attacks blocked by the browser, it would be impossible to test the ability of EPP products to block SEM. This does not mean that the EPP products are redundant, however. Figure 10 in the 2013 browser security comparative analysis for socially engineered malware, demonstrates that there are some instances where none of the browsers, including IE, detect SEM. This is especially true in the extremely early stages of an attack where EPP products enjoy a small advantage over the browsers. Additionally, browsers offer no protection against malware originating from the network or from removable media. Browsers will rarely protect against exploits, another prevalent source of system compromise. 1 browser- security- comparative- analysis- socially- engineered- malware 2

3 McAfee Norton Trend Micro Kaspersky F-Secure 98% AVG 91% Microsoft 83% 75% 80% 85% 90% 95% Combined Protection Rate Figure 1 Combined Protection Rate for SEM EPP products rely upon sophisticated cloud- based technologies to prevent the download of known and unknown malware. If a file is not blocked by the cloud- based technologies, the EPP product will again scan it on- access using signatures and heuristics to further protect the user. Figure 1 displays the effectiveness of the tested products in blocking SEM attacks when downloads are attempted or when execution is blocked on- access. NSS Labs Findings: Robust CAMP implementations significantly improve protection and level the playing field. Consumers have an increasing number of effective EPP products from which to choose. The average amount of time to block new SEM significantly differentiates the top products from second- tier products. Price, perceived system impact, and technical support have become the key differentiators for the best performing EPP products. 3

4 Table of Contents Tested Vendors... 1 Tested Products... 1 Environment... 1 Overview... 2 NSS Labs Findings:... 3 Analysis... 5 Ramping Up to Peak Protection... 7 Consistency of Protection... 8 Average Time to Block... 8 Who Needs EPP Anyway?... 9 Test Methodology Contact Information Table of Figures Figure 1 Combined Protection Rate for SEM... 3 Figure 2 Combined Block Rates by Technology... 6 Figure 3 Average SEM Block rate on Download... 7 Figure 4 Malware Detection Response Histogram... 7 Figure 5 Socially Engineered Malware Block Rate Over Time... 8 Figure 6 Average Time to Add Protection

5 Analysis NSS tested EPP products for SEM protection effectiveness in a continuous series of tests that were run from December 15, 2013 through January 13, ,000 test cases were run with 549 unique URLs and 463 unique malware samples. In 86 instances identical malware samples existed at different URLS. If identical malware samples are blocked at one URL but not another it is essential that the EPP product block the malware on the host. NSS attempted to execute malware that initially evaded the cloud technology to determine if the EPP products succeeded at protecting the user. Microsoft currently attempts to position its EPP product, Microsoft Security Essentials, as the baseline by which all other products are judged, rather than trying to position itself as a competitor. The reason for this approach is twofold. First and foremost, Microsoft wants to force EPP vendors to constantly improve the technology upon which Microsoft depends to protect the Windows brand. The second reason for positioning Microsoft as the baseline is to assure customers they are receiving value for their product selection. In this test of SEM protection, all of the EPP solutions outperformed Microsoft Security Essentials (MSE) in all but one category. Two products failed to perform better than MSE in the average time required to add protection against new threats. The scores for the products performing above the MSE baseline have only a 9% spread in effectiveness, with MSE scoring 17% less effective than the leading performers. CAMP has emerged at the forefront of proactive protection technologies. 2 Despite advanced heuristic approaches to identify malicious software, a well- designed CAMP implementation provides the most robust malware download protection against SEM and drive- by attacks. Technically, CAMP technology does not block malware, it blocks files based upon reputational characteristics. The technology is capable of achieving an exceptionally high rate of blocking malicious files while blocking only a small percentage of benign programs. EPP products layer additional protective mechanisms in the cloud with CAMP. 2 takes- scammers- camp 5

6 McAfee Norton Trend Micro Kaspersky 98.4% 98.0% 98.9% 98.5% 1.6% 2.0% 0.7% 1.1% F- Secure 96.9% 1.6% AVG 89.3% 1.6% Microsoi 83.4% 0.0% 75% 80% 85% 90% 95% Block Rate on Download Block Rate on Execujon Figure 2 Combined Block Rates by Technology The proportion of malware blocked on download disproves the persistent myth that antivirus products use only static signatures. The vast majority of SEM is blocked prior to landing on the hard drive. In order to prevent malware from reaching the host, strong reputational mechanisms are required. Figure 2 illustrates the effectiveness of cloud technology against SEM. 98% or more of the SEM attacks are blocked on download. All of the SEM blocking performed by MSE is accomplished using download protection technologies. With EPP products performing comparably on combined protection scores, differentiation of effectiveness requires additional metrics. The time required to achieve maximum effectiveness is one of the metrics used to differentiate products with comparable overall protection scores. Figure 3 isolates the effectiveness of the tested products blocking on download, which is where most of the protection is offered. The combined SEM block rate is the true measure of protection; however, an evaluation of the relative effectiveness in download protection reveals where investment in cloud technology can augment security effectiveness. 6

7 Trend Micro McAfee Norton Kaspersky AVG F-Secure Microsoft 99% 98% 98% 96% 89% 85% 81% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Average Block Rate on Download Figure 3 Average SEM Block rate on Download Ramping Up to Peak Protection 90% 80% 70% Coverage % 60% 50% 40% 30% 20% 10% 0% 1d 2d 3d 4d 5d 6d 7d Total Trend Micro 98.7% 98.9% 98.9% 98.9% 98.9% 98.9% 98.9% 98.9% Kaspersky 96.0% 96.7% 96.9% 96.9% 96.9% 96.9% 96.9% 98.5% McAfee 98.2% 98.4% 98.4% 98.4% 98.4% 98.4% 98.4% 98.4% Norton 97.1% 97.6% 97.6% 97.6% 97.6% 97.8% 97.8% 97.8% F-Secure 67.0% 81.5% 87.2% 89.4% 90.0% 90.6% 91.4% 96.9% AVG 83.1% 85.4% 86.2% 86.2% 86.2% 86.2% 86.2% 89.3% Microsoft 79.1% 80.9% 81.1% 81.1% 81.1% 81.1% 81.1% 83.4% Figure 4 Malware Detection Response Histogram 7

8 The histogram in Figure 4 begins to differentiate products in the early stages of blocking downloads from new URLs containing SEM. While there is a significant spread in performance at the end of the first day of testing, by the end of the testing period, five of the seven products tested are virtually tied with a spread of three percentage points. Consistency of Protection 90% 80% 70% 60% 50% 40% 30% 20% Trend Micro McAfee Norton Kaspersky AVG F-Secure Microsoft 10% 0% Figure 5 Socially Engineered Malware Block Rate Over Time Figure 5 depicts malware protection over time. Trend Micro Titanium Maximum Security proved the most consistent product for SEM download protection over time, maintaining download protection against SEM for most of the test. Of all the products tested, McAfee Security Center started strongest at download protection, rarely dropping below 98% effectiveness. Norton Internet Security offered as high as SEM download protection, generally sustaining protection rates above 96%. Kaspersky Internet Security rarely dropped below 95% download protection against SEM, peaking at. AVG Internet Security maintained protection rates within 10% of the leaders in this test; however, at the start of the interval, protection levels fell as low as 60% and then dipped to 80% by the end of the test. The results of the download protection against SEM over time begin to differentiate products, particularly for consistently high protection levels. Average Time to Block The average amount of time required for a vendor to add detection for a new threat is an important metric. Throughout the testing, little has differentiated the top four products. The average time to block separates the top three products in the field from the other four in Figure 6. Average time to block is also the only metric in which the baseline set by MSE has not been exceeded by all of the products. 8

9 Product Hours Trend Micro 0.24 McAfee 0.36 Norton 0.43 Kaspersky Microsoft AVG F- Secure Average Figure 6 Average Time to Add Protection Trend Micro Titanium Maximum, McAfee Security Center, and Norton Internet Security all averaged between 15 and 30 minutes to add detection for new threats. The next closest competitor, Kaspersky Internet Security, beat the average of the products in this metric, but still took more than 20 times longer to add protection than did Norton Internet Security. Both AVG and F- Secure were well below the MSE baseline. Who Needs EPP Anyway? Consumers who are educated about social engineering attacks generally will not be victims of SEM, regardless of the EPP selection. Consumers who use current operating systems and current versions of Internet Explorer and Chrome are far less reliant on their EPP product to protect them from SEM attacks. Users who are not proficient in identifying social engineering attacks and who use browsers such as Firefox, Opera, and Safari are most reliant on EPP products with strong reputational systems for protection against SEM attacks. Even with education and the use of more secure browsers, a second line of defense is of benefit. The ability to protect against exploits requires different technology than the approaches to blocking SEM. Drive- by downloads may evade reputational systems and rely on host- based protection mechanisms. Malware introduced through vectors other than web browsing requires protection that browsers cannot offer. The combination of SEM protection testing, exploit protection testing, and phishing protection testing ultimately provides a more comprehensive picture of the overall protection capabilities of an EPP product. 9

10 Test Methodology Security Stack: Test Methodology v1.5 A copy of the test methodology is available on the NSS Labs website at Contact Information NSS Labs, Inc. 206 Wild Basin Rd Building A, Suite 200 Austin, TX USA +1 (512) info@nsslabs.com This analyst brief was produced as part of NSS Labs independent testing information services. Leading products were tested at no cost to the vendor, and NSS Labs received no vendor funding to produce this analyst brief NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please note that access to or use of this report is conditioned on the following: 1. The information in this report is subject to change by NSS Labs without notice. 2. The information in this report is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at the reader s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this report. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON- INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader s expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners. 10