THREAT ISOLATION TECHNOLOGY PRODUCT ANALYSIS

Transcription

1 THREAT ISOLATION TECHNOLOGY PRODUCT ANALYSIS v Jayendra Pathak, Ken Baylor, Ph.D Overview NSS Labs performed an independent test of the threat isolation technology. The product was subjected to thorough testing at the NSS facility in Austin, Texas, based on the Security Stack methodology v1.5 available on This test was conducted free of charge and NSS did not receive any compensation in return for Bromium s participation. For additional information on Isolation technology, please refer to the NSS Analysis Brief entitled Defeating Advanced Malware in 2013: Leveraging Threat Isolation Technologies to Avoid Infection. While the companion Comparative Analysis Reports (CAR) on security, performance, management, and total cost of ownership (TCO) will provide comparative information about all tested products, this individual Product Analysis Report (PAR) provides detailed information not available elsewhere. NSS evaluation of isolation technologies is designed to determine the effectiveness of a given solution at protecting against new and highly evasive attacks as used in Targeted Persistent Attack (TPA) campaigns. The testing focus is on desktop endpoint protection rather than data center server implementations. Product Results v % Embedded Exploits & Malware 100% Drive- by Exploits & Malware 100% Performance Overhead 9% Using the default policy, the isolated 100% of attacks against desktop applications, preventing them from compromising or altering the system, while incurring a total performance overhead of 9%.

2 Table of Contents Overview... 1 Security Effectiveness... 3 Performance... 4 Boot Time... 4 Memory Utilization... 4 Time to start an application (warm start)... 5 Internet Explorer Microsoft Word Microsoft Excel Management & Configuration... 6 Total Cost of Ownership (TCO)... 7 Installation (in Hours)... 7 Purchase Price and Total Cost of Ownership... 8 Value: Total Cost of Ownership Per Protected User... 8 Detailed Product Scorecard... 9 Test Harness Details Test Methodology Contact Information Table of Figures Figure 1: Security Effectiveness: Coverage by Exploit Type... 3 Figure 2: Performance: Increase in Boot Time... 4 Figure 3: Performance: Increase in Memory Utilization... 4 Figure 4: Performance: IE8 Warm Start... 5 Figure 5: Performance: MS Word 2010 Warm Start... 5 Figure 6: Performance: MS Excel 2010 Warm Start NSS Labs, Inc. All rights reserved. 2

3 Security Effectiveness The objective of the Security Stack methodology (v1.5) is to determine the effectiveness of the threat isolation and protection offered by a system under test (SUT) against drive- by exploits and embedded exploits. This test is ideal for assessing the ability of isolation technologies to combat modern malware since it is a live test of the most current and most evasive malware. For threat isolation testing, NSS defines success based upon the product successfully isolating the malicious binary delivered from the exploit and executed/installed on the system. No traces of the malicious sample should remain in the system once the isolated task is closed. NSS defines a failure based the exploit successfully downloading installing/executing malware, and where traces of the malicious code remain on the host system once the task is closed. The assessment comprises three main categories: 1) Embedded exploits. These include malicious PDF documents that will trigger a vulnerability in a PDF reader when opened in order to install malware. 2) Drive- by exploits. These include malicious URLs that will trigger a vulnerability in different combinations of operating system (OS) and application(s), resulting in malware installation. 3) Custom payload targeting and exploitation. 180# 160# 140# 120# 100# 80# 60# 40# 20# 100%# 90%# 80%# 70%# 60%# 50%# 40%# 30%# 20%# 10%# Tested# Blocked# Block#Rate# 0# Live#Embedded#Exploits# Live#Drive=by#Exploits/Malware# Custom#payload#targeFng#&# exploitafon# 0%# Figure 1: Security Effectiveness: Coverage by Exploit Type Despite these attacks being at the cutting edge of current malware technology, performed well. The SUT defeated all 166 embedded exploits (delivered via to an Outlook client), all 153 drive by samples (via HTTP and HTTPS) and all 15 Metasploit attacks, which incorporated advanced obfuscation and evasion techniques in an attempt by to bypass protection. Due diligence was taken to ensure that no traces of malicious binaries remained on the system once each task was closed. Task- specific tools were used to determine whether malicious binaries remained on the system, and whether or not they were capable of changing system settings NSS Labs, Inc. All rights reserved. 3

4 Performance Since host- based software can have a considerable impact on the usability of a workstation, the isolation technology test measures the performance and memory utilization of the software stack. Each test is first performed without the security software to establish a baseline. The endpoint protection (isolation) software is then installed, and the test is rerun to generate a second set of results that are representative of the end user experience when utilizing the isolation software. The delta between the baseline and isolation results provides an accurate assessment of the overhead imposed by the SUT. Each test is executed at least three hundred eighty- five (385) times, providing a margin of error of 5%. In addition, those results that fall outside two standard deviations from the mean (statistical outliers) are discarded. This provides a confidence level of 95 out of 100. The resulting data is then averaged and reported. Boot Time Net increase in time required to boot the system. % Increase 0% 5% 10% 15% 20% 25% 22% Figure 2: Performance: Increase in Boot Time Memory Utilization Net increase in memory usage when idle. % Increase 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 16.3% Figure 3: Performance: Increase in Memory Utilization used 16.3 % more memory when installed on the system as opposed to the system that didn t have the product installed. However, once the prerequisite amount of memory is installed on the system, users will experience negligible impact on system performance NSS Labs, Inc. All rights reserved. 4

5 Time to start an application (warm start) Internet Explorer 8 % Increase 0.0% 0.5% 1.0% 1.5% 2.0% 2.5% 1.9% Figure 4: Performance: IE8 Warm Start Microsoft Word 2010 % Increase 0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0% 5.1% Figure 5: Performance: MS Word 2010 Warm Start Microsoft Excel 2010 % Increase 0.0% 0.5% 1.0% 1.5% 2.0% 2.5% 3.0% 3.5% 3.1% Figure 6: Performance: MS Excel 2010 Warm Start Test Performance Performance Impact Impact % Boot Time 5 Sec 22% Memory 170 MB 16% Launch IE8.06 Sec 2% Launch Word.09 Sec 5% Launch Excel.02 Sec 3% Average 10% 2013 NSS Labs, Inc. All rights reserved. 5

6 Management & Configuration Security solutions are complicated to deploy; essential systems such as centralized management console options, log aggregation, and event correlation/management systems further complicate the purchasing decision. Understanding key comparison points will allow customers to model the overall impact on network service level agreements (SLAs), estimate operational resource requirements to maintain and manage the systems, and better evaluate required skill / competencies of staff. As part of this test, NSS performed in- depth technical evaluations of all the main features and capabilities of the enterprise management systems offered by each vendor, covering the following key areas: General Management and Configuration how easy is it to install and configure devices, and deploy multiple devices throughout a large enterprise network? Policy Handling how easy is it to create, edit and deploy complicated security policies across an enterprise? Alert Handling how accurate and timely is the alerting, and how easy is it to drill down to locate critical information needed to remediate a security problem? Reporting how effective and customizable is the reporting capability? For additional analysis concerning enterprise management capabilities and total cost of ownership, refer to the TCO and Management Comparative Analysis Reports (CAR) NSS Labs, Inc. All rights reserved. 6

7 Total Cost of Ownership (TCO) Implementation of security solutions can be complex, with several factors affecting the overall cost of deployment, maintenance and upkeep. All of these should be considered over the course of the useful life of the solution. Product Purchase the cost of acquisition. Product Maintenance the fees paid to the vendor (including software and hardware support, maintenance and other updates). Installation the time required to take the device out of the box, configure it, put it into the network or on the endpoint, apply updates and patches, and set up desired logging and reporting. Upkeep the time required to apply periodic updates and patches from vendors, including hardware, software, and other updates. Management day- to- day management tasks including device configuration, policy updates, policy deployment, alert handling, and so on. For the purposes of this report, capital expenditure (CAPEX) - the cost of acquisition and installation and operational expenditure (OPEX) items (ongoing management and labor costs) for multiple devices plus centralized management systems are modeled in the separate Management and TCO Comparative Analysis Reports (CAR). Installation (in Hours) This table details the number of hours of labor required to install the product under test with central management. This reflects the amount of time taken for NSS engineers, with the help of vendor engineers, to install and configure the product to the point where it operates successfully in the test harness, passes legitimate traffic and blocks/detects prohibited/malicious traffic. This closely mimics a typical enterprise deployment scenario for a single desktop. Centralized management options are covered in the Management CAR. Costs are based upon the time required by an experienced security engineer ($75 per hour fully loaded), allowing us to hold constant the talent cost and measure only the difference in time required for installation. Readers should substitute their own costs to obtain accurate TCO figures. Product Product Installation (Hrs) Central Management Installation (Hrs) v NSS Labs, Inc. All rights reserved. 7

8 Purchase Price and Total Cost of Ownership Calculations are based on vendor- provided pricing information. Where possible, the 24/7 maintenance and support option with 24- hour replacement is utilized, since this is the option typically selected by enterprise customers. Prices are for 500 license and support / maintenance as well as central management (CM) to deploy and manage the product on 500 desktops. For additional TCO analysis, including different size central management, refer to the Management CAR. Product Licenses Purchase Support Year 1 Cost Year 2 Cost Year 3 Cost 3 Year TCO v $75,000 $15,000 $76,500 1 $15,000 $15,000 $106,500 Year 1 Cost is calculated by adding installation costs ($75 per hour fully loaded labor x installation) + purchase price + first- year support fees Year 2 Cost consists only of support fees Year 3 Cost consists only of support fees This provides a TCO figure consisting of software, installation and support costs only. Additional management and labor costs are excluded, since they are modeled extensively in the Management and TCO CARs. Value: Total Cost of Ownership Per Protected User There is a clear difference between price and value. The least expensive product does not necessarily offer the greatest value if it offers significantly lower performance than only slightly more expensive competitors. The best value is a product with a low TCO and high level of secure throughput (security effectiveness x performance). The following table illustrates the relative cost per unit of work performed: Protected User Product Licenses Protection v Performance Impact 3 Year TCO Price / Protected User % 10% $106,500 $237 Price per Protected User was calculated by taking the three- year TCO and dividing it by the product of the number of licenses x protection x (1 performance impact) = Price/ Protected User. Costs for central management solutions may be extra. For additional TCO analysis, including CDM, refer to the TCO and Management CARs. 1 Note that the annual support cost is already included in the purchase price for the first year NSS Labs, Inc. All rights reserved. 8

9 Detailed Product Scorecard The following chart depicts the status of each test with quantitative results where applicable. More detailed scorecards that include enterprise management capabilities and TCO calculations are available in the appropriate Comparative Analysis Reports (CAR). Description Security Effectiveness Exploits Result Live Embedded Exploits 100% Live Drive- by Exploits/Malware 100% Custom payload targeting & exploitation 100% Performance Impact OS Boot Time 22% Memory 16% Time to Start an Application Internet Explorer 8 2% Word % Excel % 2013 NSS Labs, Inc. All rights reserved. 9

10 Test Harness Details Hardware Memory CPU ASUS Ultrabook (Model No: K53E- 1BSX) 4GB RAM Intel(R) Core i5-2450m GHZ 2.50 GHZ) Security testing focused on applications that NSS research shows to be targeted actively by criminal organizations. Attacks were conducted via live drive- by exploits and live embedded exploits. Operating System Applications Windows 7 Enterprise 64 Bit FlashPlayer Adobe Reader (9.0.0) Java Run time Environment 6 Update 27 JRE6u27 Java Run time Environment 6 Update 18 Java Run time Environment 7 Java Run time Environment 7 Update 4 Vulnerable applications used during testing were those shown by NSS research to be in common use in typical enterprise environments: Applications Used for Performance Testing Internet Explorer 8 MS Word 2010 MS Excel NSS Labs, Inc. All rights reserved. 10

11 Test Methodology Methodology Version: Security Stack v1.5 All Test IDs in this report refer to the methodology document, not necessarily to sections in this report. A copy of the test methodology is available on the NSS Labs website at Contact Information NSS Labs, Inc. 206 Wild Basin Rd Building A, Suite 200 Austin, TX (512) info@nsslabs.com v This and other related documents available at: To receive a licensed copy or report misuse, please contact NSS Labs at +1 (512) or sales@nsslabs.com NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please note that access to or use of this report is conditioned on the following: 1. The information in this report is subject to change by NSS Labs without notice. 2. The information in this report is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at the reader s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this report. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON- INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader s expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners NSS Labs, Inc. All rights reserved. 11