Advanced Endpoint Protection

Transcription

1 BEST PRACTICES Advanced Endpoint Protection ENTERPRISE SELF-TESTING BEST PRACTICES PUBLISH DATE TBD Author NSS Labs Overview Security effectiveness refers to the ability of security devices to respond accurately and effectively to a range of threats. It is often the most difficult metric to evaluate because it requires expertise with potentially dangerous live exploits and malware. Independent testing houses can perform this type of testing on behalf of an enterprise when in-house expertise does not exist, but security professionals that engage these independent testing houses must first ensure that the breadth of the testing is adequate for their specific needs. It is neither necessary nor desirable to verify the effectiveness of every technology included with an advanced endpoint protection (AEP) product, but some minimal level of in-house verification of fitness for purpose is important. This type of testing should confirm that the product will not block the enterprise s legitimate traffic and applications, and that it is capable of accurately detecting and reacting to the wide range of commonly available exploits and malware used by threat actors. During testing, it is critical that fresh malware and exploits are used. A basic security effectiveness test bed can be created at a relatively low cost using virtualization technology and commonly available test tools. Virtual machines can be used to create an environment that is safe and repeatable, ensuring that live exploits and other malware do not accidentally infect other machines on the network.

2 Table of Contents Overview... 1 Analysis... 3 Create a Test Methodology... 3 Seek out sources of high-quality independent product testing and advice... 3 Determine the use case... 4 Define evaluation criteria... 4 Which exploits will be most representative of the use case you have selected?... 4 What level of coverage is required in the security effectiveness tests?... 4 Select appropriate test tools... 4 Configure the Test Tools... 6 Configure the AEP Product... 6 Run the Test... 6 False positives... 7 Evaluate the Results... 7 Scrutinize Vendor Track Record... 8 Updates... 8 Accuracy... 8 Do not confuse claims with coverage... 8 Carefully review awards and validation reports... 8 Implement Continuous Testing Initiatives... 9 Appendix A Exploit Testing Procedure Prerequisites Building a VM harness for exploit testing Step-by-step testing procedure Appendix B Malware Testing Procedure Prerequisites Building a VM harness for malware testing Step-by-step testing procedure Contact Information White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 2

3 Analysis Security effectiveness refers to the ability of complex security software to respond accurately and efficiently to a wide range of common threats (breadth), as well as to provide comprehensive protection against threats targeting specific applications (depth). This needs to be achieved without blocking or affecting legitimate traffic (accuracy). Security effectiveness is often the most difficult area to evaluate effectively since it requires expertise with attack traffic, and even potentially dangerous live threats. Effective handling of inspected traffic can include: Using reputation, signatures, heuristics, behavior, emulation, sandboxing, and other methods to recognize malicious traffic and applications Responding to malicious events according to an enterprise s endpoint security policy These steps need to be performed at the vendor-advertised hardware and software specifications without adversely affecting normal operations. Although breadth is extremely important, accuracy is critical if an AEP product blocks or affects legitimate traffic once it is deployed, its security efficacy can no longer be trusted. There are a number of key steps that should be followed to ensure success when evaluating the security effectiveness of an AEP product. Create a Test Methodology Creation of the test methodology is the most crucial step in selecting an AEP product. An effective test methodology enables the enterprise to select cost-effective security solutions that align with internal requirements for performance and system integration. Creation of a test methodology should therefore be the first step in any product evaluation and should be closely correlated to the business and functional requirements of the enterprise. Please refer to NSS Advanced Endpoint Protection Enterprise Self-Test Methodology 1 on how to go about creating a test plan to evaluate products. Seek out sources of high-quality independent product testing and advice NSS subscribers should make extensive use of the numerous resources available to them as part of their subscription services. Vendors that have never submitted their products for such tests or that rely purely on magazine reviews to market their wares should be treated with caution. It is possible that they have concerns about the performance of their own products, or that they simply do not understand the value of testing in their own QA processes. Reduce time and cost associated with testing by using independent test reports to create a short list. It is often a straightforward process to identify and eliminate devices that are unsuitable for your own environment because of issues such as poor security coverage, poor performance, or lack of management features. 1 NSS Labs Advanced Endpoint Protection Enterprise Self-Test Methodology White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 3

4 IT organizations can save significant time and money by taking and modifying one or more of NSS extensive test methodologies, which are free and available to everyone. While these methodologies will rarely be suitable immediately for your own environment (test labs have to test extremes and corner cases ), the basic types of test described within can be duplicated with changes for your own needs. Determine the use case Determine exactly how you want to test the AEP products on your short list. Do not underestimate the importance of accurately determining which assets must be protected. Are you looking to protect a client or server operating system? Which operating system do you need to protect? Which applications must be protected? These questions must be answered in order to help you understand the attack surface you are protecting. For example, independent tests may have commented adversely on a lack of coverage for Linux servers, but if you are a Microsoft Windows-only environment, that would be of little concern. Your own test methodology should then ensure that a Windows-centric policy is deployed and that security effectiveness testing focuses on Windows exploits. Define evaluation criteria Security products should be evaluated against enterprise-specific criteria, which are then used to determine test tool configurations, as well as provide benchmarks against which the test results can be evaluated. Typical questions that security testing professionals should ask include: Which exploits will be most representative of the use case you have selected? If your intended use case is a Windows client operating system, then there is little point in testing Ubuntu Linux exploits. While you may wish to include a variety of exploits to determine how the AEP product responds, you should focus your efforts on ensuring that it will specifically protect the security assets located behind it. What level of coverage is required in the security effectiveness tests? An enterprise may have the same AEP product installed on the same OS but have different security effectiveness requirements depending on use case. For example, an enterprise might complete an audit of its underlying attack surface prone to exposure and conclude that it requires a high level of prevention (for example, a block rate of between percent) for all web application-based exploits, but might have a lower prevention threshold (for example, a block rate as low as 50 percent) for reconnaissance and information-gathering attempts. For scenarios such as this, the security practitioner should focus on products that have a higher level of prevention against web application-centric exploits, and should select a higher number of exploits and malware that target web applications during proof of concept (POC) testing. Select appropriate test tools When dealing with live exploits and other malware, ensure that the test environment is completely isolated from the live network. This can be achieved by utilizing software-based test tools and virtualization technology to create an isolated test bed in a single physical machine (see Figure 1). Using a virtualized environment (such as VMware s ESX Server), it is possible to create multiple vulnerable hosts and attackers. There are a number of readily-available software-based security effectiveness testing tools, such as White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 4

5 Core Impact, Metasploit, and Traffic IQ, which can be used to launch canned or custom attacks against vulnerable hosts protected by the security software under test. However, the preferred option should be to launch live attacks while working in a virtual environment. NSS Cyber Advanced Warning System (CAWS) provides unparalleled access to live attacks that are currently utilized in the wild. CAWS provides this via a simple, easy-to-use REST API. The CAWS API provides access to traffic files in SAZ and PCAP format, which makes it easy to replay attack traffic via HTTPREPLAY or TCPREPLAY. The CAWS API also provides information on the attack surface that is associated with specific traffic. With this information, attack traffic targeting Internet Explorer can only be replayed against corporate devices that use this browser. For more information on how to consume data via the API, please refer to the CAWS API Guide. Virtual Attackers ESX Server Virtual Victim External Switch Internal Switch Figure 1 Basic Test Environment A key feature of virtualization technology for the security testing professional is the ability to reset compromised or infected virtual machines quickly and easily to their default state after each test. This should be done whether the attempted exploit was successful or not, since there is no way to know the state in which the target hosts were left following the attempt. The capability to reset virtual machines to their default state can also be used to ensure that the test environment is reset to a clean state before each new AEP product is installed, enforcing a level playing field for all products under test. Several test tools can be utilized to determine whether an attack has been successful, including: Microsoft Windows Sysinternals Process Monitor is particularly helpful for reviewing process, file, registry, and network activity. Data from the CAWS API can be used as a control set to establish whether or not the same activity occurred on the virtual machine with the AEP product installed. If the same activity did occur, then the attack was successful. Wireshark, HTTPREPLAY WINAPI Monitor Where cost restrictions make it impractical to create an in-house test network, it is still important to evaluate potential purchases either by using external testing resources (NSS subscribers can discuss the possibility of using the NSS test lab to perform competitive tests) or by installing test products in a live network. In the latter case, White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 5

6 however, it should be recognized that the type of testing available would be more restricted in order to prevent an adverse impact on business processes. Configure the Test Tools After you have identified which use case will be evaluated, your next step is to configure the test tools and the test environment. Please refer to NSS Advanced Endpoint Protection Enterprise Self-Test Methodology 2 to determine which test cases should be run. A socially engineered malware sample should be placed into a local web server so that it can be downloaded onto the client machine with the AEP product installed. The web server can be accessed through the LAMP/WAMP suite. Before introducing the sample, have Process Monitor running on the client machine with the appropriate filters installed, so you are not overwhelmed with information. Live exploits can also be hosted internally to deliver attack traffic. Live exploits that are sourced from CAWS should be placed into a machine where HTTPREPLAY is installed. Once the exploit is loaded into HTTPREPLAY, navigate to the URL/file that matches that specific exploit. At the very least, the VM under test should have the correct combination of plug-ins, operating system, and browser. Today, outdated plug-ins are by default blocked by browsers, so it is important to ensure that these plug-ins are enabled so that you will can successfully simulate attacks. Ensure that the firewall configurations on the endpoint under test are enabled to allow HTTPREPLAY traffic. Configure the AEP Product The first pass of testing should always be performed with the vendor s default or recommended policy applied to determine its efficacy in your environment. Does the policy include coverage for all of the applications you see on your network? The chances are that it will not and that tuning will be required. It is essential that you run false positives to ensure that the AEP product under test doesn t block the legitimate business application in use. If any legitimate business application that is used by an organization is blocked, then the policies, rules, and signatures that are being triggered should be tuned to ensure that business is not affected. In cases where tuning is not possible, such policies, rules, and signatures should be disabled. Configuration of the software under test should always be performed in the first instance with the aid of the vendor s engineers. Make them aware of your use case and test criteria and ensure that their configuration precisely reflects your needs. Run the Test At least part of the evaluation period for an AEP product should be in full blocking mode. Do not make a major purchasing decision based on an evaluation where the AEP product was in detect-only mode throughout the test. 2 Advanced Endpoint Protection Enterprise Self-Test Methodology White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 6

7 False positives You have a distinct advantage over independent test labs in the area of false positive testing. Performed correctly, these tests will discover the propensity for an AEP product to raise alerts and block applications that are a normal part of your environment but that have been mistakenly classified as malicious by the AEP product under test. The primary goal of false positive testing is to simulate normal user behavior on the endpoint, and testing can include a varied sample of legitimate live application traffic, different file formats, different URLs, and IP addresses. The AEP product should be able to correctly identify the benign samples and permit them to execute without flagging them as threats. Ultimately, the sanitized lab environment used for third-party tests is likely to be very different from your live network. Applications and application traffic patterns in your environment might be such that AEP products that receive almost perfect scores in lab-based false positive tests will constantly raise false alarms when placed in your own network. Make sure the test suite includes a significant number of applications that are representative of those within your own environment. An essential part of any false positive test is to deploy the AEP product in your own network in passive mode and then monitor alerts that are generated from legitimate traffic. Evaluate the Results Unfortunately, some vendors spend much of their time attempting to produce signatures that cover every single POC exploit in a commercial vulnerability research library or test tool. This is at the expense of first trying to understand the vulnerability research behind them and then writing effective vulnerability-based signatures. This is why you can occasionally find significant differences between in-house, vendor-created tests (which are tested against the basic POCs) and well-run, independent third-party tests (which use real-world, weaponized exploits). Having hundreds of signatures that perform some basic banner grabbing (looking for standard text such as copyright messages in POC code, for example) or searching for fixed patterns (such as NOP sleds) in POC code will not help to protect against a well-crafted exploit. While some vendors will claim that their in-house POC testing is adequate and that independent test labs are being too rigorous in the creation and use of multiple weaponized exploit variants, you need ask yourself just one question: which type of exploit will the average cybercriminal use against my defenses? Sometimes, vendors will write accurate signatures but make them too restrictive in the otherwise laudable quest to avoid false positives. For example, some vendors may write signatures to look for specific malware drops. These signatures are valid as an additional layer of protection, but if the same vulnerability is exploited to allow direct remote control of the target host (i.e., allow shell access), it will not be detected. The biggest problem is that customers are rarely aware that they are exposed by sloppy signature writing, since a serious compromise requires that perfect storm of a well-written exploit executed against a vulnerable asset, which is protected by a poorly written signature. And, of course, if the signature never fired, then the event was never logged, and it could be quite some time before the compromise is discovered by other means. Thus, it is important to verify breadth and accuracy of coverage before purchasing and deploying a complex security product. No vendor should ever achieve 100 percent coverage in independent tests (unless vendor engineers are provided with the exploits in advance and allowed to tune device signatures accordingly). However, vendors that are worthy of further consideration are those with results that do not demonstrate a significant difference between their own testing and independent testing. White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 7

8 For more information on evaluating test results, please refer to NSS Advanced Endpoint Protection Group Test reports (AEP Test Methodology v1.0). 3 Scrutinize Vendor Track Record No matter how promising an advanced endpoint product appears during testing, it should be recognized that vendors are never so responsive as they are during the sales cycle. Prospective purchasers should read NSS research and speak to NSS analysts and vendor customers (current and previous, if possible) to determine key points regarding a vendor s track record: Updates Ensure that the vendors you have shortlisted have adequate resources committed to monitoring and responding to current threats. If a threat gains notoriety, then the security vendor must be able to respond quickly to offer a rapid work-around, which would then be followed by a targeted update/patch. Where signature/filter updates are part of the service, ensure that these occur frequently. Investigate recent updates to ensure that the most serious security alerts have been covered effectively. Accuracy Perform research to determine a vendor's track record with false positives. How often do signatures have to be revised after initial release? How easy is it to evade the signatures with small changes in code or basic evasion techniques? Third-party test reports can be used for much of this data. Do not confuse claims with coverage It is easy for vendors to dispute the results of independent tests including those that your organization performs by claiming that mistakes have been made in testing or that exploits used are not valid. They will point to results that highlight a lack of coverage for vulnerability X by showing you their signatures for exploit X, and it is easy to be swayed. However, if you have used a live exploit against a vulnerable host protected by an advanced endpoint product, and you have been granted a shell or root/administrator access on that vulnerable host, then the product has failed. Any vendor that argues against this demonstrates a serious lack of understanding. Carefully review awards and validation reports Study the claims of independent validation reports and awards carefully. If all a vendor can show you is a Microsoft PowerPoint slide full of magazine awards, this indicates a lack of understanding of the need for good testing, or a lack of confidence in the product. Make sure the methodologies used in independent reports are sound, and make sure that the testing house has a pedigree in security testing specifically. 3 NSS Labs Advanced Endpoint Protection Group Test reports (Test Methodology v1.0) White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 8

9 Implement Continuous Testing Initiatives Do not restrict testing to the purchasing cycle alone it should be an integral part of your ongoing security maintenance regime. Following initial deployment, perform a full benchmark test to acquire a baseline. Then, each time you apply a new firmware update, signature pack update, or change in security policy (however minor), the AEP product should be retested and the results compared against the baseline. This way, it is possible to monitor and correct detrimental changes in performance or security effectiveness. White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 9

10 Appendix A WARNING: Do not proceed if you are not familiar with security product testing or the implications of using live malware and exploits. Exploit Testing Procedure When dealing with live exploits, ensure that the test environment is completely isolated from the live network. This can be achieved by utilizing software-based test tools and virtualization technology to create an isolated test bed in a single physical machine (see Figure 1). Prerequisites 1. Make sure your test environment is isolated during POC testing. 2. Make sure you understand the implications of using live malware and exploits 3. You must have access to the NSS Labs CAWS portal. Building a VM harness for exploit testing The virtual machine (VM) environment is created to evaluate the efficacy of the product under the test. Wherever possible, it should closely mimic the production environment while remaining isolated from it. This can be done by isolating the network configuration from the rest of the production network as well as by making non-persistent images or snapshots of the VM images. The nonpersistent disk feature is widely available within the VMware environment. This feature ensures that any changes made to the disk by malware are discarded once the VM is powered off. This feature is used if the snapshot feature of the hypervisor is not being utilized. Another way to ensure that changes are not written to the disk permanently is with the snapshot feature. Once the Windows OS is fully configured in the VM environment with the appropriate tools for this analysis, take a snapshot to ensure you can revert to the clean state of the VM. The best way to isolate the network configuration is to use a host only or custom network configuration. Remove the physical device s virtual interface from these network segments to better isolate the environment. Step-by-step testing procedure 1. Log into the CAWS API to download the latest exploits. 2. Please follow the CAWS API guide: 3. The API returns an API date/time stamp for the exploit identified, the initial infected URL, and the platform that is being exploited. 4. Download the.saz file from the CAWS API. 5. Client environment setup: Configure multiple client machines so that one can act as a proxy server to deliver the exploit and the others can act as the victim machines. 6. The Proxy server machine must be running on Windows Rename the.saz file as a.zip file and extract the folder named RAW from the.zip file. Place the RAW file into the proxy server machine. 8. Download the HTTPREPLAY application. 9. Install HTTPREPLAY in the proxy server. White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 10

11 10. Navigate to the install path of HTTPREPLAY, and type HTTPREPLAY RAW into the command line. The RAW is the folder we extracted from the.saz file from the CAWS API. 11. A report will be generated on the proxy machine that shows the http request and response associated with the RAW file. 12. You are now ready to serve the exploit. o At the very least, the VM under test should have the correct combination of plug-ins, operating system, and browser. Today, outdated plug-ins are by default blocked by browsers, so it is important to ensure that these plug-ins are enabled so that you will can successfully simulate attacks. o Ensure that the firewall configurations on the endpoint under test are enabled to allow HTTPREPLAY traffic 13. Configure the victim environment as per the target identified by the CAWS API. 14. Download the Sysinternals tool set from and start the tool that you want to use. NSS recommends using Process Monitor and Process Explorer and applying the browser process as the filter in Process Monitor to ensure that large amounts of information can be filtered. 15. Navigate to the URL listed as part of the CAWS API for the.saz traffic. 16. Typically, if an exploit succeeds, a payload will drop on the victim machine. This payload can easily be identified using Process Monitor and Process Explorer. If the Hash of the payload matches up with the hash listed in CAWS API, this indicates the exploit has succeeded. 17. Repeat steps 2, 3, 4, 10, 13, 15, and 16 for every exploit you want to test. White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 11

12 Appendix B WARNING: Do not proceed if you are not familiar with security product testing or the implications of using live malware and exploits. Malware Testing Procedure When dealing with live malware, ensure that the test environment is completely isolated from the live network. This can be achieved by utilizing software-based test tools and virtualization technology to create an isolated test bed in a single physical machine (see Figure 1). Prerequisites 1. Make sure the environment is isolated during POC testing. 2. Make sure you understand the implications of using live malware and exploits 3. You must have a vendor SFTP account set up with NSS Labs. Building a VM harness for malware testing The virtual machine (VM) environment is created to evaluate the efficacy of the product under the test. Wherever possible, it should closely mimic the production environment while remaining isolated from it. This can be done by isolating the network configuration from the rest of the production network as well as by making non-persistent images or snapshots of the VM images. The nonpersistent disk feature is widely available within the VMware environment. This feature ensures that any changes made to the disk by malware are discarded once the VM is powered off. This feature is used if the snapshot feature of the hypervisor is not being utilized. Another way to ensure that changes are not written to the disk permanently is with the snapshot feature. Once the Windows OS is fully configured in the VM environment with the appropriate tools for this analysis, take a snapshot to ensure you can revert to the clean state of the VM. The best way to isolate the network configuration is to use a host only or custom network configuration. Remove the physical device s virtual interface from these network segments to better isolate the environment. Step-by-step testing procedure 1. Log into sftp.nsslabs.com using the credentials you received when you purchased the Advanced Endpoint Protection Enterprise Product Validation Kit. 2. You will find multiple folders in the SFTP location, each with a date stamp. 3. Log into the folder with the earliest date. Note: It is important to select the folder with the earliest date to ensure you get the latest/fresh malware samples. 4. Prepare a Windows machine with your security product installed and create a snapshot in VM environment. 5. Start downloading the files from the folder with the earliest date. 6. If the files are detected by the AEP product, this typically means that the file has been identified as a threat. If the files are not detected by the AEP product, try to execute them. If you can execute the files, then you know that the AEP product has not blocked the threat. If you cannot execute the files, then you know that the AEP product has blocked the threat. 7. Reset the VM to the baseline snapshot created in step Repeat steps 3, 5, 6, and 7 until you are done. White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 12

13 Contact Information NSS Labs, Inc. 206 Wild Basin Rd Building A, Suite 200 Austin, TX USA This report was produced as part of NSS Labs independent testing information services. Leading products were tested at no cost to the vendor, and NSS Labs received no vendor funding to produce this report NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a retrieval system, ed or otherwise disseminated or transmitted without the express written consent of NSS Labs, Inc. ( us or we ). Please read the disclaimer in this box because it contains important information that binds you. If you do not agree to these conditions, you should not read the rest of this report but should instead return the report immediately to us. You or your means the person who accesses this report and any entity on whose behalf he/she has obtained this report. 1. The information in this report is subject to change by us without notice, and we disclaim any obligation to update it. 2. The information in this report is believed by us to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at your sole risk. We are not liable or responsible for any damages, losses, or expenses of any nature whatsoever arising from any error or omission in this report. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND EXCLUDED BY US. IN NO EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY, OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and/or software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet your expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners. White Paper_Advanced Endpoint Protection Testing for the Enterprise TBD_XXXX17 13