TEST METHODOLOGY. Advanced Endpoint Protection: Enterprise Self-Test Methodology

Transcription

1 TEST METHODOLOGY v0.9 DRAFT

2 Table f Cntents 1 Intrductin Abut This Test Methdlgy Security Effectiveness Perfrmance Deplyment Test Metrics Security Effectiveness False Psitive Testing Malware and Explits Perfrmance Resurce Overhead during Nrmal Operatins Times/Speeds during Transfer Operatins Ttal Cst f Ownership and Value... 9 Cntact Infrmatin _

3 1 Intrductin 1.1 Abut This Test Methdlgy The scpe f the Advanced Endpint Prtectin Enterprise Self-Test Methdlgy includes: Security effectiveness Perfrmance The bjectives f this methdlgy are t validate if advanced endpint prtectin (AEP) prducts being tested are able t: Prevent and detect against malware threats within an enterprise envirnment. Prvide visibility int threats fr the end user/enterprise. Based n the needs identified in NSS research, the fllwing are cnsidered typical AEP prduct deplyment requirements: Centralized deplyment and management f multiple AEP prducts Centralized enterprise-class threat reprting and management visibility int the install base Cnfigured t receive endpint client-based prduct and sftware updates at a minimum (stand-alne r pushed frm the endpint management server) NSS cnsiders the fllwing t be essential capabilities f an AEP prduct, and they will be evaluated during the AEP grup test: Inbund threat detectin and preventin (prir t executin) Detectin and preventin f threats n access and during executin f the threats Security Effectiveness NSS has created a unique testing infrastructure the NSS Labs Live Testing harness that incrprates multiple prduct cmbinatins, r stacks, within the attack chain. Each stack cnsists f either an OS alne, r f an OS with additinal applicatins installed (fr example, a brwser, Java, and Adbe Acrbat). These stacks make up the BaitNET test harness, which is als the primary engine fr the Cyber Advanced Warning System (CAWS). Access t this cntent allws an enterprise t determine the attack surface applicable fr any given cmbinatin f OS and applicatins installed n any particular hst. This test envirnment is nt a facsimile, nr is it a snapsht in time; the test harness cmprises real stacks that are cnnected t a live Internet feed 24x7x365. This sphisticated research and test infrastructure allws NSS t mnitr, track, and participate in actual cybercrime campaigns run by threat actrs. Fr further infrmatin, refer t the Cyber Advanced Warning System Technical White Paper. 1 This methdlgy utilizes real threats and attack methds that exist in the wild and that are currently being used by cybercriminals and ther threat actrs, based n attacks cllected frm NSS glbal threat intelligence netwrk. This mnitring infrastructure currently spans 37+ cuntries arund the glbe, with mre being added weekly. It 1 Cyber Advanced Warning System Technical White Paper, NSS Labs _

4 harvests live malware, explits, and blended threats frm the Internet t test the efficacy f security prducts in real time. Each prduct must be able t prevent and detect against threats while prviding end-t-end visibility thrugh event(s) lg generated by the endpint prduct. Security efficacy testing may be cnducted ver each f the fllwing infectin vectrs: HTTP: Web-based attacks where the user is deceived int clicking n a malicius link t dwnlad and execute malware, r where the user merely needs t visit a web page hsting malicius cde in rder t be infected via explits Inbund, -based attacks where the user is deceived int clicking n a malicius link t dwnlad and execute malware r where the user merely needs t visit a web page hsting malicius cde t be infected via explits. A user can als be deceived int dwnlading and executing malicius attachments Perfrmance It is imprtant t determine the AEP prduct s impact n the hst system perfrmance. This includes netwrk, CPU, and memry usage. In rder t capture the relatinship between prtectin and perfrmance, this methdlgy includes the fllwing test measurements Perfrmance f the AEP prduct during nrmal peratins Perfrmance f the AEP prduct during transfer peratins 1.2 Deplyment An AEP prduct shuld be deplyed as an agent n a Micrsft Windws endpint. The agent shuld be deplyed via a centralized deplyment and management platfrm that has the capability t deply multiple endpint clients simultaneusly. Centralized enterprise-threat class reprting and management visibility fr the install base shuld be supplied either via a virtual management server r via a clud-based ffering. The endpint client shuld be deplyed in the default preventin and/r prtectin mde with clud access. It must be cnfigured t receive endpint client-based prduct and sftware updates at a minimum (autmatic r pushed frm the server). 1.3 Test Metrics The AEP prduct will be evaluated fr security efficacy using ne r mre f the fllwing metrics: Overall security efficacy (prtectin); i.e., blck rate Overall respnse rate t threats Overall prtectin ver time _

5 2 Security Effectiveness The aim f this sectin is t verify that the AEP prduct is capable f preventing and detecting threats while als cntinuusly and accurately lgging them, bth prir t and during executin. At the same time, the prduct must remain resistant t false psitives. This test sectin utilizes real threats and attack methds that exist in the wild and are actually in use by cybercriminals and ther threat actrs, based n attacks cllected frm NSS glbal threat intelligence netwrk. Samples will be surced frm nging, fresh campaigns that are ccurring n the wild. The ultimate gal f any attack n a cmputer system is t gain access t a target hst and attempt t perfrm an unauthrized actin resulting in a cmprmise f an asset r data. Cmputer systems are designed with many levels f prtectin t prevent unauthrized access and grant authrized access. Hwever, intruders may use cmmn techniques t circumvent these levels f prtectin, such as targeting vulnerable services, invking backdr privilege escalatin, r replacing key perating system files. AEP prducts prtect against autmated and manual threats by leveraging the fllwing key capabilities: Inbund threat detectin and preventin (prir t executin) Detectin and preventin f threats n access and during executin f the threats Mst AEP prducts are designed t prtect laptp and desktp clients and client-based services/applicatins; therefre, they shuld be tested by attempting t cmprmise client applicatins r client-based services, such as but nt limited t, web brwsers (e.g., Micrsft Internet Explrer, Mzilla Firefx), clients (e.g., Micrsft Outlk, Mzilla Thunderbird, Ltus Ntes), desktp publishing and ffice prductivity tls (e.g., Adbe Acrbat, Micrsft Office), and media players (e.g., Windws Media Player, Apple QuickTime, Real Audi/Vide). This sectin verifies that the AEP prduct is capable f accurately preventing and detecting threats and acting n a wide range f cmmn and sphisticated malware while remaining resistant t false psitives. The actins perfrmed n malicius threats can be dne via any number f the fllwing: Threat blcking Threat preventin Threat islatin/quarantine Enterprises deply AEP prducts with the vendr s default plicy; therefre, it is imperative that such plicies are made generally available t the public at the time f testing. Prir t installing an AEP prduct, an enterprise must validate the baseline vulnerabilities f the hst systems as well as the number f successful attacks n each hst cnfiguratin. The target hst systems shuld be restred t a clean, uncmprmised state and shuld be installed and cnfigured based n best practices. The AEP prduct shuld be updated t ensure it is prviding the latest prtectin. 2.1 False Psitive Testing The ability f the AEP prduct t crrectly identify and allw benign traffic is as imprtant as its ability t prvide prtectin against malicius cntent. The AEP prduct accmplishes this by detecting, preventing, and cntinuusly mnitring threats while at the same time allwing nn-malicius traffic t pass. This test will include a varied sample f legitimate applicatin traffic, which shuld prperly be identified and allwed. Additinally, legitimate applicatin traffic shuld mimic that which is in the enterprise envirnment. After cmpletin f the false psitive test and prir t Security Effectiveness testing, an enterprise must validate that false psitive alerts _

6 are disabled n its AEP prducts, r it will be penalized. If an AEP prduct raises any false psitive alerts during testing, these shuld be scred against the vendr. 2.2 Malware and Explits The mst cmmn ways in which systems are cmprmised is thrugh scially engineered malware and via driveby dwnlads, als knwn as explits. In bth cases, the gal is t cmprmise the end user s machine and achieve the bjective f the threat actr. AEP prducts typically have the fllwing pprtunities t prtect assets frm threat actrs wh are utilizing either scial engineering r explits: Befre the malware r explit is dwnladed During the malware r explit s dwnlad r after it has been dwnladed In thery, reputatin systems are suppsed t warn users nt t dwnlad a threat; reputatin, signature, and pattern-matching engines are suppsed t blck a threat as it is dwnlading; and heuristics/behaviral engines, and virtual analysis are suppsed t prevent a threat frm installing. Als, advances in hardening technlgies at the perating system level, such as ASLR, DEP, and stack ckies, are suppsed t prevent an explit frm cmprmising ne applicatin and gaining cmplete cntrl f a system. Hwever, mst threats bypass these prtectins and are successfully installed. AEP prducts shuld pssess the ability t cntinuusly mnitr an endpint fr indicatrs f cmprmise. This methdlgy will fcus n scial engineering techniques that deceive users int dwnlading malicius files that are delivered via the fllwing infectin vectrs. (Refer t the Security Effectiveness sectin fr mre infrmatin.) HTTP Offline vectrs (Hsts infected utside the crprate netwrk and subsequently attached t the netwrk) Figure 1 Malware Infectin Vectrs A summary f these cnfiguratins can be fund in Figure 2. _

7 AEP Functinality Offline Mde Online Mde Reputatin NO NO Signature updates NO YES Clud-based sandbx analysis NO YES Figure 2 Operatinal Mdes _

8 3 Perfrmance Hst-based sftware can have a cnsiderable impact n the usability f an AEP prduct. In rder t capture the relatinship between prtectin and perfrmance, this methdlgy includes the fllwing test measurements. 3.1 Resurce Overhead during Nrmal Operatins The fllwing applicatins are mst cmmnly used by enterprises and their times t start (warm start) will be measured in the test (current and mst widely used versins are included). Each test is first perfrmed withut the AEP prduct installed n the system in rder t establish a baseline. The AEP prduct is then installed and the test is run again t determine the impact n perfrmance. Micrsft Outlk Micrsft Wrd The net increase in time t pen Wrd dcuments f varying file sizes is evaluated. File sizes used: 100 KB, 500 KB, 1 MB, 3 MB, 10 MB, 20 MB Micrsft Excel The net increase in time t pen Excel dcuments f varying file sizes is evaluated. File sizes used: 100 KB, 500 KB, 1 MB, 3 MB, 10 MB, 20 MB Micrsft PwerPint The net increase in time t pen Excel dcuments f varying file sizes is evaluated. File sizes used: 1 MB, 3 MB, 10 MB, 20 MB Adbe Reader Micrsft Internet Explrer Open t the default web page n the lcal system Mzilla Firefx Open t the default web page n the lcal system Ggle Chrme Open t the default web page n the lcal system 3.2 Times/Speeds during Transfer Operatins The AEP prduct will be evaluated fr nrmal peratinal transfer times and speeds fr varius executable files. The net increase in time t pen cpy varying file types and sizes is evaluated. File sizes used: 1 MB, 3 MB, 10 MB, 20 MB, 100 MB, 500 MB, 1 GB _

9 4 Ttal Cst f Ownership and Value Implementatin f security slutins can be cmplex, with several factrs affecting the verall cst f deplyment, maintenance, and upkeep. All f these shuld be cnsidered ver the curse f the useful life f the prduct. Prduct Purchase The cst f acquisitin Prduct Maintenance The fees paid t the vendr (including sftware, maintenance, and updates) Installatin The time required t cnfigure the security prduct, deply it in the netwrk, apply updates and patches, and set up desired lgging and reprting Upkeep The time required t apply peridic updates and patches frm vendrs, including sftware updates. _

10 Cntact Infrmatin NSS Labs, Inc. 206 Wild Basin Rd. Building A, Suite 200 Austin, TX USA This and ther related dcuments available at: T receive a licensed cpy r reprt misuse, please cntact NSS Labs NSS Labs, Inc. All rights reserved. N part f this publicatin may be reprduced, cpied/scanned, stred n a retrieval system, ed r therwise disseminated r transmitted withut the express written cnsent f NSS Labs, Inc. ( us r we ). Please read the disclaimer in this bx because it cntains imprtant infrmatin that binds yu. If yu d nt agree t these cnditins, yu shuld nt read the rest f this reprt but shuld instead return the reprt immediately t us. Yu r yur means the persn wh accesses this reprt and any entity n whse behalf he/she has btained this reprt. 1. The infrmatin in this reprt is subject t change by us withut ntice, and we disclaim any bligatin t update it. 2. The infrmatin in this reprt is believed by us t be accurate and reliable at the time f publicatin, but is nt guaranteed. All use f and reliance n this reprt are at yur sle risk. We are nt liable r respnsible fr any damages, lsses, r expenses f any nature whatsever arising frm any errr r missin in this reprt. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND EXCLUDED BY US. IN NO EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY, OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This reprt des nt cnstitute an endrsement, recmmendatin, r guarantee f any f the prducts (hardware r sftware) tested r the hardware and/r sftware used in testing the prducts. The testing des nt guarantee that there are n errrs r defects in the prducts r that the prducts will meet yur expectatins, requirements, needs, r specificatins, r that they will perate withut interruptin. 5. This reprt des nt imply any endrsement, spnsrship, affiliatin, r verificatin by r with any rganizatins mentined in this reprt. 6. All trademarks, service marks, and trade names used in this reprt are the trademarks, service marks, and trade names f their respective wners. _