Component Protection Metrics for Security Product Development: CheckVir Endpoint Test Battery

Transcription

1 Component Protection Metrics for Security Product Development: CheckVir Endpoint Test Battery Ferenc Leitold Veszprog Ltd. Kai Yu Trend Micro, China Anthony Arrott Trend Micro, USA Abstract The CheckVir Endpoint Test Battery was used to perform iterative private tests on pre-release builds of version 6 of the Trend Micro Titanium Maximum Security Windows endpoint product. The test battery includes separate measurements of the protection provided by the consumer endpoint security product including measurements of: blocking malicious URLs; blocking transfers of malware files; detecting and removing rootkits; blocking phishing URLs; blocking browser exploits; cleaning malware infections; blocking malware execution; and detecting and removing stored malware files. All tests were conducted as a private piggyback on regularly scheduled public tests of the new version s predecessor and currently released versions of competitor peer products. 1. Introduction The CheckVir Endpoint Test Battery was used by the Trend Micro Titanium version 6 (Ti6) development team to test pre-release versions of the consumer Windows endpoint internet security product. The tests were conducted as private piggyback tests on the regular public CheckVir Endpoint Test Battery that Veszprog conducts each month benchmarking most of the major internet security products for Windows endpoints. The public tests include the current release of Trend Micro s Titanium version 5 (Ti5), sold under the product name, Titanium Maximum Security The piggyback tests subjected pre-release versions of Ti6 to the same set of external independent benchmarking tests against competitor products using the same testing conditions and attack vectors that Ti6 would be tested in upon its public release. Individual constituent tests of the CheckVir Endpoint Test Battery were used to isolate the protection measurement of specific security engines and services that make up the multi-dimensional security solutions contained in an internet security product like Ti6. The separate component tests also provided direct quantitative comparison with the similar engines and services in competitor products. The primary goal of component testing is to eliminate difficulties encountered in using end-to-end or whole product testing with pre-release builds of a new product version. Without individual component tests, product developers and company threat researchers must undertake difficult analyses to determine which service or engine in the integrated product is responsible for instances of missed attacks in end-to-end and whole product tests. With component tests, missed attack samples can go directly to the engine or service team responsible for the protection component measured in the test. 2. Methods Testing procedures are executed automatically, semi-automatically or manually using a special frame system. This automatic system provides a database accessible on the Internet including the scanning results related to each version. This system includes the following parts and they works as follows in the case of testing under Windows (Figure 1). 1

2 computer is to distinguish between the network traffic of the virus protection and the malware. The traffic of the malware is forwarded to the "malware proxy" server however the network traffic of the virus protection is forwarded to the internet. Thus solutions may use internet connection, so the usage of "cloud technology" is not limited. Figure 1. Technical background Clients: These computers are able to execute different tasks related to different testing methods of the particular protection. Client computers have exactly the same hardware and software. (Even the cards are inserted into same slots.) Clients can execute testing procedures in virtual and in native environment as well. A debian Linux system and perl scripts are dealing with changing the image of the Windows operating system (including the protection) and execute them periodically. The Windows system includes installed scripts thus they can execute the steps of the corresponding testing procedure. For security reasons computer executing testing tasks are connected to the internet via a special firewall. Once the testing procedure executed all of information from the used image are saved and after the analysis of information the results are transferred directly to the Webserver computer. Webserver: It collects test results in its database and provides it accessible via its web page. Archiver: All of information about executed tests is archived by this computer. It includes test results, log files and images as well as data required for testing. Controller: This computer manages the whole process on different parts of the system. Firewall: There is a firewall between the inner (red) and outer (blue) networks. It is used for managing the system only, it is not required by the automatic working mechanism of the system. Firewall & router: There is a special firewall and router among the client computers, the "malware proxy" server and the internet. The main task of this "Malware proxy" server: This server can store the content of malicious sites related to the certain time. So it can simulate the tested part of the internet for clients. It is used in the case of dynamic testing (when the malware code is executed, or a malicious site is opened). This frame system can enable that the set of tested procedures can be increased by producing some new scripts with the related data. 3. Endpoint Test battery Testing procedures of Checkvir Endpoint Test Battery includes nine different testing procedures. Five of them are related to malicious files and four of them are related to malicious URLs: 1. a. In the case of Static on-demand testing the threat detection and removing capabilities of the protection are tested against malicious codes stored in local files. b. The Static on-access testing method tests the threat detection and removing capabilities of the protection during the copying procedure of malicious files. 2. The Dynamic execution testing method tests the threat detection and blocking capabilities of the protection during the execution procedure of a malicious file. 3. The 0hour testing method tests the threat detection and blocking capabilities of the protection during the opening of the recently received malicious URL site. This test is repeated several times to check protection changes. 4. The Remediation testing method tests the threat detection and restoring capabilities of the protection against an installed malware on a machine. 5. The Rootkit testing method tests the threat detection and restoring capabilities of the protection against an installed rootkit on a machine. 2

3 6. The Exploit testing method tests the threat detection and blocking capabilities of the protection against an exploit attack from another computer. A special Linux server with the metasploit software is built to provide exploit attacks to clients. 7. The Dynamic URL method tests the threat detection and blocking capabilities of the protection during the opening of a malicious URL site. 8. The Phishing testing method tests the threat detection and blocking capabilities of the protection during the opening of a phishing URL site. 4. Results Measures of protection by web reputation V-3 Blocking Zero-Hour Web Threats 8 Blocking Zero Hour Web Threats All of these nine testing procedures are evaluated using the corresponding user situation. In the case of malicious files user can - execute on-demand scan (1a), - copy the malicious file (1b), - execute the malicious file (2), - try to restore the infected system (4 and 5). In the case of malicious URLs the user can open the malicious site depending on its type (malicious executable: 3 and 7, exploit: 6, phishing: 8) and in the case of executables the user can try to execute that. The Trend Micro Ti6 development team made use of the following tests as part of their program of Ti6 pre-release testing and product modification cycle: Measures of protection by web reputation V-3 Blocking Zero-Hour Web Threats V-7a Blocking Malicious URLs V-8 Blocking Phishing URLs Measures of protection by malware file detection V-1a V-1b Detecting Stored Malware Files Blocking Malware File Transfers Measures of protection by behavioral monitoring V-2 Blocking Executed Malware Measures of protection from browser exploits V-6 Blocking Exploit Attacks Measures of protection by disinfection after detection V-4 Cleaning Malware Infections Measures of protection by detecting & removing rootkits V-5 Detecting & Removing Rootkits V-7a layer % 53% 64% 1 st 1 st Test V-3 Apr-Aug attack samples Blocking Malicious URLs by the exposure Blocking Malicious URLs 23% 69% 74% 1 st 1 st Test V-7a Apr-Aug attack samples ( ) 3

4 V-7b Blocking Malicious URLs by all layers Blocking Web Threats (any layer) 65% 7 84% 4 th 3 rd Test V-7b Apr-Aug attack samples ( ) V-8 Blocking Phishing URLs Measures of protection by malware file detection V-1a Detecting Stored Malware Files Detecting Stored Malware Files 89% 89% 96% 3 rd 1 st Test V-1a-D Apr-Aug ,634 attack samples 8 Blocking Phishing URLs Cleaning Stored Malware Files % 74% 2 nd 1 st % 88% 72% 2 nd 5 th Test V-8 Apr-Aug attack samples ( ) Test V-1a-C Apr-Aug ,634 attack samples 4

5 V-1b Blocking Malware File Transfers Blocking Malware File Transfers 8 Measures of protection from browser exploits V-6 Blocking Exploit Attacks Blocking Browser Exploit Attacks % 94% 84% 2 nd 4 th Test V-1b Apr-Aug ,303 attack samples Measures of protection by behavioral monitoring % 89% 4 th 1 st (tie) Test V-6 Jul-Aug attack samples V-2 Blocking Executed Malware Measures of protection by disinfection Blocking Executed Malware 2 nd 2 nd 11% 12% 14% V-4 Cleaning Malware Infections 8 4 Cleaning Malware Infections 81% 81% 8 Test V-2a Apr-Aug attack samples 2 4 th 4 th Test V-4 Apr-Aug attack samples ( ) 5

6 Measures of protection from rootkits V-5 Detecting & Removing Rootkits Detecting & Removing Rootkits % 8 83% 3 rd 3 rd Test V-5 Apr-Aug attack samples ( ) 5. Analysis Testing procedures were executed using Trend Micro Titanium 5 and the new beta 6 versions from April June testing results can be seen on Table 2. In the case of malicious file related tests, differences between the two tested Titanium versions were related to mainly the dynamic execution test only (2b: 28% and 4). In the case of malicious URL related tests Titanium 6 was better in 3 (0hour), 7 (dynamic URL) and 8 (phishing) tests, however Titanium 5 was better in the case of 6 (exploit) test, which was related to a bug in Titanium 6 already fixed in the next version. Test results of Checkvir Endpoint Test Battery include statistical information of all testing procedures and about summary as well. On the other hand all of test cases described in details. For example the 7 Dynamic URL testing cases can include details about the flow of the testing procedure. Figure 2. Detection and blocking by the infection layer In Figure 2 the testing flow can be seen when the url was not blocked, the browser offered the saving of the malicious file (first screenshot) and later the malicious file may not be saved (second screenshot). In Figure 3 the exposure layer blocked the opening procedure in browser it displayed a warning as well. 6

7 - That they would have had enough time to develop a patch in time for the final release; - That the Titanium product managers would have allowed a patch to the release candidate so late in the version development cycle. Blocking Browser Exploit Attacks successive Ti6 beta tests % 8 78% 7 65% th 5 th 3 rd 4 th 6 th 1 st 1 st (tie) (tie) avg* Ti 5 β1 β2 β3 β4 β5 GM Test V-6 Apr-Aug attack samples (May-Aug) McAfee, Symantec, & Webroot Figure 3. Detection and blocking by the exposure layer 6. Conclusion The CheckVir Endpoint Test Battery provided the Trend Micro Ti6 development team with valuable information on the protection ability of the interim alpha and beta builds. By using repeated component tests, the developers were able to pinpoint problems as they arose and respond during the product development cycle. An example of this is the Browser Engine at Trend Micro. They were able to use the Test V-6 (Blocking Browser Exploit Attacks) tests to see results from multiple iterations of their engine integrated into the Titanium product. As the final release approached, they noticed a drop off in Ti6 protection for Test V-6. A late patch to a release candidate build solved the problem, as can be seen in Figure 4. In the absence of iterations of the browser exploit test revealing the nature and seriousness of the problem, it is unlikely that: - The Browser Engine Team would have been aware of the problem; Figure 4. Iterations of test V-6 revealed a hidden problem with the Browser Engine that was fixed by a late patch as seen in the performance of Ti6 β 5 build. 7. References [1] Leitold, F. (1995). Automatic Virus Analyser System, Proceedings of the 5th International Virus Bulletin Conference, Boston USA, 1995, pp [2] Leitold, F. (2002). Independent AV testing, Proceedings of the 11th International EICAR Conference, Berlin, Germany, [3] Leitold, F.: Experience of AV testing, Proceedings of the 12th International EICAR Conference, Copenhagen, Denmark, 2003 [4] Kárpáti, N.; Leitold, F.: CheckVir anti-virus testing and certification, Proceedings of the 13th International EICAR Conference, Luxemburg, 2004 [5] Leitold, F.: The solution in the naming chaos, Proceedings of the 14th International EICAR Conference, Malta, 2005 [6] Leitold, F.: CheckVir Realtime Antimalware Testing and Certification, Proceedings of the 18th International EICAR Conference, Berlin, Germany,

8 [7] EICAR Cyber Attack Methods Detection & Information Exploitation Research Project, [8] Order To Come To Virus Naming Chaos, November 24, 2004 [9] S. Gordon; R. Ford: REAL WORLD ANTI-VIRUS PRODUCT REVIEWS AND EVALUATIONS THE CURRENT STATE OF AFFAIRS, nal.pdf [10] A. Marx: A Guideline to Anti-Malware-Software testing Proceedings of the 12th International EICAR Conference, Copenhagen, Denmark, 2003 [11] M. Morgenstern; A. Marx: Testing of "Dynamic Detection", AVAR Conference, 2007, &lang=0 [12] AMTSO documents, 8