SKD Labs Test Report. A Comparative Test on Anti-Malware Products in the China Region

Transcription

1 Test Report 2013 No.3 A Comparative Test on Anti-Malware Products in the China Region

2 SKD Labs is a leading independent testing facility in the China market and specializes in functionality testing, performance validation and certification of information security products and services. It develops and implements the highest testing standards to ensure maximum product effectiveness in functionality and performance. As West Coast Labs (China), SKD Labs independent testing and technical consultancy services benefit information security vendors to gain market advantage and to compete with established brands not only in China but also throughout markets worldwide. Through its conferences, seminars, and information publishing activities, SKD Labs develops awareness among users of security products both consumers and business users of the threats to information security and the products available to deal with those threats. Its Starcheck certification has been widely recognized by networking and information security industry, business and home users in the China market. CONTACT Jesse Song Operations Director jsong@skdlabs.com SKD Labs Suite A1822, Floor 18, Tower A, Top Electronic City No. 3 Haidian Ave., Haidian District Beijing, China Post Code:

3 Starcheck Certification Logo

4 Table of Contents 1 Introduction Significance of the Test Products Tested Test Environment Testing Vector/Metrics and Scoring System Testing Results On-demand On Access (Dynamic) Cleaning & Disinfection Malicious and Phishing Website Blocking Defense and Protection Capability False Positive Scan Time Message Control & Quietness Browse Pictures, etc Copy Files Install Other Programs Log On/Log Off Open Websites Run Games Run Multimedia Run Office Programs Run Other Programs Installation/Uninstallation Overall User Experience Evaluation Overall Evaluations: Quality of Protection and User Experience Baseline Concept Presentations Green line in the graph represents Microsoft baseline On-demand On-access (Dynamic) Cleaning and Disinfection Malicious and Phishing Website Blocking Defense and Protection Capability User Experience Total Evaluation: Protection Capability and User Experience Selecting the Most Suitable AV Product Several Notes... 33

5 1 Introduction Research has shown that the spread and effectiveness of malware have become more and more dependent on language and cultural factors. Regionalization of malware ecosystem has become a trend. The need for regionalized testing to accommodate the unique needs of specific geographies has been growing. SKD Labs West Coast Labs (China) recently carried out a comprehensive test on anti-malware products (Chinese edition) in the China environment. A Comparative Test on Anti-Malware Products in the China Region Page 1

6 2 Significance of the Test Evaluate performance of anti-malware products (Chinese edition) in the real China environment. This is the first such test that has been carried out. Provide information for China Internet users to choose the most suitable product. It is estimated that there are around 700 million of Internet users in China. Protect them from malware will be beneficial for the global Internet security. A Comparative Test on Anti-Malware Products in the China Region Page 2

7 3 Products Tested Instead of relying on vendors submitting test versions of their products, we downloaded the real versions from their websites. All products tested are Chinese editions. Vendor Product (Chinese Edition) MD5 avast Avast! Internet Security AFC1B7D001D0DC583F426F8B7B0D7D AVG AVG Antivirus Free A1B7B9A0B51960C7DAEDB Avira Avira Internet Security 2013 A945D3BBD8E058C5B7D749127AC2EBFF Bitdefender Bitdefender Internet Security 2012 F75A47AF CF9811D5F629D740 ESET ESET NOD32 Antivirus A3B5F6C40E54E770EFB4E564AB94F9 JIANGMIN JIANGMIN SPEED ESSENTIAL Antivirus Software 00AC8D CE2B48FA93D0514FD5 Kaspersky Kaspersky Internet Security 2013 F0FC DA300008DB991FC5A Kingsoft New Kingsoft Antivirus 2013 SP3.5 0C84CD39DA42506B5769D2FEA37B0748 McAfee McAfee Internet Security2013 6AC4F69003CFC9C11C3BECE65BA5EF9E Microsoft Microsoft Security Essentials F5A623A12FE9F5C7E930E6F8E17B4A1B Panda Panda Internet Security C539B403870FC3ABC1229FF7 Qihoo Qihoo 360 Antivirus DC9C85E4A B59D14FF8D998 Rising Rising Antivirus V16 A34F77A2AB67C7EDC332C53C47BC87E1 Symantec Symantec Norton Internet Security FE47AAF337BB56D1A94B553AF97DA Tencent Tencent PC Manager 8.0 BD F85BBC4E8D318F13A3F21E Trend Micro PC-cillin 2013 cloud security software 3F8B4F14BE0AFBC49FB93D87EA699B4E TWISTER full version enhanced version 6.0 TWISTER Antivirus E9D64F25CB7B6CF835E45B All the products were updated before testing. Default settings were used. A Comparative Test on Anti-Malware Products in the China Region Page 3

8 4 Test Environment The test was conducted under Chinese Windows 7 64 bit. Besides applications that come with Windows 7, more than 15 popular Chinese applications were installed to simulate a real Chinese end user computer environment. Applications Version MD5 Winrar D65455DA57E0199A8CE16 谷歌浏览器 m 8B65EDE6F97087A DF797FB1B 暴风影音 C74E466F54A C92691B1B69D1 千千静听 D BDCDE653FE36226FD15 QQ E43B20D37F3EC487F9518EA3D 迅雷 F30B8FAF12C4D01EFC4098CB45A7 搜狗输入法 6.5 AD72ADDAFF2DBB56AA4FED070BE32FF5 Windows 优化大师 D6E58338DB85BDF7F3CDD062EA22 有道词典 5.4 C7D8DF9581AB4BFABF491B84EBDFB9CA Adobe Reader A7A5F9DA27C00D455DF175E3179FB15 Foxmail 7.0 9D B755B4DD23FBC36FCC63B2 WPS A ABE1A809EADBB6FCCDEF9D6 91 手机助手 F868694D0F4CF021835A267923F45398 飞信 F1010D6F915FA2251E05970A99ED Adobe Flash Player DD9C388BBAEF4731C87E E3 A Comparative Test on Anti-Malware Products in the China Region Page 4

9 5 Testing Vector/Metrics and Scoring System This comprehensive test consists of seventeen sub-tests. Each sub-test is assigned points that are used in calculating the final score for each product tested. Test Name Points On-demand 15 On-access (Dynamic) 15 Cleaning & Disinfection 10 Malicious and Phishing Website Blocking 10 User Experience False Alert/Positive 5 Scan Time 1 Message Control & Quietness 1 Browse Pic,etc. 1 Copy Files 1 Install Other Programs 1 Log On/Log Off 1 Open Websites 1 Run Games 1 Run Multimedia 1 Run Office Programs 1 Run Other Programs 1 Installation/uninstallation 1 The highest score for a product is 67. A Comparative Test on Anti-Malware Products in the China Region Page 5

10 6 Testing Results 6.1 On-demand Test collection consists of 12,000 samples which were considered the most prevalent at the time of the test. Based on prevalence, they were divided into five groups. A weight was added to each group based on sample prevalence. The score was calculated based on the formula: 15-5/2400 G1 4/2400 G2 3/2400 G3 2/2400 G4-1/2400 G5 Gx represents the number of missed samples in the group x. The highest score for this sub-test is 15 points. Products Score Detection rate (raw data) Kaspersky % Tencent % Bitdefender % AVG % Rising % Qihoo % avast % Kingsoft % Symantec % McAfee % A Comparative Test on Anti-Malware Products in the China Region Page 6

11 Trend Micro % Microsoft % Panda % JIANGMIN % ESET % TWISTER % Avira % It can be seen that the weight does affect the ranking that is based on detection rate. 6.2 On Access (Dynamic) On-access/dynamic test was conducted through downloading, file copying, and folder viewing. In this test, 12,000 samples were used. They were divided into five groups. A weight was added to each group based on sample prevalence. The score was calculated based on the formula: 15-5/2400 G1 4/2400 G2 3/2400 G3 2/2400 G4-1/2400 G5 Gx represents the number of missed samples in the group x. The highest score for this sub-test is 15 points. Products Score Detection Rate (Raw Data) Kaspersky % Tencent % avira % A Comparative Test on Anti-Malware Products in the China Region Page 7

12 Bitdefender % AVG % avast % JIANGMIN % Rising % Qihoo % Trend Micro % Microsoft % Symantec % Kingsoft % TWISTER % Panda % ESET % 6.3 Cleaning & Disinfection This sub-test is to test AV solutions capability in repairing infected files. In this test, 100 infected samples were used. The virus families include Alman, dundun, hidfold, induce, sality, virut, parite, etc. AV solutions are required to identify and repair those samples. Simply deleting will not be counted for points. After repairing, all the files must be able to run normally. 0.1 point will be awarded for each sample repaired successfully. The highest score for this sub-test is 10 points. A Comparative Test on Anti-Malware Products in the China Region Page 8

13 Products Cleaning & disinfection Kaspersky 7.9 Microsoft 7.1 McAfee 7.1 Bitdefender 6.2 Symantec 6.1 Trend Micro 6.1 Panda 5.9 Qihoo 5.7 Rising 5.3 Tencent 5.3 JIANGMIN 5.2 Avira 4.9 ESET 4.5 Kingsoft 3.8 AVG 3.5 avast 2.1 TWISTER Malicious and Phishing Website Blocking According to a survey, more than 80% of Chinese Internet users reported that they had experiences of being attacked when visiting a malicious website. This sub-test is to test AV solutions capability in responding and blocking newly-found malicious and phishing websites. Blocking downloaded files is not included in this sub-test. It is tested under on-access/dynamic test. 500 URLs were selected from our newly-found malicious & phishing website collection. These website samples were in Chinese and are specifically targeting A Comparative Test on Anti-Malware Products in the China Region Page 9

14 Chinese users. The highest score for this sub-test is 10 points. The formula used to calculating score is: 10 (Number of blocked URLs / (500 Invalid URLs)) Invalid URLs were those URLs that were not accessible during testing. Products M & P website blocking Tencent 8.42 Qihoo 7.32 Microsoft 5.80 Symantec 3.86 avast 3.74 Trend Micro 3.23 Kingsoft 2.20 Panda 1.99 McAfee 0.66 AVG 0.46 Kaspersky 0.45 Rising 0.32 ESET 0.21 Bitdefender 0.11 Avira 0.04 JIANGMIN 0.00 TWISTER 0.00 A Comparative Test on Anti-Malware Products in the China Region Page 10

15 The results show that Chinese vendors are doing much better than international vendors. 6.5 Defense and Protection Capability Defense and protection capability includes on-demand, on-access, cleaning/disinfection, and website blocking. Products On-demand On-access Cleaning & disinfection Website blocking Total Tencent Qihoo Kaspersky Microsoft Symantec Trend Micro Bitdefender McAfee avast Rising AVG Kingsoft Panda JIANGMIN Avira ESET TWISTER A Comparative Test on Anti-Malware Products in the China Region Page 11

16 6.6 False Positive The highest score for this sub-test is 5 points. One false positive will cause reduction of one point. We created a clean system environment. Besides Chinese Windows 7 64 bit operating system along with its standard applications, more than 15 popular Chinese applications were installed. There was no malware item installed on the system. AVG and McAfee are the only two products that produced false positives. Each of them generated two false alerts. Products False Positive Score avast 5 Avira 5 Bitdefender 5 ESET 5 JIANGMIN 5 Kaspersky 5 Kingsoft 5 Microsoft 5 Panda 5 Qihoo 5 Rising 5 Symantec 5 Tencent 5 Trend Micro 5 A Comparative Test on Anti-Malware Products in the China Region Page 12

17 TWISTER 5 AVG 3 McAfee Scan Time On-demand folder scanning was used. The highest score for this sub-test is 1 point. Products Scan Time(min) Qihoo 6 Tencent 7 Avira 11 Kingsoft 14 TWISTER 18 avast 18 JIANGMIN 21 AVG 25 Symantec 30 Rising 31 McAfee 45 Kaspersky 50 Bitdefender 60 Panda 67 ESET 100 Microsoft 235 Trend Micro >300 A Comparative Test on Anti-Malware Products in the China Region Page 13

18 6.8 Message Control & Quietness This sub-test is to check whether popup or other message displaying is controllable or selectable. The highest score for this sub-test is 1 point. Products Message control & quietness Score Rising 1 avast 1 AVG 1 Bitdefender 1 ESET 1 Kaspersky 1 Panda 1 Symantec 1 Trend Micro 1 Qihoo 0 Tencent 0 Kingsoft 0 Microsoft 0 JIANGMIN 0 TWISTER 0 avira 0 McAfee 0 A Comparative Test on Anti-Malware Products in the China Region Page 14

19 6.9 Browse Pictures, etc. This sub-test is to measure AV solutions effects on browsing pictures, photos, et al. The highest score is 1 point. Products Browse pic. (ms) Qihoo 2670 Tencent 2996 JIANGMIN 3126 Microsoft 3528 Rising 4026 avast 4102 AVG 5204 ESET 5227 Panda 5672 Kingsoft 6196 Symantec 6508 avira 7173 Kaspersky 7761 McAfee Bitdefender Trend Micro TWISTER Copy Files This sub-test is to measure AV solutions effect on file copying. The highest score is 1 point. A Comparative Test on Anti-Malware Products in the China Region Page 15

20 Products Copy files(ms) TWISTER JIANGMIN avira Kingsoft Qihoo avast Tencent ESET Microsoft McAfee Trend Micro Panda Bitdefender AVG Kaspersky Symantec Rising Install Other Programs This sub-test is to measure AV solutions effect on installing application programs. The highest score is 1 point. Products Install other programs(ms) Rising Panda Tencent avira A Comparative Test on Anti-Malware Products in the China Region Page 16

21 Kaspersky Qihoo AVG ESET Microsoft JIANGMIN TWISTER Kingsoft Bitdefender avast McAfee Trend Micro Symantec Log On/Log Off This sub-test is to measure AV solutions effect on computer logon/logoff operations. The highest score for this sub-test is 1 point. Products Logon & Logoff(s) Tencent 174 McAfee 198 ESET 199 avast 202 JIANGMIN 204 Avira 204 Trend Micro 204 Qihoo 210 A Comparative Test on Anti-Malware Products in the China Region Page 17

22 Microsoft 213 Kaspersky 216 Kingsoft 218 Bitdefender 219 AVG 239 Rising 246 TWISTER 249 Symantec 255 Panda Open Websites This sub-test is to measure AV solutions effect on users visiting websites. The highest score is 1 point. Products Open websites(ms) Tencent avast ESET Trend Micro Bitdefender Kingsoft Qihoo JIANGMIN avira AVG Kaspersky Rising McAfee A Comparative Test on Anti-Malware Products in the China Region Page 18

23 Microsoft TWISTER Panda Symantec Run Games This sub-test is to measure AV solutions effect on playing games on computer. The highest score is 1 point for this sub-test. Products Run games (ms) Kaspersky 9542 avast 9552 Kingsoft 9641 Tencent 9697 ESET 9826 Rising 9882 Panda JIANGMIN Qihoo Bitdefender avira AVG Microsoft McAfee TWISTER Symantec Trend Micro A Comparative Test on Anti-Malware Products in the China Region Page 19

24 6.15 Run Multimedia This sub-test is to measure AV solutions effect on playing movies, music, and other multimedia files. The highest score for this sub-test is 1 point. Products Run multimedia (ms) AVG avast Tencent avira Panda JIANGMIN Microsoft Kaspersky TWISTER Rising Trend Micro ESET Kingsoft Qihoo Bitdefender McAfee Symantec Run Office Programs This sub-test is to measure AV solutions effect on using office applications. The highest score for this sub-test is 1 point. A Comparative Test on Anti-Malware Products in the China Region Page 20

25 Products Run office programs(ms) AVG Microsoft ESET Tencent Rising Kingsoft avast JIANGMIN avira McAfee Kaspersky TWISTER Qihoo Symantec Trend Micro Bitdefender Panda Run Other Programs This sub-test is to measure AV solutions effect on running general applications. The highest score for this sub-test is 1 point. A Comparative Test on Anti-Malware Products in the China Region Page 21

26 Products Run other programs(ms) JIANGMIN 3180 avast 3191 Tencent 3215 AVG 3216 ESET 3246 Trend Micro 3248 Kingsoft 3251 Microsoft 3255 Bitdefender 3266 Rising 3272 avira 3285 Panda 3287 TWISTER 3327 Kaspersky 3418 McAfee 3419 Qihoo 3444 Symantec Installation/Uninstallation This sub-test is to check whether AV solutions can be installed and uninstalled properly. All the AV products tested performed well and each of them received 1 point that is the highest score for this sub-test. A Comparative Test on Anti-Malware Products in the China Region Page 22

27 6.19 Overall User Experience Evaluation Products User experience total score avast Rising ESET Tencent Kaspersky JIANGMIN Avira Kingsoft Panda Qihoo Bitdefender Microsoft AVG TWISTER Trend Micro Symantec McAfee 9.14 A Comparative Test on Anti-Malware Products in the China Region Page 23

28 7 Overall Evaluations: Quality of Protection and User Experience Products Total Score Tencent Qihoo Kaspersky Microsoft avast Bitdefender Rising Trend Micro Symantec McAfee Kingsoft AVG JIANGMIN Panda Avira ESET TWISTER A Comparative Test on Anti-Malware Products in the China Region Page 24

29 8 Baseline Concept Presentations Green line in the graph represents Microsoft baseline 8.1 On-demand 8.2 On-access (Dynamic) A Comparative Test on Anti-Malware Products in the China Region Page 25

30 8.3 Cleaning and Disinfection 8.4 Malicious and Phishing Website Blocking 8.5 Defense and Protection Capability Defense and protection capability includes on-demand, on-access, A Comparative Test on Anti-Malware Products in the China Region Page 26

31 cleaning/disinfection, and website blocking. 8.6 User Experience A Comparative Test on Anti-Malware Products in the China Region Page 27

32 8.7 Total Evaluation: Protection Capability and User Experience A Comparative Test on Anti-Malware Products in the China Region Page 28

33 9 Selecting the Most Suitable AV Product One of the objectives of the test is to provide information for China Internet users to choose the most suitable AV product. A suitable anti-malware solution not only provides the best protection but also produces the least negative effect on tasks that users carry out on the computer. The following presentation shows AV solutions performance on certain tasks along with their respective protection capability evaluations. Browse pic. Website visit Run games Run multimedia User experience Protection Qihoo Tencent Kaspersky AVG Avast Tencent Tencent avast avast avast Rising Qihoo JIANGMIN ESET Kingsoft Tencent ESET Kaspersky Microsoft Trend Micro Tencent Avira Tencent Microsoft Rising Bitdefender ESET Panda Kaspersky Symantec avast Kingsoft Rising JIANGMIN JIANGMIN Trend Micro AVG Qihoo Panda Microsoft Avira Bitdefender ESET JIANGMIN JIANGMIN Kaspersky Kingsoft McAfee Panda avira Qihoo TWISTER Panda avast Kingsoft AVG Bitdefender Rising Qihoo Rising Symantec Kaspersky Avira Trend Micro Bitdefender AVG Avira Rising AVG ESET Microsoft Kingsoft Kaspersky McAfee Microsoft Kingsoft AVG Panda McAfee Microsoft McAfee Qihoo TWISTER JIANGMIN Bitdefender TWISTER TWISTER Bitdefender Trend Micro Avira Trend Micro Panda Symantec McAfee Symantec ESET TWISTER Symantec Trend Micro Symantec McAfee TWISTER The following chart not only shows AV products performance in browsing pictures A Comparative Test on Anti-Malware Products in the China Region Page 29

34 and protection, but also shows the difference in performance among those products. For example, the chart shows that Tencent and Qihoo provide the best protection and produce the least effect on tasks of browsing pictures and photos. The following chart not only shows AV products performance in visiting websites and protection, but also shows the difference in performance among those products. For example, the chart shows that Tencent provides the best protection and produces the least effect on task of visiting websites. A Comparative Test on Anti-Malware Products in the China Region Page 30

35 The following chart not only shows AV products performance in playing movies & music and in protection, but also shows the difference in performance among those products. A Comparative Test on Anti-Malware Products in the China Region Page 31

36 The following chart simultaneously displays each product s performance in five categories: browsing pictures, visiting websites, running games, playing movies, and playing music along with its protection capability. The information will help end users select the most suitable product they need. A Comparative Test on Anti-Malware Products in the China Region Page 32

37 10 Several Notes Cloud technology. In general, cloud technology makes processing unknown files/samples more efficient. However, in our stress test, a large number of unknown files may jam the connections to cloud servers. Large size of files could not be processed as well. Online criminals may take advantage of these weaknesses. Phishing and malicious websites specially targeting at Chinese users. It should be noted that international vendors are doing poorly in this sub-test. Product interface. All of the international vendors rely on direct translation of original user interface. In most cases, this direct translation does not serve Chinese users well. Compatibility between AV solutions. In the test, we did discover that one AV solution could crash another if they were installed on the same PC. The incompatibility could also cause PC crashing. It is commonly believed that AV products will perform the same in different language operating system environments. Our test shows that AV solution performance may be different in different language environments. For example, one product experienced frozen problem in processing one sample in the Chinese environment, which did not show in the English operating system environment. A Comparative Test on Anti-Malware Products in the China Region Page 33

38 2013 SKD Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please note that access to or use of this document is conditional on the following: 1. The test included in this report was independently conducted by SKD Labs. SKD Labs is solely responsible for the contents of this report. 2. The information in this document is believed by SKD Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this document are at the reader s sole risk. SKD Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this document. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY SKD LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON- INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY SKD LABS. IN NO EVENT SHALL SKD LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This document does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader s expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This document does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this document are the trademarks, service marks, and trade names of their respective owners. A Comparative Test on Anti-Malware Products in the China Region Page 34