Anti-Virus Testing and AMTSO

Transcription

1 Anti-Virus Testing and AMTSO Patrik Runald, Security Response Manager Helsinki Kuala Lumpur San Jose Tokyo London Paris Milan Hong Kong Mumbay Warsaw Munich Copenhagen Brussels Oulu Utrecht Oslo Stockholm

2

3 3

4 Growth of malware Malware 700, , , , , , , , , , , , ,000 50, Data source: F-Secure 平成 20 年 3 月 27 日 4

5 5

6 6

7 Attack Response Attack Response ATTACK Malicious attachments RESPONSE Dropping EXEs and PIFs Drive-by-Downloads to exploit Windows vulnerabilities Attacking Web browser vulnerabilities Attacking browser plugins and Add-ons (think Quicktime, Flash, Acrobat) Automatic Windows Updates Browsers automatically prompt for new versions Nothing much yet 平成 20 年 3 月 27 日 7

8 Disclaimer Material and comments in this presentation doesn t necessarily represent the actions or methods used by F-Secure Corporation. 8

9 Problem with AV testing Doesn t always state testing methodology Requires deep knowledge about antivirus technologies Requires extensive collection or malware Doesn t test all parts of an antivirus solution Engines can be modified/tweaked to do well in tests Heavily biased toward static file scanning Testing same as 10 years ago 9

10 Testing today On-demand testing WildList/WildCore Large (zoo) Retrospective Test product after x amount of time On-access Barely happens, too difficult Reaction tests How long did it take for vendor x to detect sample y 10

11 WildList/WildCore A collection of malware samples in-the-wild WildCore is a set of replicated virus samples that represents the real threat to computer users When a virus is reported by two or more reporters it goes into the WildCore Easy to pass as the list is known 11

12 Zoo collections Large collection of samples Shouldn t contain trash but often does Can be tricked by setting paranoid settings/heuristics Could mean detection based on another vendor False positives is a problem but can be whitelisted 12

13 Retrospective Scan a sample collection (WildCore or Zoo) after x amount of time Define a date and freeze the updates at that time Check detection rates on new samples Doesn t take into account generics on new families Unless correlated with false positive testing will benefit high paranoid heuristics 13

14 On-access Every file would have to be executed. Enough said 14

15 Reaction testing How long does it take for a vendor to detect a specific sample Less interesting today as there are so many new samples Not only that, they are polymorphing 15

16 Bias toward file scanning Why? Because it s easy and fast Today's products are multi-layered (firewall, HIPS, buffer overflow, IDS etc) 16

17 Let s look at some tests 17

18 18

19 19

20 AV-Test March 2008 Vendor detected 3/2008 % detected A comprehensive review should not only concentrate on detection AVK (G Data) % WebWasher-GW % scores of the on-demand scanner, as this would give a user only a TrustPort % very AntiVir misleading (Avira) and limited view of the product's capabilities. 99.3% Avast! (Alwil) % Trend Micro % Sophos % When comparing the security of cars, we would not only focus on Ikarus % the safety Nod32 (Eset) belts, but also check the ABS system (anti-lock 97.8% braking Microsoft % system), one or more airbags, crush zones, the ESP (electronic BitDefender % stabilization Kaspersky program) as well as constructional changes 97.2% and many F-Secure % other features which make a car secure. F-Prot (Frisk) % escan % In order ZoneAlarm to get a more comprehensive impression of the 96.4% products, AVG % one should Norton (Symantec) not only look at this test, but also compare the 95.7% results of Panda % various tests and the products' performance over time and their McAfee % Rising on-going development % Norman % 20

21 AV-Comparatives Feb

22 AV-Comparatives Feb

23 VirusInfo 23

24 VirusTotal 24

25 So what can be done? 25

26 26

27 27

28 28

29 AMTSO ALWIL Software IBM 25 AV-Comparatives member organizations Kaspersky Lab AV-Test.org McAfee, Inc. Started Feb 4 AVG Technologies Microsoft Corp. Provide Avira GmbH a forum to discuss testing Norman ASA Develop Bit9 and publish standards for Panda testing Security incl. best practices BitDefender PC Tools Provide Doctor Web, tools and Ltd. resources Sana Security Provide ESET analysis and review current Secure testscomputing F-Secure Corporation Sophos Plc Next meeting April 30 G DATA Software Symantec Corporation Hispasec Sistemas Trend Micro Inc. Virusbuster Ltd. 29

30 Finally we re making changes 30

31 Patrik Runald Security Response Manager F-Secure Corporation 31