Open Source IDS Rules Comparison Report July 2014

Transcription

1 Open Source IDS Rules Comparison Report July 2014 DOCUMENT DETAILS Author Created on Internal Reference Simon Wesseldine 15th July 2014 VERSION CONTROL Version Release Date Overview of Changes th July 2014 Initial Release th July 2014 Para. 1.3 text change st July 2014 Traffic file references added st July 2014 Spelling st July 2014 Graph Changes. DOCUMENT APPROVAL Name Position Date Ray Bryant CEO of idappcom 21st July 2014 DISTRIBUTION Name Position Company The text in this document may be reproduced (excluding logos) free of charge in any format or medium providing that it is reproduced accurately and not used in a misleading context. The material must be acknowledged as copyright to idappcom ltd. Copyright idappcom Limited. All rights reserved.

2 CONTENTS Title Page 1.0 Executive Summary Introduction Aim Test Overview Traffic Files Intrusion Detection Systems (IDS) Configuration Files Rule Sets Affective Rules Alerting Results 5 Appendixes A. Results Graphs A-1 B. Traffic File Lists B-1 Page i

3 1.0 Executive Summary 1.1 Overview On the 15th July 2014 idappcom ltd carried out monthly testing of the Open Source IDS rules sets available on the internet. Each rule set was tested with the last three month's of exploit traffic files from Traffic IQ Professional, totalling 342 traffic files. The aim of the test was to compare the detection capabilities of each rule set in various configurations and combinations. 1.2 Results The results show how many traffic files were correctly identified, how many were incorrectly identified and how many did not get detected at all Although the configuration of Emerging Threats and Sourcefire Vulnerability Research Team (VRT) rules loaded together was a noisy combination, it had the greatest success in detecting the exploit traffic This combination still only provided protection against 56% of the exploits replayed by Traffic IQ. Rule Sets vs Exploits Detected Key Observations. "Not one rule set could provide 100% detection against Traffic IQ files". "All rule sets and all configurations allowed more than 43% of Traffic IQ attack files through undetected". "Rule sets loaded with the default policies provide poor detection and protection capability against Traffic IQ attack files". Page 1

4 2.0 Introduction 2.1 Idappcom ltd was formed in Guernsey In 2009 operations moved to the UK. Since then idappcom have supplied software and hardware that enhances network security, reducing risk and lowering cost of ownership of devices. 2.2 Products are constantly evolving in response to the needs of idappcom customers. Idappcom have traditionally supplied security vendors and test labs and now supply new versions of their products specifically developed for the end user. Years of expertise and development knowledge are now available to the global market to assist their customers achieving the highest levels of network threat protection. 2.3 Idappcom's flagship application is Traffic IQ Professional, which is traffic replay software that is capable of replaying thousands of IP packet captures, safely and reliably through IT networks. Each session can be configured with any IP or MAC address and each packet shall be monitored for correct delivery and receipt. 2.4 As well as being able to replay regular pcaps, Traffic IQ is fully loaded with a library of exploit packet captures that can be used to test security devices on the network, such as IDS, IPS, Firewall, etc. Traffic IQ can report on what exploits were blocked and what exploits were allowed through security devices. 2.5 Every month the Traffic IQ library is updated with between 100 and 200 new traffic files that have been created from vulnerabilities that have been disclosed the months before. 2.6 Every month each new traffic file is tested against the latest versions of Open Source IDS Rules to see what is detected and to see what is not. This report describes the process and defines the variables used in the tests. 3.0 Aim 3.1 The aim of the tests is to compare the detection capabilities of the latest released Open Source IDS rules against the latest released Traffic IQ traffic files. 4.0 Test Overview 4.1 The physical testing environment is a closed network consisting of the following components: Host with Ubuntu LTS Operating System Host with Microsoft Windows Operating System Open Source IDS installed on Ubuntu Host Traffic IQ installed on the Windows Host Syslog Server installed on the Windows Host Network cables. Page 2

5 4.2 Figure 4.1 shows the network topology. 4.3 Each IDS was configured to send alerts to the Syslog Server on the Windows Host. 4.4 Traffic IQ was loaded with the last three months of traffic files and each traffic file was replayed through the IDS one at a time. At the end of each traffic file replay, the Syslog Server was refreshed and checked for the presence of an alert. 4.5 Each rule set under test was loaded individually at first and then rule sets were combined to test if the detection capability increased when multiple sets were used together. 5.0 Traffic Files 5.1 The Traffic IQ traffic files used in this months test are listed at Appendix B to this document. 6.0 Intrusion Detection Systems (IDS) 6.1 There were two IDS engines used during the testing SNORT version Build Suricata version Configuration Files Figure Configuration of the snort.conf and suricata.yaml files were maintained at the default level, as set by the IDS vendors. 7.2 Where traffic files had protocols on non-standard TCP or UDP ports, the snort.conf and suricata.yaml file variables were updated accordingly, e.g. $F_PORTS updated to include TCP port 2021 or $HT_PORTS to include Page 3

6 8.0 Rule Sets 8.1 The following rule sets and settings were included in the testing: Sourcefire Vulnerability Research Team (VRT) (Open) Connectivity Policy Balanced Policy Security Policy All rules turned on Emerging Threats (Open) Default setting (as downloaded from internet) All rules turned on Emerging Threats (Open) Suricata Default setting (as downloaded from the internet). 8.2 All rule sets were downloaded on 14th July Affective Rules 9.1 The table below shows the breakdown of the rules loaded in the different configurations and the total affective rules loaded at runtime. Basic Decoder Pre-processor Shared Object Total Affective Sourcefire VRT (Open) Connectivity Balanced Security All Rules On Emerging Threats (Open) Default (as downloaded) All Rules On* Suricata (as downloaded) *not including Deleted and Policy rules Page 4

7 10.0 Alerting 10.1 The types of alert recorded were split into four categories which are defined below: True Positive. A true positive was recorded when a traffic file was re-played and an alert was seen that correctly identified the vulnerability that was being exploited Generic Positive. A Generic Positive was recorded when a traffic file was re-played and an alert was seen that identified a type of payload, but did not identify the vulnerability that was being exploited (e.g. Shellcode detection) False Positive. A False Positive was recorded when a traffic file was re-played and an alert was seen that did not correctly identify the vulnerability that was being exploited False Negative. A False Negative was recorded when a traffic file was re-played and no alerts were seen False Positives alerts from noisy signatures have not been recorded in the results if they were accompanied by a True Positive or a Generic Positive alert. Those traffic files have been recorded as being identified and alerted on correctly Where a single traffic file has generated more than one True Positive or Generic Positive alerts, the results have only been incremented by one. The same can be said for False Positive alerts Results 11.1 The results for each test are recorded in the tables below. The result graphs are shown at Appendix A to this document Key to tables. = True Positive = Generic Positive = False Positive = False Negative 11.3 Sourcefire VRT Results. TIQ Files Sourcefire VRT (Open) Connectivity Balanced Security All Rules On Jun May Apr Page 5

8 11.4 Emerging Threats Results. TIQ Files Emerging Threats (Open) Default All Rules On* Jun May Apr ET Suricata Default Sourcefire VRT and Emerging Threats together Results. Sourcefire VRT & Emerging Threats (Open) Default & Security All Rules On* Page 6

9 Appendix A Results Graphs Total combined percentage results for April, May & June 2014 Sourcefire Vulnerability Research Team (VRT) Connectivity VRT Connectivity Policy Balanced VRT Balanced Policy Page A - 1

10 Security VRT Security Policy All Rules On VRT All Rules On Page A - 2

11 Emerging Threats (ET) Default ET Default All Rules On ET All Rules On Page A - 3

12 Suricata ET Suricata VRT and ET Together Default & Security ET Default & VRT Security Policy Page A - 4

13 All Rules On ET & VRT All Rules On Page A - 5

14 Appendix B Traffic File Lists For a full description of each traffic file used in the test, please visit or enter into your Internet Browser URL field. IQID Month IQID Month IQID Month 8728 April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April April 2014 Page B - 1

15 IQID Month IQID Month IQID Month 8824 April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May April May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May May 2014 Page B - 2

16 IQID Month IQID Month IQID Month 8947 May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June May June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June June 2014 Page B - 3

17 BACK PAGE (INTENTIONALLY LEFT BLANK) Idappcom Limited Unit 6 Rural Enterprise Centre Eco Park Road Ludlow Shropshire SY8 1FF UK Freephone: +44 (0) US Freephone: sales@idappcom.com