Open Source IDS Rules Comparison Report July 2014 DOCUMENT DETAILS Author Created on Internal Reference Simon Wesseldine 15th July 2014 VERSION CONTROL Version Release Date Overview of Changes 1.0 15th July 2014 Initial Release. 1.1 19th July 2014 Para. 1.3 text change. 1.2 21st July 2014 Traffic file references added. 1.3 21st July 2014 Spelling. 1.4 21st July 2014 Graph Changes. DOCUMENT APPROVAL Name Position Date Ray Bryant CEO of idappcom 21st July 2014 DISTRIBUTION Name Position Company The text in this document may be reproduced (excluding logos) free of charge in any format or medium providing that it is reproduced accurately and not used in a misleading context. The material must be acknowledged as copyright to idappcom ltd. Copyright idappcom Limited. All rights reserved.
CONTENTS Title Page 1.0 Executive Summary 1 2.0 Introduction 2 3.0 Aim 2 4.0 Test Overview 2 5.0 Traffic Files 3 6.0 Intrusion Detection Systems (IDS) 3 7.0 Configuration Files 3 8.0 Rule Sets 4 9.0 Affective Rules 4 10.0 Alerting 5 11.0 Results 5 Appendixes A. Results Graphs A-1 B. Traffic File Lists B-1 Page i
1.0 Executive Summary 1.1 Overview. 1.1.1 On the 15th July 2014 idappcom ltd carried out monthly testing of the Open Source IDS rules sets available on the internet. Each rule set was tested with the last three month's of exploit traffic files from Traffic IQ Professional, totalling 342 traffic files. The aim of the test was to compare the detection capabilities of each rule set in various configurations and combinations. 1.2 Results. 1.2.1 The results show how many traffic files were correctly identified, how many were incorrectly identified and how many did not get detected at all. 1.2.2 Although the configuration of Emerging Threats and Sourcefire Vulnerability Research Team (VRT) rules loaded together was a noisy combination, it had the greatest success in detecting the exploit traffic. 1.2.3 This combination still only provided protection against 56% of the exploits replayed by Traffic IQ. Rule Sets vs Exploits Detected 200 180 160 140 120 100 80 60 40 20 0 186 97 76 97 84 164 26 12 1 1.3 Key Observations. "Not one rule set could provide 100% detection against Traffic IQ files". "All rule sets and all configurations allowed more than 43% of Traffic IQ attack files through undetected". "Rule sets loaded with the default policies provide poor detection and protection capability against Traffic IQ attack files". Page 1
2.0 Introduction 2.1 Idappcom ltd was formed in Guernsey 2004. In 2009 operations moved to the UK. Since then idappcom have supplied software and hardware that enhances network security, reducing risk and lowering cost of ownership of devices. 2.2 Products are constantly evolving in response to the needs of idappcom customers. Idappcom have traditionally supplied security vendors and test labs and now supply new versions of their products specifically developed for the end user. Years of expertise and development knowledge are now available to the global market to assist their customers achieving the highest levels of network threat protection. 2.3 Idappcom's flagship application is Traffic IQ Professional, which is traffic replay software that is capable of replaying thousands of IP packet captures, safely and reliably through IT networks. Each session can be configured with any IP or MAC address and each packet shall be monitored for correct delivery and receipt. 2.4 As well as being able to replay regular pcaps, Traffic IQ is fully loaded with a library of exploit packet captures that can be used to test security devices on the network, such as IDS, IPS, Firewall, etc. Traffic IQ can report on what exploits were blocked and what exploits were allowed through security devices. 2.5 Every month the Traffic IQ library is updated with between 100 and 200 new traffic files that have been created from vulnerabilities that have been disclosed the months before. 2.6 Every month each new traffic file is tested against the latest versions of Open Source IDS Rules to see what is detected and to see what is not. This report describes the process and defines the variables used in the tests. 3.0 Aim 3.1 The aim of the tests is to compare the detection capabilities of the latest released Open Source IDS rules against the latest released Traffic IQ traffic files. 4.0 Test Overview 4.1 The physical testing environment is a closed network consisting of the following components: 4.1.1 Host with Ubuntu 12.04 LTS Operating System. 4.1.2 Host with Microsoft Windows Operating System. 4.1.3 Open Source IDS installed on Ubuntu Host. 4.1.4 Traffic IQ installed on the Windows Host. 4.1.5 Syslog Server installed on the Windows Host. 4.1.6 Network cables. Page 2
4.2 Figure 4.1 shows the network topology. 4.3 Each IDS was configured to send alerts to the Syslog Server on the Windows Host. 4.4 Traffic IQ was loaded with the last three months of traffic files and each traffic file was replayed through the IDS one at a time. At the end of each traffic file replay, the Syslog Server was refreshed and checked for the presence of an alert. 4.5 Each rule set under test was loaded individually at first and then rule sets were combined to test if the detection capability increased when multiple sets were used together. 5.0 Traffic Files 5.1 The Traffic IQ traffic files used in this months test are listed at Appendix B to this document. 6.0 Intrusion Detection Systems (IDS) 6.1 There were two IDS engines used during the testing. 6.1.1 SNORT version 2.9.6.1 Build 56. 6.1.2 Suricata version 2..0. 7.0 Configuration Files Figure 4.1 7.1 Configuration of the snort.conf and suricata.yaml files were maintained at the default level, as set by the IDS vendors. 7.2 Where traffic files had protocols on non-standard TCP or UDP ports, the snort.conf and suricata.yaml file variables were updated accordingly, e.g. $F_PORTS updated to include TCP port 2021 or $HT_PORTS to include 8081. Page 3
8.0 Rule Sets 8.1 The following rule sets and settings were included in the testing: 8.1.1 Sourcefire Vulnerability Research Team (VRT) (Open). https://www.snort.org/rules/snortrules-snapshot-2961.tar.gz?oinkcode=[oinkcode] 8.1.1.1 Connectivity Policy. 8.1.1.2 Balanced Policy. 8.1.1.3 Security Policy. 8.1.1.4 All rules turned on. 8.1.2 Emerging Threats (Open). https://rules.emergingthreatspro.com/open/snort-2.9.6/emerging.rules.tar.gz 8.1.2.1 Default setting (as downloaded from internet). 8.1.2.2 All rules turned on. 8.1.3 Emerging Threats (Open) Suricata. https://rules.emergingthreats.net/open/suricata/emerging.rules.tar.gz 8.1.3.1 Default setting (as downloaded from the internet). 8.2 All rule sets were downloaded on 14th July 2014. 9.0 Affective Rules 9.1 The table below shows the breakdown of the rules loaded in the different configurations and the total affective rules loaded at runtime. Basic Decoder Pre-processor Shared Object Total Affective Sourcefire VRT (Open) Connectivity 1059 149 0 6 1214 Balanced 4985 149 4 109 5247 Security 7373 149 21 171 7714 All Rules On 21202 149 268 285 21904 Emerging Threats (Open) Default (as downloaded) 15947 0 0 0 15947 All Rules On* 17354 0 0 0 17354 Suricata (as downloaded) 14299 0 0 0 14299 *not including Deleted and Policy rules Page 4
10.0 Alerting 10.1 The types of alert recorded were split into four categories which are defined below: 10.1.1 True Positive. A true positive was recorded when a traffic file was re-played and an alert was seen that correctly identified the vulnerability that was being exploited. 10.1.2 Generic Positive. A Generic Positive was recorded when a traffic file was re-played and an alert was seen that identified a type of payload, but did not identify the vulnerability that was being exploited (e.g. Shellcode detection). 10.1.3 False Positive. A False Positive was recorded when a traffic file was re-played and an alert was seen that did not correctly identify the vulnerability that was being exploited. 10.1.4 False Negative. A False Negative was recorded when a traffic file was re-played and no alerts were seen. 10.2 False Positives alerts from noisy signatures have not been recorded in the results if they were accompanied by a True Positive or a Generic Positive alert. Those traffic files have been recorded as being identified and alerted on correctly. 10.3 Where a single traffic file has generated more than one True Positive or Generic Positive alerts, the results have only been incremented by one. The same can be said for False Positive alerts. 11.0 Results 11.1 The results for each test are recorded in the tables below. The result graphs are shown at Appendix A to this document. 11.2 Key to tables. = True Positive = Generic Positive = False Positive = False Negative 11.3 Sourcefire VRT Results. TIQ Files Sourcefire VRT (Open) Connectivity Balanced Security All Rules On Jun 102 0 0 0 102 3 0 0 99 7 0 0 95 39 0 0 63 May 123 0 0 0 123 1 0 0 122 10 0 0 113 56 3 0 64 Apr 117 1 0 0 116 8 0 0 109 9 0 0 108 58 8 0 51 Page 5
11.4 Emerging Threats Results. TIQ Files Emerging Threats (Open) Default All Rules On* Jun 102 21 0 1 80 23 0 1 78 May 123 37 2 0 84 44 2 0 77 Apr 117 18 6 1 92 22 6 4 85 ET Suricata Default 21 0 1 80 36 0 0 87 18 1 0 98 11.5 Sourcefire VRT and Emerging Threats together Results. Sourcefire VRT & Emerging Threats (Open) Default & Security All Rules On* 25 0 1 76 44 0 1 57 42 2 0 79 69 3 0 51 22 6 1 88 62 8 4 43 Page 6
Appendix A Results Graphs Total combined percentage results for April, May & June 2014 Sourcefire Vulnerability Research Team (VRT) Connectivity 1 0 0 341 VRT Connectivity Policy Balanced 12 0 0 330 VRT Balanced Policy Page A - 1
Security 26 0 0 316 VRT Security Policy All Rules On 153 11 0 178 VRT All Rules On Page A - 2
Emerging Threats (ET) Default 76 8 2 256 ET Default All Rules On 89 8 5 240 ET All Rules On Page A - 3
Suricata 75 1 1 265 ET Suricata VRT and ET Together Default & Security 89 8 2 243 ET Default & VRT Security Policy Page A - 4
All Rules On 175 11 5 151 ET & VRT All Rules On Page A - 5
Appendix B Traffic File Lists For a full description of each traffic file used in the test, please visit http://www.idappcom.com/db/search.php or enter http://www.idappcom.com/db/search.php?fsearch=[iqid],[iqid],[iqid] into your Internet Browser URL field. IQID Month IQID Month IQID Month 8728 April 2014 8760 April 2014 8792 April 2014 8729 April 2014 8761 April 2014 8793 April 2014 8730 April 2014 8762 April 2014 8794 April 2014 8731 April 2014 8763 April 2014 8795 April 2014 8732 April 2014 8764 April 2014 8796 April 2014 8733 April 2014 8765 April 2014 8797 April 2014 8734 April 2014 8766 April 2014 8798 April 2014 8735 April 2014 8767 April 2014 8799 April 2014 8736 April 2014 8768 April 2014 8800 April 2014 8737 April 2014 8769 April 2014 8801 April 2014 8738 April 2014 8770 April 2014 8802 April 2014 8739 April 2014 8771 April 2014 8803 April 2014 8740 April 2014 8772 April 2014 8804 April 2014 8741 April 2014 8773 April 2014 8805 April 2014 8742 April 2014 8774 April 2014 8806 April 2014 8743 April 2014 8775 April 2014 8807 April 2014 8744 April 2014 8776 April 2014 8808 April 2014 8745 April 2014 8777 April 2014 8809 April 2014 8746 April 2014 8778 April 2014 8810 April 2014 8747 April 2014 8779 April 2014 8811 April 2014 8748 April 2014 8780 April 2014 8812 April 2014 8749 April 2014 8781 April 2014 8813 April 2014 8750 April 2014 8782 April 2014 8814 April 2014 8751 April 2014 8783 April 2014 8815 April 2014 8752 April 2014 8784 April 2014 8816 April 2014 8753 April 2014 8785 April 2014 8817 April 2014 8754 April 2014 8786 April 2014 8818 April 2014 8755 April 2014 8787 April 2014 8819 April 2014 8756 April 2014 8788 April 2014 8820 April 2014 8757 April 2014 8789 April 2014 8821 April 2014 8758 April 2014 8790 April 2014 8822 April 2014 8759 April 2014 8791 April 2014 8823 April 2014 Page B - 1
IQID Month IQID Month IQID Month 8824 April 2014 8865 May 2014 8906 May 2014 8825 April 2014 8866 May 2014 8907 May 2014 8826 April 2014 8867 May 2014 8908 May 2014 8827 April 2014 8868 May 2014 8909 May 2014 8828 April 2014 8869 May 2014 8910 May 2014 8829 April 2014 8870 May 2014 8911 May 2014 8830 April 2014 8871 May 2014 8912 May 2014 8831 April 2014 8872 May 2014 8913 May 2014 8832 April 2014 8873 May 2014 8914 May 2014 8833 April 2014 8874 May 2014 8915 May 2014 8834 April 2014 8875 May 2014 8916 May 2014 8835 April 2014 8876 May 2014 8917 May 2014 8836 April 2014 8877 May 2014 8918 May 2014 8837 April 2014 8878 May 2014 8919 May 2014 8838 April 2014 8879 May 2014 8920 May 2014 8839 April 2014 8880 May 2014 8921 May 2014 8840 April 2014 8881 May 2014 8922 May 2014 8841 April 2014 8882 May 2014 8923 May 2014 8842 April 2014 8883 May 2014 8924 May 2014 8843 April 2014 8884 May 2014 8925 May 2014 8844 April 2014 8885 May 2014 8926 May 2014 8845 May 2014 8886 May 2014 8927 May 2014 8846 May 2014 8887 May 2014 8928 May 2014 8847 May 2014 8888 May 2014 8929 May 2014 8848 May 2014 8889 May 2014 8930 May 2014 8849 May 2014 8890 May 2014 8931 May 2014 8850 May 2014 8891 May 2014 8932 May 2014 8851 May 2014 8892 May 2014 8933 May 2014 8852 May 2014 8893 May 2014 8934 May 2014 8853 May 2014 8894 May 2014 8935 May 2014 8854 May 2014 8895 May 2014 8936 May 2014 8855 May 2014 8896 May 2014 8937 May 2014 8856 May 2014 8897 May 2014 8938 May 2014 8857 May 2014 8898 May 2014 8939 May 2014 8858 May 2014 8899 May 2014 8940 May 2014 8859 May 2014 8900 May 2014 8941 May 2014 8860 May 2014 8901 May 2014 8942 May 2014 8861 May 2014 8902 May 2014 8943 May 2014 8862 May 2014 8903 May 2014 8944 May 2014 8863 May 2014 8904 May 2014 8945 May 2014 8864 May 2014 8905 May 2014 8946 May 2014 Page B - 2
IQID Month IQID Month IQID Month 8947 May 2014 8988 June 2014 9029 June 2014 8948 May 2014 8989 June 2014 9030 June 2014 8949 May 2014 8990 June 2014 9031 June 2014 8950 May 2014 8991 June 2014 9032 June 2014 8951 May 2014 8992 June 2014 9033 June 2014 8952 May 2014 8993 June 2014 9034 June 2014 8953 May 2014 8994 June 2014 9035 June 2014 8954 May 2014 8995 June 2014 9036 June 2014 8955 May 2014 8996 June 2014 9037 June 2014 8956 May 2014 8997 June 2014 9038 June 2014 8957 May 2014 8998 June 2014 9039 June 2014 8958 May 2014 8999 June 2014 9040 June 2014 8959 May 2014 9000 June 2014 9041 June 2014 8960 May 2014 9001 June 2014 9042 June 2014 8961 May 2014 9002 June 2014 9043 June 2014 8962 May 2014 9003 June 2014 9044 June 2014 8963 May 2014 9004 June 2014 9045 June 2014 8964 May 2014 9005 June 2014 9046 June 2014 8965 May 2014 9006 June 2014 9047 June 2014 8966 May 2014 9007 June 2014 9048 June 2014 8967 May 2014 9008 June 2014 9049 June 2014 8968 June 2014 9009 June 2014 9050 June 2014 8969 June 2014 9010 June 2014 9051 June 2014 8970 June 2014 9011 June 2014 9052 June 2014 8971 June 2014 9012 June 2014 9053 June 2014 8972 June 2014 9013 June 2014 9054 June 2014 8973 June 2014 9014 June 2014 9055 June 2014 8974 June 2014 9015 June 2014 9056 June 2014 8975 June 2014 9016 June 2014 9057 June 2014 8976 June 2014 9017 June 2014 9058 June 2014 8977 June 2014 9018 June 2014 9059 June 2014 8978 June 2014 9019 June 2014 9060 June 2014 8979 June 2014 9020 June 2014 9061 June 2014 8980 June 2014 9021 June 2014 9062 June 2014 8981 June 2014 9022 June 2014 9063 June 2014 8982 June 2014 9023 June 2014 9064 June 2014 8983 June 2014 9024 June 2014 9065 June 2014 8984 June 2014 9025 June 2014 9066 June 2014 8985 June 2014 9026 June 2014 9067 June 2014 8986 June 2014 9027 June 2014 9068 June 2014 8987 June 2014 9028 June 2014 9069 June 2014 Page B - 3
BACK PAGE (INTENTIONALLY LEFT BLANK) Idappcom Limited Unit 6 Rural Enterprise Centre Eco Park Road Ludlow Shropshire SY8 1FF UK Freephone: +44 (0)800 680 0791 US Freephone: +1 888 433 8835 email: sales@idappcom.com