CorreLog IP Block List and Reputation Database Application Notes

Transcription

1 CorreLog IP Block List and Reputation Database Application Notes As a standard feature of the CorreLog Server software, CorreLog Inc. synthesizes and maintains a robust list of IP address subnets with bad reputations, which are saved in the "@@ip_blocklist@@ " list macro at the CorreLog Server site. This list is updated each week by CorreLog, and placed at a well-known URL on the CorreLog website, permitting easy access by CorreLog users and licensees. Specific tools are available from CorreLog to allow this reputation database to be automatically downloaded to each server, thereby maintaining a current list of subnets with bad reputations, installed at each CorreLog Server site. This application note furnishes information on the actual block list and reputation database feed, as well as information on how to configure the system to automatically download this list via an Adapter at the CorreLog Server. Component Overview The CorreLog IP Block List Reputation Database system several consists of several main parts. Block List Macro. The actual reputation database is contained in a standard CorreLog "List" type macro (viewable via the "Correlation > Config > Lists" screen.) This list macro has the and consists of 1000's of subnet addresses that identify devices with a bad reputation. Block List Feed URL. CorreLog furnishes a URL that contains a list of addresses suitable for use with list macro described above. CorreLog synthesizes a new block list periodically (about once each week) that is acquired from various public providers. (The providers are documented in a later section of the manual here.) Block List Adapter Screen. CorreLog furnishes a screen that permits the operator to configure automatic retrieval of the above feed, and incorporate changes into list macro. This automates the process of acquiring and maintaining the list of devices with bad reputations.

2 Auto-Update Feed Adapter CorreLog Server includes a simple adapter that adds a screen component to assist with automatic updates of the reputation database. This adapter is available on request, and creates the "System > Tools > Auto-Update > IP Reputation DB" tab on the system the system, which permits the administrator to schedule the fetching of the standard reputation database and report. This adapter screen is depicted below: The above screen is a standard CorreLog Server dialog. The operator clicks the "Edit" button to edit the parameters, then clicks "Save" to save the parameters for future operation. By default, the feed is fetched at the start of each month from the correlog.com website. The feed can be fetched immediately by clicking the "Download" button. The operator can view the feed process log, and can access the full report data via links at the top of the screen. Page: 2

3 The various fields and controls for this screen are as follows: Refresh Button. This button refreshes the screen with the latest information. If a new list is being downloaded, this causes the "Status" line (beneath the button) to refresh showing the progress of the Auto-Feed operation. Edit Button. This button allows the "Feed Master Enable", the "Feed URL", the "Scheduled Execution", and the "Exclude" settings to be edited. (These feels are further identified below.) Run Report Button. This button immediately fetches the feed. (Otherwise, the operator can wait for the scheduled execution, specified via the "Edit" screen.) Feed Master Enable. This setting can be changed via the "Edit" button, and is the master enabled for the scheduled feed update. A value of "Enabled" will enable the periodic process. A value of "Disabled" will disable the process (but still allow reports to be fetched automatically via the "Run Report" button. Feed URL. This setting is the URL to the feed site. Unless otherwise instructed or advised by support, the value should not be changed, and is configured to correctly access the reputation database described in this document. Proxy URL. This setting is the URL to a proxy server (if required). The proxy HTTP server should be specified as a standard URL and port number combination. If no HTTP proxy server is required or exists, then this field should be left blank to directly fetch files from the "Feed URL". Schedule Execution. This setting permits the operator to specify the schedule of when the feed is fetched from the Feed URL. The value is reflected in the "System > Scheduler" screen. The operator can select "weekly", "monthly" or an advanced schedule. (See notes on the "System > Scheduler" screen in other documents for a further discussion of controls.) Exclude Single References. This setting can be changed via the "Edit" button, and indicates the rigorousness of the list. By default, the value is "No", which indicates any subnet in the IP Block list feed will be regarded as having a bad reputation. All entries in the block list will appear in list macro. Setting the value to "Yes" requires the entry to be referenced at least twice (i.e. included in at least two lists described earlier.) This can be used to reduce false positives in some environments, by requiring the IP to be recognized by at least two lists. Identify Bad Subtest. This setting can be changed via the "Edit" button, and will output a subnet block address if more than 25 different IP addresses in the subnet are identified to have a bad reputation. This can enhance security, but Page: 3

4 can also cause false positives. Share Threat Intelligence. Adjusting this value to "Yes" will cause the top 10 devices that match the IP blocklist to be posted to the CorreLog corporate website (via an HTTP Post request.) This occurs after the IP blocklist is fetched. No other data or corporate information is shared, and the posting is completely anonymous. Setting the value to "Yes" assists CorreLog engineering with constructing the weekly reputation database. List Metric Values. The bottom of the screen indicates the number of IP addresses in the list, the size of the file, and the time that the file was downloaded. These metrics correspond to the operating lists on the system, and depend on when the list was fetched, and whether "Exclude Single References" is set to "Yes" or "No". Handling False Positives The easiest way to reduce the number of false positives for the reputation database is to simply set the "Exclude Single References" value to "Yes", which means that an IP address is not identified as having a bad reputation unless it appears on at least two lists. This will reduce the number of entries in immediate. (Note that after making this type of change to the "Edit" screen, the user should click "Run Report" to fetch the new reputation database.) Another way of handling false positives is to add any IP addresses used by your organization (which may appear in the CorreLog list, but are necessary or known to your organization) to macro on the "Correlation > Config > Lists" screen. This macro typically contains a list of IP addresses which are not blocked under any circumstances. (The user simply updates the list of IP addresses like any other list macro.) The correlation rules in the "Correlation > Threads" screen reference a rule "@@ip_blocklist@@ and which indicates that a match has to occur in macro, and NOT occur in list. Note that if you update list with an item, that item will be eliminated next time the feed is executed (typically on a weekly basis.) Therefore, you should not modify This is not a problem with list, which is entirely defined by your organization and never modified by CorreLog feeds or upgrade procedures. Finally, if you have chronic problems with certain ranges of devices, you should contact CorreLog support to review your situation. The CorreLog reputation database is easily modified to exclude certain IP addresses that may be necessary for your site. Page: 4

5 Block List Feed Information IP address information is obtained from various well-known sites that supply public domain access to IP and network reputation data. To qualify as a source site, CorreLog requires that the site be well-known and have a valid and verified WHOIS database entry. Additionally, CorreLog applies other proprietary validity checks to each IP entry. Specific sites accessed by this CorreLog initiative are as follows: (List Identifier: L_MYIP) - This website contains records in htaccess format, updated approximately once each week. WHOIS Contact Info: Michael Williams, Delta Consultants Ltd, 8 Copthall, Roseau Valley, NE (List Identifier: L_EMTHR) - This list contains raw IPs for the firewall IP block lists, derived from Spamhaus ( Top Attackers listed by DShield ( and Abuse.ch. WHOIS Contact Info: Proofpoint, Inc., 892 Ross Drive, Sunnyvale, CA (List Identifier: L_DSH) - This list contains current IP block list information, updated approximately once each week. The top 2000 sites are incorporated into the CorreLog IP block list. DShield operates a comprehensive threat website since WHOIS Contact Info: Johannes Ullrich, PO Box 13314, Jacksonville, Florida (List Identifier: L_FIREH) - This is a firewall blacklist maintained by firehol.org The list is suitable for protection on all internet facing servers, routers and firewalls, The list includes various IP sources: bambenek_c2, dshield, feodo, fullbogons, palevo,spamhaus_drop, spamhaus_edrop, sslbl, and zeus_badips. WHOIS Contact Info: Firehol Organization, 96 Mowat Ave, Toronto Ontario Canada. (List Identifier: L_MALC) - This is a firewall block list maintained by malc0de.org. WHOIS Contact Info: Dreamhost 417 Associated RD #324, BREA, CA (List Identifier: L_ZEUS) - This list is maintained by the Swiss Information Security Research Association, containing severe abuses. WHOIS Contact Info: Swiss Information Security Research Association SISRA, Bernet Monika, CH Zurich, Switzerland Page: 5

6 (List Identifier: M_TCRWD) - This list contains IP block list information that has been manually voted on by threatcrowd.org participants. The list is updated at least monthly. (See for information.).whois Contact Info: Chris Doman, 27 Bramley close, Colchester, Essex GB, CO38RU (List Identifier: D_TCRWD) - This list is similar to the above list, but contains domain names that have been manually voted on (see above.) This list is updated at least monthly. CorreLog performs a DNS lookup of these domains to acquire the IP address of each domain, which is incorporated into the CorreLog IP block list. (List Identifier: D_HOSTF) - This is a domain list contains domains that are engaged in malware distribution. (EMD, Exploit Malware Distribution) CorreLog performs a DNS lookup of these domains to acquire the IP address of each domain, which is subsequently incorporated into the CorreLog IP block list. WHOIS Contact Info: Robert Hafner, Malwarebytes, Corporation (List Identifier: D_MALW) - This is a domain list similar to above, except is a more general list furnished by This is a comprehensive list updated monthly, mainly used as a DNS sinkhole. CorreLog performs a DNS lookup of these domains to acquire the IP address of each domain, which is subsequently incorporated into the CorreLog IP block list. WHOIS Contact Info: ISK Analytics, LLC, 4370 W 109th Street, Suite 250, Leawood, KS (List Identifier: L_BLDE) - This website contains a comprehensive set of about 40,000 blocked IP addresses. The site is a free and voluntary service provided by a Fraud/Abuse-specialist, whose servers are often attacked. The site works in conjunction with and other sites. WHOS Contact Info: Martin Schiftan, Tumblingerstr , 80337, Munchen, Germany NOTE: Addresses in this list are first checked with matches to other lists, since this particular list has been shown to be filled with false positives. Hence, no address in this list is included unless it appears in other lists above. Page: 6

7 Data Reporting As supporting documentation for this IP data, CorreLog generates a comprehensive report that can be used to determine the status of each IP address entry. This report is generated for each new reputation database update, and is downloadable from the CorreLog website. This report contains the following metrics. Subnet Address. This is the address subnet entry in the reputation database. There are typically between 10 and 30 thousand entries in the list. These items are added to list macro (either manually or by the "Auto- Feed" adapter described in later sections. Country Code and Name. This is the two letter country code and full country name for the subnet entry in the database. This is the same information as found in the Geo-IP database. Registration Date. This is the date and time of the registration for the IP address hostname. Reference Hostname. This is the hostname entry corresponding to the first device on the subnet (if known.) The value may be "Unknown". Reference List Identifier and Reference Count. This is the name of the block list (corresponding to the "List Identifiers" in the previous section) that indicates where this entry can be found. If the IP address matches multiple lists, only the first list identifier is provided. This field also lists is the number of lists that match the specified subnet name. This value can also be displayed by entering the IP address of a device in the standard "DNS Tool". See additional note below. Persistence. This flag is either "Yes" or "No", and indicates the particular device has previously been found on the system (Yes) or is a new device (No). This can be used to determine how promiscuous this subnet is. The report information and corresponding block list are available via separate URLs, typically downloaded automatically via the "Auto-Update Feed Adapter" described in the next section. The user can access the report information either from the Adapter plug-in screen, or via the "DNS Tool" screen (available via the "More" menu in the upper right of all screens.) Page: 7

8 Summary and Additional Notes 1. The reputation database is configured within the system, residing in the "Correlation > Config > Lists" screen, within the macro. This list can be manually modified (but any changes will be lost during the next update of the system.) This particular list macro is used by various preconfigured correlation threads and alerts. 2. After installing the REPDB adapter, the operator can navigate to the "System > Tools > Auto Update" tab to view the IP Reputation Database screen, as depicted in this manual. This screen contains controls, status, and debug information necessary to download the reputation database and update the list macro. 3. After installing the REPDB adapter, the administrator should edit the "IP Reputation Database" screen and set the "Scheduled Execution" time to be some value other than "None" for automatic updates to occur. (Otherwise and update occurs only when the user clicks the "Run Report" button on the screen.) NOTE: By default no automatic updates occur until the user sets the scheduled time to something other than the default "None" value. 4. The "GenRepDB.exe" program, which is responsible for obtaining the reputation database, is automatically configured to run by setting the "Scheduled Execution" time above. This program also appears on the "System > Scheduler" screen. 5. The "CorreLog\feeds" folder contains files used by the system, including MD5 checksums and other identification information. These files should not be modified without assistance from support. 6. No updates occur if any errors are encountered with the process, including errors with checksums on the files. In this case, the user should click the "Process Log" link to diagnose the issue. 7. The "CorreLog\feeds\GET_IP_FEED.bat" file is actually responsible for downloading the files from the reputation database using the "wget.exe" program (where the "wget.exe" program is added to the "system" folder by the installation package.) The "CorreLog\feeds\GET_IP_FEED.log" file contains the last transcript of the download operation, useful for debug and analysis. CorreLog's IP Reputation Database feed, while publicly available, may be disabled for specific users and sites if the URL is over-accessed. Sites should not download the reputation database more than once a week, except under certain circumstances. If the user cannot obtain the reputation database for any reason, contact CorreLog support for assistance. Page: 8

9 For Additional Help And Information Detailed specifications regarding the CorreLog Server, add-on components, and resources are available from our corporate website. Test software may be downloaded for immediate evaluation. Additionally, CorreLog is pleased to support proof-ofconcepts, and provide technology proposals and demonstrations on request. CorreLog, Inc., a privately held corporation, has produced software and framework components used successfully by hundreds of government and private operations worldwide. We deliver security information and event management (SIEM) software, combined with deep correlation functions, and advanced security solutions. CorreLog markets its solutions directly and through partners. We are committed to advancing and redefining the state-of-art of system management, using open and standards-based protocols and methods. Visit our website today for more information. CorreLog, Inc. mailto:support@correlog.com Page: 9