Documentation for: MTA developers

Transcription

1 This document contains implementation guidelines for developers of MTA products/appliances willing to use Spamhaus products to block as much spam as possible. No reference is made to specific products. Rather, the focus is on the actions to be taken. It will be assumed throughout that DQS is used as a delivery mechanism. API access Spamhaus databases are consulted through DNS queries, where the object to be queried (an IP or a domain/hostname) is prepended to a DNSBL domain zone. DNSBL domain zones are called <key>.<name>.dq.spamhaus.net, where <name> is the zone name and can be sbl, xbl, sbl-xbl, pbl, zen, dbl or zrd (see table below), while <key> is a 26-character code specific to each customer. Keys corresponding to terminated contracts no longer work. Also, IP and domain services can be individually turned on and off for a certain key. If a service is off, a DNS "refused" answer packet is returned, resulting in a SERVFAIL DNS answer. The software should make sure that the key actually works before using it, and the check must be done for both IP and domain services if both are being used. We recommend the following test queries of type A to make sure that the service is active; for IP data access: <key>.sbl.dq.spamhaus.net This query will return if the query service for IP addresses is active, SERVFAIL if it is not. Similarly, for domain data: RFC Access methods as well as implementation details on DNSBLs are described in RFC5782 Query failure If a DNSBL query returns SERVFAIL, something is off test.<key>.dbl.dq.spamhaus.net This query will return if the query service for domain data is active, SERVFAIL if it is not. 1 of 9

2 The Spamhaus DNSBL zones The following table summarizes the DNSBL zones that can be queried, the Spamhaus databases they are connected to, and the possible return codes (A record in the DNS answer). In all cases a TXT lookup returns a URL pointing to a description of the problem. NXDOMAIN answers indicate that the IP address or domain queried is not listed. DNSBL Type Database Return Codes sbl IP SBL SBL (manually managed) CSS (automated) DROP (always in addition to SBL) xbl IP XBL CBL from to.7 - currently unused sbl-xbl IP SBL+XBL SBL CSS DROP CBL (see details above) Return Codes DNSBLs return one or more A records for positive replies. Each returned A record (associated with a different IP) is used to represent a specific message pbl IP PBL entry maintained by ISP entry maintained by Spamhaus zen IP SBL+XBL+PBL SBL CSS DROP CBL PBL PBL (see details above) dbl Domain DBL low-reputation domain phishing-related domain malware-related domain botnet C&C domain abused-legit domain abused redirector abused domain used in phishing abused domain used by malware abused domain hosting C&C ERROR: IP query against a domain list zrd Domain ZRD domain first seen between 0 and 2 hours ago domain first seen between 2 and 3 hours ago [ ] domain first seen between 23 and 24 hours ago 2 of 9

3 Query structure The zones sbl, xbl, sbl-xbl, pbl and zen answer queries for IP addresses, while dbl and zrd answer queries for domains. One must avoid to send IP queries to domain databases or domain queries to IP databases. IPs can be either IPv4's or IPv6's, as described below. Queries for IPv4 addresses. In this case, the four octets of the IP addresses are inverted in the query. For instance, to query sbl about the listing status of one would use <key>.sbl.dq.spamhaus.net Queries for IPv6 addresses. In this case the address must be transformed into a nibble format, which means all the hex digits of the expanded IPv6 address in reverse order and separated by dots. For instance, to query sbl about the listing status of 2001:db8:7ca6:22::45 one would use RDNS The format used to represent IP addresses in DNSBL queries is the same used by the in-addr.arpa and ip6.arpa zones to represent reverse DNS lookups a.c. 7.8.b.d <key>.sbl.dq.spamhaus.net Queries for domains or hostnames. In this query the domain/ hostname is simply prepended to the dbl (or zrd) DNSBL zone name: example.com.<key>.dbl.dq.spamhaus.net It must be emphasized that both dbl and zrd are wildcarded zones that do not consider the hostname part of fully qualified domain names. Therefore, fully qualified domain names can be inserted as they are in the DNS queries, without having to strip away the domain part. For both IP and domain datasets, a reply providing one or more A records within /8 is considered a positive reply (meaning the queries resource is listed), whereas an NXDOMAIN (host not found) represents a negative reply (resource not listed). Querying code is expected to go through all the A records provided by a positive reply and act accordingly to each one, as opposed to just picking up the first entry, as that single reply may not be the one the specific check was supposed to trigger upon. Therefore, for example, checking the IP against the zen zone may return the following: <key>.zen.dq.spamhaus.net. 60 IN A <key>.zen.dq.spamhaus.net. 60 IN A <key>.zen.dq.spamhaus.net. 60 IN A <key>.zen.dq.spamhaus.net. 60 IN A Indicating that this IP is listed in SBL ( ), in the CSS component of SBL ( ), it is part of a DROP IP range ( ) and is listed in the CBL (part of XBL) as well ( ). It is not listed on PBL. 3 of 9

4 Queries returning IPs outside /8 are absolutely not expected and mean something is interfering with the DNS resolution process. Such replies must be discarded and the DNS resolution chain investigated to exclude the misbehaving actor. Zones, Databases and Datasets Some words are worth spending clarifying the differences between the concepts of zones, databases and datasets and their meaning in the Spamhaus data. A zone is referred to a DNS API endpoint used to access a certain set of databases (one or more). For example, zen is the way endpoint exposing the three databases named sbl, xbl and pbl together, so they can all be queried with a single request. A database is a corpus of data that is distributed as a single entity, and is composed of one or more datasets. For example, the sbl database groups together the actual sbl dataset, the css dataset and the DROP dataset. Usually the record returned by each dataset can be distinguished based on the return code, in order to allow the querying software to take different actions based on the actual dataset matched by the query. A dataset is a set of records that share the same purpose and policy, and are usually built by the same processes. Sometimes this actually hides an additional layer of datasets that are in truth separate in terms of origin for technical reasons, but are seen as a single entity as they can just be treated the same way from the consumer point of view. An example of this is css, that is in truth composed of the two separate datasets ccs4 (for IPv4 data) and css6 (for IPv6 data). Sometimes the distinction above can be perceived as confusing, as some zones have the same name as a database and a dataset. This happens for historical reasons: the SBL was originally a database consisting of a single dataset and published as a zone. With time other datasets started being created and added to the same zone as integrations. Usually this is not a big issue, as when it comes to consuming the data all the querier sees and should care about is the return code received as reply to a query, as this is independent from the zone the query was performed against. References to -for example- listed by sbl should therefore be intended as listed by the sbl dataset, unless specified otherwise. Datasets enumeration and description SBL ( ) It s a manually maintained list of abuse-related resources, not necessarily of exclusively SMTP emitters. Resources that can be listed in the SBL are for example webservers or DNS servers (sometimes, even routers) providing service to abusing actors, either as a result of a compromise or because they're dedicated to that purpose. In general, outright blocking at the SMTP level a source that is listed by the SBL is supposed to be safe in terms of false positives. 4 of 9

5 Another usage with a fairly low false-positive rate is checking the IPs contained in the Received headers of the messages (socalled Deep-Header Parsing, or DHP). Due to the characteristics above, however, other uses are possible: for example, a sender whose domain is served by an SBL-listed DNS server has a non-trivial probability of being abusive too. Similarly, if the message contains URLs resolving to SBL-listed addresses, there's a reasonable chance the message is abusive. However, use of the SBL for these specific purposes is encouraged only within scoring systems, as a contribution to a decision taken upon multiple factors. CSS ( ) It's a completely automated sublist, listing SMTP emitters associated with a low reputation or confirmed abuse. This can either mean a resource controlled by an abusing actor or a compromised host. Its usage should be limited to the sending IP and can be used to outright reject the delivery. DROP ( ) It's an additional flag added to SBL listings, indicating that the resource is known to be controlled by a bad actor, usually part of IP resources assigned to known rogue entities. Bulletproof hosters and similar shady operators are a typical example. It is strongly suggested to avoid any kind of interaction with entities listed by this dataset. XBL ( ) It's a list of IPs hosting compromised host. As these hosts are generally used to emit spam (among other bad deeds) the first suggested use for this dataset is to outright block SMTP deliveries coming from an IP listed by it. Hosts can be compromised and be used for abusive purposes even without actually emitting spam, however. For this reason other usages are possible: for example if an URL contained in the message body points to an exploited webserver, there's a non-trivial chance that the message itself is spam, pointing the recipient to an abusive URL that will be redirecting him to the spammer's website or -in the worst case- downloading malware of some sort. Using the XBL to check the IPs URLs point to is therefore possible and suggested, but only as part of a scoring system where this is one of the indicators taken into account. Similarly, using DHP against the XBL is possible, but the chance of false positive can be quite significant, particularly in cases where the source is on a dynamically assigned address (meaning the sender inherited an IP that hosted a compromised system hours before) or in case of NAT (where one host is compromised but most others are not, but all share the same public IP); therefore, it should only be used in a scoring system. 5 of 9

6 PBL ( ) It's not -strictly speaking- an abuse-related list: it's a list of dynamic and low-security IP space. In general, it's address space that should never host an SMTP server, therefore any SMTP connection coming from this IP space is almost certainly abusive. Since every message has to originate somewhere, DHP against PBL makes no sense and is highly discouraged. On the other hand, scoring based on PBL for URLs is possible, although not particularly performant. Two return codes are associated with this dataset, telling whether the nature of the listed subnet has been inferred by Spamhaus ( ) or indicated directly by the ISP responsible for the network ( ). DBL ( ) It s a database of domains with a poor reputation. What the DBL does is effectively keeping track and computing a reputation score for every domain seen on the Internet and produce a list of those that are above a certain threshold have been observed active in the last X days Different return codes are used to tag the type of abuse the domain has been observed involved in whenever that information is available. One thing that should be noted is that not all the records have the same meaning in term of badness : basically two separate sets of return codes are provided: identify resources that are considered inherently bad or associated with a low reputation. In general, it means that the domain is safe to block according to Spamhaus data identify domains that -while not inherently bad - have been observed involved in abuse. Briefly referred to as abused-legit the typical example of this is a domain that due to a security issue is currently serving malicious contents. This second set of return codes is only suggested for use in scoring systems. If queried for an IP, the DBL will return a positive reply with the return code : this should be under any aspect treated as an error code, with the meaning IP queries not supported. ZRD ( ) It s a database of domains that have been observed for the first time less than one day before. Given the amount of abusive domains registered every day just to be thrown away immediately after minutes of use, these domains should and can be treated with extreme prejudice. The fourth octet of the return code is used to indicate the time elapsed since its first observation (from 0 to 24 hours). 6 of 9

7 Checking SMTP messages In an SMTP transaction, the sending host opens a TCP connection on port 25 and basically sends the following commands: HELO myname.helodomain.com MAIL FROM: <user@senddomain.com> RCPT TO: <ouruser@ourdomain.com> DATA Headers Body (with EHLO possibly in place of HELO). Before the DATA stage even starts -and therefore before the message is actually transmitted- the SMTP protocol gives the following four parameters that can be used to check the sender's reputation: connecting IP address (IP) reverse DNS (PTR) of the connecting IP address, if present (Domain) domain used in HELO/EHLO like helodomain.com in the example above (Domain) domain used in MAIL FROM (envelope from) like senddomain.com in the example above (Domain) Spamhaus recommends the following actions based on these parameters before getting to the DATA stage: reject the transaction if the connecting IP is listed by the SBL zone components, by XBL or by PBL (in other words, any hit against the zen zone, as long as the return code is contained in /8) reject the transaction if the reverse DNS of the connecting IP (when defined) is listed by DBL with a return code lower than reject the transaction if the domain used in HELO/EHLO is listed by DBL with a return code lower than reject the transaction if the domain used in MAIL FROM is listed by DBL with a return code lower than reject the transaction if the reverse DNS of the connecting IP (when defined) is listed by ZRD with a return code lower than or equal to reject the transaction if the domain used in HELO/EHLO is listed by ZRD with a return code lower than or equal to trigger greylisting if the domain used in MAIL FROM is listed by ZRD with a return code lower than or equal to Numeric HELO Despite being an error code, can still be useful when the resource checked against the DBL is an HELO string, as no valid HELO string can be in the form of an IP address Note that DBL return codes larger than refer to abused legitimate domains and they should be used only in contents analysis of message bodies to prevent false positives. 7 of 9

8 Also note that the "24" in the ZRD rules is the maximum number of hours elapsed from the first observation of the domain, and can be decreased for a less aggressive behaviour toward new domains appearing on the Internet. SMTP transactions not rejected by the criteria above should be accepted and subjected to the contents analysis described below. Both the message headers and the message body are transmitted within the SMTP DATA command. For platforms and traffic volumes that allow this, the contents analysis should preferably be done while the original SMTP connection is still open, at the end of the DATA stage but before OK'ing the transmission to the sender. This would give the opportunity to issue an immediate rejection based on contents to the sending server, rather than accepting and bouncing it later as a non-delivery notification to the envelope sender. The envelope sender is often forged in spam, and such non-delivery notifications would turn the receiving server into a backscatter spam source. "Spam folders" are commonly used to avoid this problem, but not notifying the sender in any way could also be a problem in case a legitimate message is flagged as spam. Immediate rejections during the SMTP DATA stage do not cause backscatter. Headers checks We suggest to check the following actions: score the message negatively if an IP address appearing in the second Received: line, or deeper ones when present, is listed in SBL, CSS or XBL flag the mail as spam if the domain appearing in the From: user@fromdomain.com line (if present) is listed by DBL with a return code lower than flag the mail as spam if the domain appearing in the Reply-To: user@replytodomain.com line (if present) is listed by DBL with a return code lower than flag the mail as spam if the domain appearing in the Message-ID: <string@msgiddomain.com> line is listed by DBL with a return code lower than Again, DBL return codes larger than refer to abused legitimate domains and we recommend to use them only to score URLs in message bodies to prevent false positives. Also note that PBL listings should never be used as a spam criterion for originating IPs appearing in Received: header lines. Legitimate messages are normally originated by IP addresses listed in PBL, and they must not be penalized in any way for this reason. Body checks After properly decoding the message (that can use particular character sets, be encoded in Base64, etc), we recommend to identify all the URLs, including addresses, appearing in the message body, and then extract IP addresses and domains out of these URLs. 8 of 9

9 Then the following checks can be operated: score the message negatively if any URL contains an IP address listed by the SBL or XBL zones components (any return code) score the message negatively if any URL contains a domain/ hostname listed by DBL (any return code, including the abuse legit ones, although different scores should be applied to the two groups) or by ZRD (any return code, or return codes limited to a maximum of N hours from the appearance of the domain) Optionally one can also: score the message negatively if any URL contains a domain/ hostname authoritatively served by a nameserver whose IP address is listed by SBL score the message negatively if any URL contains a domain/ hostname authoritatively served by a nameserver whose domain is listed by DBL with a return code lower than Numeric URLs Remember that DBL is not expected to receive any IP-based query. If the URL contained in the message body is pointing to a raw IP, that should be checked against IP-based databases only 9 of 9