grey listing

From Wikipedia, the free encyclopedia

If you are experiencing mail delivery problems relating to grey listing, please contact our support team.

 

Grey listing (or graylisting) is a new method of blocking significant amounts of spam at the mail server level, but without resorting to heavyweight statistical analysis or other heuristically and error-prone approaches. Consequently, implementations are fairly lightweight, and may even decrease network traffic and processor load on your mail server.

Grey listing relies on the fact that most spam sources do not behave in the same way as "normal" mail systems. Although it is currently very effective by itself, it will perform best when it is used in conjunction with other forms of spam prevention. For a detailed description of the method, see the Whitepaper.

 

The term Grey listing is meant to describe a general method of blocking spam based on the behaviour of the sending server, rather than the content of the messages. Grey listing does not refer to any particular implementation of these methods. Consequently, there is no single Grey listing product. Instead, there are many products that incorporate some or all of the methods described here.

 

How it works

 

Typically, a server employing grey listing will record the three pieces of data known as a "triplet" for each incoming mail message:

• The IP address of the connecting host
• The envelope sender address
• The envelope recipient address (es)

This is checked against the mail server's internal database. If this triplet has not been seen before (within some configurable period), the email is grey listed for a short time (also configurable), and it is refused with a temporary rejection with a SMTP 4xx error code. The assumption is that since temporary failures are defined in the SMTP-related RFCs, a legitimate server will try again to deliver the email.

 

The temporary rejection can be issued at different stages of the SMTP dialogue, allowing an implementation to store more or less data about the incoming message. The trade-off is more work and bandwidth for more exact matching of retries with original messages. Rejecting a message after its content has been received allows the server to store a choice of headers and/or a hash of the message body.

 

In practice, most grey listing systems do not require an exact match on the IP address and the sender address. Because large senders often have a pool of machines that can send (and resend) email, IP addresses that have the most-significant 24 bits (/24) the same are treated as equivalent, or in some cases SPF records are used to determine the sending pool. Similarly, some e-mail systems use unique per-message return-paths, for example variable envelope return path for mailing lists, Sender Rewriting Scheme for forwarded e-mail, Bounce Address Tag Validation for backscatter protection, etc. If an exact match on the sender address is required, every e-mail from such systems will be delayed. Some grey listing systems try to avoid this delay by eliminating the variable parts of the VERP by using only the sender domain and the beginning of the local-part of the sender address.

 

Why it works

 

Grey listing is effective because many mass email tools used by spammers will not bother to retry a failed delivery, so the spam is never delivered. A spam sender may retry with a different sender, and possibly a different message, because it has a queue of victims rather than the proper queue of messages that regular mail servers maintain.

 

In addition, if a spammer does retry a delivery after the waiting period has expired, any one of a number of automated spam traps will have had a good chance of identifying the spam source and listing both the source and the particular message in their databases. Thus, these subsequent attempts are more likely to be detected as spam by other mechanisms than they were before the grey listing delay.

 

Advantages

 

The main advantage from the users' point of view is that grey listing requires no additional configuration from their end. If the server utilizing grey listing is configured appropriately, the end user will only notice a delay on the first message from a given sender, so long as the sending email server is identified as belonging to the same white listed group as earlier messages. If mail from the same sender is repeatedly grey listed it may be worth contacting the mail system administrator with detailed headers of delayed mail.

 

From a mail administrator's point of view the benefit is twofold. Grey listing takes minimal configuration to get up and running with occasional modifications of any local white lists. The second benefit is that rejecting email with a temporary 451 error (actual error code is implementation dependent) is very cheap in system resources. Most spam filtering tools are very intensive users of CPU and memory. By stopping spam before it hits filtering processes, far fewer system resources are used. This allows more layers of spam filtering or higher throughput since greylisting can easily be configured as a first line of defense with SpamAssassin etc. handling messages that go through.

 

Some grey listing packages support an SQL backend which allows for a distributed multiple-server frontend to be deployed with the same greylisting data on all frontends.

 

Disadvantages

 

Perhaps the most significant disadvantage of grey listing is the fact that, like some other spam mitigation techniques, it destroys the near-instantaneous nature of email people have come to expect. A customer of a grey listing ISP cannot always rely on getting every email in a pre-determined amount of time. However, the original specification for email states that it is not a guaranteed delivery mechanism and not an instantaneous delivery mechanism. This means that grey listing is a perfectly legitimate process and does not break any protocols or rules. Traditionally, grey listing is very good at flushing out poorly configured mail servers that cannot maintain state, queue email correctly, or retry delivery within a reasonably short time. Mail servers that are properly configured and fully conform to SMTP generally have no problems with grey listing techniques and delays are very small so as not to be a problem.

 

If mail from a particular frequent sender is sent from any of several mail servers, mail may be delayed unless the grey listing server recognises the different servers as belonging to the same white listed group.

 

On a technical level, some SMTP clients and SMTP servers acting as clients may interpret the temporary rejection as a permanent failure. Old clients conforming only to the obsolete specification (RFC 821) and ignoring its recommendations may give up on delivery after the first failed attempt—RFC 821 states that clients "should" retry messages rather than using the word "must". RFC 2119 dictates that "should" means recommended and to ignore at your own risk, and it is a violation of the current SMTP standard for the client to fail to retry. The current SMTP specification (RFC 5321) clearly states that "the SMTP client retains responsibility for delivery of that message" (section 4.2.5) and "mail that cannot be transmitted immediately MUST be queued and periodically retried by the sender." (Section 4.5.4.1).

 

This problem can affect SMTP clients in unexpected ways. Most MTAs will queue and retry messages, but a small number do not. A similar concern exists for applications which act as SMTP clients and fail to incorporate any form of queuing for deferred SMTP mail. This can be mitigated on the sending side by configuring the application to use a local SMTP server as an outbound queue, instead of attempting direct delivery. For the server operator who uses grey listing, clients which are known to fail on temporary errors can be supported by white listing or exception lists.

 

Some MTAs, upon encountering the temporary failure message from a grey listing server on the first attempt, will send a warning message back to the original sender of the message. The warning message is not a bounce message, but it is often formatted similarly to one and reads like one. This practice often causes the sender to believe that the message has not been delivered, when in fact the message will be delivered successfully at a later time.

 

When a mail server is grey listed, the duration of time between the initial delay and the re-transmission is variable. Some mail servers use a default of four hours, though most will retry sooner. Most open-source MTAs have retry rules set to attempt delivery after around fifteen minutes (Sendmail default is 0, 15, ..., Exim default is 0, 15, ..., Postfix default is 0, 16.6, ..., Qmail default is 0, 6:40, 26:40, ..., Courier default is 0, 5, 10, 15, 30, 35, 40, 70, 75, 80,...). Microsoft Exchange defaults to 0, 1, 2, 22, 42, 62 ...

Greylisting delays much of the mail from non-whitelisted mail servers - not just spam - until typical patterns of communication are recorded by the grey listing system. For best results, white listing should be used extensively. A static list of public servers worth being whitelisted can be found in the official repository.

 

Also, legitimate mail might not get delivered if the retry doesn't come within the time window the grey listing software uses, or if the retry comes from a different IP address than the original attempt. When the source of an email is a server farm or goes out through an anti-spam mail relay service, it is likely that on the retry a server other than the original server will make the next attempt. Since the IP addresses will be different, the recipient's server will fail to recognize that the two attempts are related and refuse the latest connection as well. This can continue until the message ages out of the queue if the number of servers is large enough. Such server farming techniques can be construed as breaking RFCs detailed above since the original sending machine has absolved itself of the responsibility of mail delivery by tossing it back into the pool, which breaks the state of the mail delivery process. This problem can partially be bypassed by identifying and whitelisting such server farms in advance. However, it is not possible on a distributed network the size of the Internet to maintain a complete list of all such server farms.

 

Grey listing can be a particular nuisance with websites that require you to create an account and confirm your email address before you can begin using them. If the sending MTA of the site is poorly configured, greylisting may delay the initial email containing your signup confirmation link, thus introducing a waiting period even though the actual website may have attempted to send out your email confirmation code immediately.

 

Almost all stock configured Sendmail MTAs (sendmail being the most widely deployed MTA on the internet) will retry after a few minutes, leading to typical delays of under 10 minutes in most cases (still dependent on the greylisting configuration). Grey listing is particularly effective in many cases at weeding out misconfigured MTAs, and is gaining in popularity as a very effective anti-spam tool. Those MTAs that do not correctly handle grey listing will, by their very nature, become less numerous over time.

 

In order for grey listing to work for a particular domain, all backup mail servers (as specified by lower-priority MX records for the domain) must implement the greylisting policy as well.

Also, if certain details of the sending vary and the receiving MTA is not programmed to notice this, a message may be grey listed eternally and never delivered.

If you are experiencing mail delivery problems relating to greylisting, please contact our support team.