Email Spam Filtering

What is the Spam problem, and what is BrightMail?

In recent years, unsolicited bulk emails (spam) have become an increasing problem. Many people are finding their inboxes flooded with increasingly offensive spam - perhaps 100 spam messages per useful message.

Dealing with this problem is quite tricky, because there is no objective definition of what precisely constitutes a spam email, and thus the problem of taking an email message and reliably categorising it as spam or non-spam is a hard problem, with no single perfect solution. However, this problem is receiving serious attention world-wide nowadays, and several systems are now available which use a variety of techniques to detect spam.

CSG have investigated several spam-detection techniques - SpamAssassin and Dspam in the past - and have now settled on a commercial anti-spam and anti-virus package called BrightMail from Symantec. BrightMail automatically downloads new anti-spam and anti-virus tests - updating itself every few minutes - and claims to largely solve the spam problem by detecting the bulk of spam mail - but not all - while having an astonishingly low 1-in-a-million false positive rate.

What does CSG's installation of BrightMail do?

On both Departmental mail servers the Exim mail server software now hands all email messages to BrightMail for checking before Exim's normal mail processing starts. BrightMail runs all its tests on the message and comes to a verdict.

BrightMail verdicts, and Exim's response to them, can be summarised as:

• BrightMail Verdict	Exim Response
  Non-spam	Leave unmarked; deliver normally
  Known virus	Delete it silently!
  Virus checking failure [rare]	Mark it by prefixing "[WARNING: NOT VIRUS CHECKED]" onto the Subject line; then deliver normally!
  Spam content - 99.9999% certain	Mark it by adding the headers X-BrightMail-Spam-Flag: YES and X-Spam-Flag: YES, then deliver normally
  Spam from a blocked (recently blacklisted) IP address	Mark it by adding the headers X-BrightMail-Spam-Blocked: YES and X-Spam-Flag: YES, then deliver normally
  Might be spam [rare]	Mark it by adding the headers X-BrightMail-Spam-Maybe: YES and X-Spam-Flag: YES, then deliver normally

Note that the X-Spam-Flag: YES header is added to maintain backwards compatibility with the previous SpamAssassin system.

Any of these headers may be used by your .forward file in order to handle spam specially, we'll cover this shortly.

Please understand clearly that this is all that BrightMail does. It scans messages, deletes viruses, and then (optionally) adds some of the headers as shown above. It does not delete spam email, store spam messages in a different place, register spam messages with off-site spam databases, or anything else like that.

So what happens to messages marked as Spam?

This is where you come in. BrightMail adds the X-BrightMail-* and X-Spam-* headers whether you like it or not. But unless you do something, messages marked as spam will still come straight into your Inbox mail folder (your .email file) just as they always did - albeit invisibly marked with the Spam headers. This document assumes that you are not happy with this - that you want the spam singled out and dealt with separately. Now, you could do this mail filtering either (portably) in the Exim mail server or (not portably) in the mail client (an increasing number of email clients provide some type or other of rule-based mail filtering to be done, but each differs significantly). This document only describes the Exim server-side approach - to do server-side mail filtering with Exim, you need to construct an Exim .forward file containing a suitable mail filtering rule.

Don't worry if you've never heard of a .forward file before - to aid people set up their spam filtering, we have dropped in a default .forward file for everyone who didn't already have one. Alternatively, if you want to read up on the Exim .forward file, we have written a general purpose guide describing Exim's forward file - you're welcome to read it for more information, but if all you care about is spam checking, read on.