DoC Computing Support Group


Email: Rule based mail filtering with the Exim .forward file

What is Mail Filtering?

Filtering is the general name given to setting up mail delivery rules so that some specific types of email are delivered into alternative mail folders rather than your Inbox. Many mail clients have mail filtering capabilities, but this document describes delivery time mail filtering, i.e. mail filtering that is invoked automatically at the point where a new email message is being delivered to you by our mail server software - Exim. Please note that the Dept has recently decided to migrate to the College Exchange email service over the next few months, if you have already been migrated to Exchange then this page is not for you - however our main Email Guide contains a section on Exchange, which includes a link to setting up Exchange-based server-side rules via Outlook.

What is Exim and how does it deliver email?

Exim is DoC's current mail server software and it that handles all aspects of delivering email in the Department. It is an open-source mail server with a great deal of configuration flexibility, written by Philip Hazel at the University of Cambridge. More information about it may be found here. Exim is set up on both Departmental mail servers, the two machines which comprise smtp.doc.ic.ac.uk and also on all the Departmental home directory servers (stork, crane, kestrel, osprey and lycan). But note the point about Exchange in the previous paragraph - this uses a completely different architecture.

When an email arrives at either of the Departmental mail servers from outside, the following happens:

  1. The message is automatically passed through a commercial Anti-virus program called Sophos for checking whether any of the message's attachments are viruses. Such messages are rejected early on, via the SMTP protocol. Similarly, messages containing certain executable filename extensions are immediately rejected.
  2. The message is automatically passed through to a commercial Anti-spam system called BrightMail for checking whether the message is spam. Messages detected as spam have some special X-BrightMail-Spam and X-Spam headers added for you to test on later.

  3. Then Exim's normal mail processing starts, expanding email aliases or local mailing list names into the corresponding sets of individual mail addresses. Some of these mail addresses may be offsite, and these are sent out across the Internet. Marked spam to offsite addresses will not be sent offsite, but simply discarded.
  4. Each of the remaining mail addresses now corresponds to an individual DoC user. Exim looks up which fileserver that user's home directory is stored on and forwards the message to that machine.
  5. The message is now received by a separate copy of Exim (quite likely a different version!) running on the appropriate home directory server.
  6. This becomes the user (for security reasons) and then checks whether or not the user has a safe .forward file in their home directory - one that can only be written to by the user, i.e. the permissions have been set using the command: chmod u=rw,go=r ~/.forward.
  7. If the user has a safe ~/.forward file, Exim now reads the file and then runs through the rules contained (which we describe in more detail below).
  8. If no rule accepts the message and marks it as delivered - or if there is no (safe) ~/.forward file to use - then Exim performs a normal email delivery, appending it to the end of the user's Inbox (the .email file) obeying sensible locking precautions.

What is the Exim .forward file?

The Exim ~/.forward file is the main vehicle for adding specific rules for how to process certain kinds of email. Originally, the .forward file used to be a very simple file that contained a comma-separated list of email addresses to forward the email to (or names of commands to pipe the email to to deliver it). While this style is still valid, Exim has generalised this into a pretty powerful programming language (defined by the Exim Filter Specification documentation), which essentially allows you to write a mail processing program - comprising sequences of condition/action blocks - to determine how an incoming message is delivered.

All DoC users should now have a ~/.forward file in their home directory. If by any chance you do not have such a file, please email help@doc and ask us for assistance in setting it up.

If you like to get stuck right in, the Exim filter documentation is very complete, and contains some excellent examples of .forward syntax.

Warning about editing your ~/.forward file

Your ~/.forward file is effectively a computer program written in a precise language, executed by the Exim mail server on home directory servers. All of this raises the possibility of errors in your ~/.forward file, which have rather a bad consequence:

  1. Exim will generate a mail delivery failure message to each person who emails you - i.e. to the Sender, not to you!
  2. Exim will then deliver the email to your .email file bypassing all your normal ~/.forward rules, so loads of spam etc will suddenly appear in your Inbox.

In consequence, we recommend the following:

  1. Always edit your ~/.forward on Linux, to ensure that Unix-style line endings are present (rather than Windows or Mac-style line endings). Exim treats non-Unix line endings as syntax errors.

  2. After each alteration to your ~/.forward file, you should run the program to test it. How? By sending yourself a test email. That way, you are the sender, so you receive the mail delivery error message if you made a mistake.

  3. If this happens, you should fix the problem immediately. The best thing is to be prepared beforehand - to have a previous known working copy of your ~/.forward file available, and to then move your broken ~/.forward to BROKEN_FORWARD for later debugging, and copy KNOWN_GOOD_FORWARD to ~/.forward. Only then do you try to debug the problem in your BROKEN_FORWARD.

It is possible to construct a valid ~/.forward file with rules causing a loop, this can cause significant disruption to the mail servers and is guaranteed to make us very cross with you!

If all this sounds a bit too complicated, we are very happy to assist you in setting up your ~/.forward file after discussing what you want to do. Contact us for assistance explaining clearly what you'd like to do.

Exim mail filtering rules - what are they?

An Exim ~/.forward file is essentially a sequence of (condition, action) rules. A rule can be an unconditional action, but more usually is a conditional rule (i.e. an if...then...elsif...else..endif. The conditions are full boolean expressions which typically compare predefined string-valued variables to literal strings (or Perl-style regular expressions). The string-valued variables you have access to typically correspond to email headers in the message being checked. The actions are things like deleting a message, saving a message to a file (appending it), piping it to an external command or forwarding it onto another email address. So, for example, the rule:

if $header_from: contains "@a.domain.that.i.hate"
then
   seen finish        # delete all emails from that email domain
endif

serves to delete all messages (and finish processing afterwards) that come from a particular email domain that I really dislike.

Why does seen finish mean delete? finish means cease processing rules, and the seen modifier marks the message as having been delivered, thus suppressing normal delivery.

Please see the Exim Filter Specification documentation for more details.

General structure of the Exim ~/.forward file

We suggest that all users' Exim .forward files should be of the following general structure:

# Exim filter  <<== do not edit or remove this line!
#
#       See the Exim filter document (http://www.exim.org/docs/filter.html)
#       for details of the Exim filter language used here.
#

#
#    1. Logging.  You can arrange for exim to log messages to a
#       log file in order to help you debug subtle .forward problems.
#
logfile $home/.exim_filter_log 0644

#
#       Now, add log write commands like the following line (currently
#       commented out) wherever you like in this file, either unconditionally
#       to log everything, or conditionally in an "if".
#
#logwrite "$tod_log msgid: $message_id, from: $header_from:, subject: $header_subject:"

#
#    2. Pre-Spam exceptions:
#
#       If you regularly receive emails that are mistakenly categorised as
#       spam, put a "deliver this kind of message normally" rule here.  eg:
#       if $h_From: contains "wibble@wobble" then unseen finish endif'
#

#
#    3. Spam filtering:
#
#       divert to spam folder if BrightMail thinks it's definitely spam,
#       from a blocked IP address, or maybe spam.  BrightMail also marks
#       it in the old spamassassin style, so you can simply use:
#
if $h_X-Spam-Flag: is "YES" then
        save IMAP/Spam
        finish
endif

#
#    4. Post-Spam, Pre-Error exceptions:
#
#       Some Imperial distribution lists have null return paths, so we
#       should force them to deliver normally.

if      $header_to: contains "-dl@imperial.ac.uk"
or      $header_cc: contains "-dl@imperial.ac.uk"
then
        unseen finish
endif

#
#    5. Error Message handling:
#
#       Deliver all marked email error messages direct to your inbox.
#
#       This used to be the very first rule but spammers are now sending
#       spam error messages to take advantage of this, so we recommend it
#       here, much later: after spam detection.
#
if error_message then unseen finish endif

#
#    6. Post-spam, post-error processing:
#
#       place any additional rules you like here.
#

#
#    7. When going on vacation:
#
#       Setup .vacation.msg; delete and then touch the vacation database,
#       i.e. rm ~/.vacation.{dir,pag}; touch ~/.vacation.{dir,pag}
#       and then uncomment the following line, changing my_mailname to your
#       long form mail name (eg. joe.bloggs08)
#
#unseen pipe "/usr/bin/vacation -a my_mailname $local_part"

Please note that this standard ~/.forward has changed in October 2008, so you might want to replace your previous standard ~/.forward with the following link:

There is a standard `~/.forward` file that we recommend which has the above overall structure, with the details filled in.

In this structure, after logging is set up, there are four main sections where you might insert rules:

  1. pre-spam: Rules you want before dealing with messages marked as spam (the pre-spam section).
  2. post-spam, pre-error: Rules you want before dealing with messages marked as errors, such as College distribution lists.
  3. post-error: Any rules you want after dealing with errors.
  4. vacation/out-of-office processing is always at the end (but commented out until it's needed). For instructions about activating and deactivating vacation processing see our Out-of-Office guide.

Where might you place the above delete messages from a domain that I hate rule? Most likely in the pre-spam section (especially as the rule is deleting messages); but perhaps in the post-spam, pre-error section.

Example: Forwarding mail offsite to one person

Here, you need to think whether you want a local copy of all email kept here at DoC (i.e. forward to someone offsite AND deliver to me normally) or whether you want no local copy here at DoC (i.e. just forward to someone offsite).

If you want the pure forwarding (with no local copy) behaviour, and are starting with a .forward file that is essentially the same as the standard one recommended above, then you should insert the rule:

deliver you@offsite.email.address

into the post-spam section (there's no point in forwarding spam!)

If you want forwarding with local copy behaviour, then the rule is:

unseen deliver you@offsite.email.address

Note that if you want pure mail forwarding for a significantly long period, it is more efficient to Contact us and let us know where to forward your email. This can bypass reading and parsing your .forward file entirely.

Example: Delivering all email to a single command via a pipe Use a command (in the post-spam section) of the form:

pipe "command_name"

If you want to send the email into pipe and still have a copy delivered normally into your Inbox, then the rule is:

unseen pipe "command_name"

One possible program to run via a pipe is procmail - which is an alternative to the Exim filtering system that some users prefer (but that we do not recommend because it has a tendency to occasional drop messages into /tmp on the home directory servers!).

Example: Subject or From based rules

See Exim's examples section for several such examples.

Example: Handling messages marked as spam See The Spam Guide for our discussion of spam filtering. You basically choose one of 3 basic rules and insert it into the above general structure.

Example: Setting up a vacation message

If you start with the basic .forward file as described in the general structure section, there's already a ready but commented out example of vacation in there, basically set up as an unseen pipe "vacation..." rule. For instructions about setting the vacation message, activating and deactivating vacation processing see our out-of-office guide

A worked example rule (gerbil-fanciers)

Here is a complete worked example of setting up (from scratch) a .forward file to do normally spam processing, and have a rule:

Suppose you are on a gerbil-fanciers mailing list, and that all messages from that list have the Sender: header field set to gerbil-fanciers-admin@mailman.god.help.us. This is sufficiently unusual that you might decide that you'd like all messages from that Sender to be stored in your IMAP Gerbils mail folder for easier deleting... sorry, I meant reading. Here's how you'd do that in Exim filter syntax:

#
# - divert gerbil-fanciers mailing list messages to IMAP/Gerbils
# my IMAP folder directory is ~/IMAP
#
if $header_sender: contains "gerbil-fanciers-admin@mailman.god.help.us"
then
   save IMAP/Gerbils
   finish
endif

To set up a complete .forward file, let's start with the recommended basic Exim filter as described in the general structure of an Exim filter file, and then add the above rule in a sensible place (let's say the post-spam location). Proceed as follows:

  • On a Linux machine (or using an ssh window to a Linux machine):
  • Using a web browser on Linux, download the standard basic .forward file - which is very similar to the version shown above, with a few details added.
  • Copy or rename your downloaded file into your home directory called .forward (if you downloaded it into your home directory, you'd use the command mv eximforward.txt .forward). Then check it's contents (less .forward).
  • Now, using your favourite Unix editor, edit the .forward file and insert the above rule into the chosen (post-spam) location, right after the post-spam comment section.
  • Make sure your .forward file is not writable by other users: chmod go-w .forward
  • Immediately send yourselves an email. Check you don't receive a mail delivery failure message (broken .forward file!).
  • If you got it right, the next time a gerbils-fanciers email arrives, it'll be appended to the end of the file ~/IMAP/Gerbils (of course, that IMAP directory must exist first for this to work, so do mkdir ~/IMAP right now if it doesn't already exist!). Your new Gerbils IMAP folder should become available when you next use your favourite IMAP-based email client (you need to tell some email clients to look for new IMAP folders, via a menu option like rescan or resubscribe).
 
 

guides/email/filtering (last edited 2009-06-17 16:43:47 by dcw)