Rob Newman

Web Development, Realtime Databases, Geoscience Visualization

Click the box titles below to expand:

How To :: Antelope – Create an Antelope Mail Archive

NOTE: Updating from Antelope 4.7 to Antelope 4.8 broke the mail archive. See Addendum below for the fix....

One of Antelope's many tools developed by Kent Lindquist is the Mail Parser tool (man mail_parser). This tool essentially routes incoming emails to a series of customized Perl handlers. In this case, what we want to do is route the incoming mail messages to a Datascope database, which will store the emails and allow us to explore them in dbe, or through a web interface.

In addition to routing the email messages we also want to check for spam, and trash the email if it has been flagged as spam. We do this by first parsing the mail message through Procmail filters.

First things first, however. This how–to assumes that you have an account set up by your system administrator that has been created solely as an email archiver. Everything that we do in this how–to takes place in this accounts home directory. This how–to also assumes that you are very comfortable using the command line and understand command line tools like ls and cat.

0. Create a backup mail directory

First create a backup mail directory that will store ALL messages that come in to the account:

user@localhost 53> mkdir /path/to/useraccount/Backup_mail
user@localhost 54> cd /path/to/useraccount/Backup_mail
user@localhost 55> mkdir backupmail

1. Create a .forward file

First up, we need to create a .forward file. This is a standard UNIX method of applying various tools to your incoming mail messages. A typical use of a .forward file is to re–route email to another account, or to automatically reply to a mail message with the vacation program (man vacation).

So, lets have a look at the .forward file for the mail archive we are creating:

user@localhost 56> cat .forward
"|/bin/cat >> /path/to/useraccount/Backup_mail/backupmail"
"|IFS=' ' && exec /path/to/bin/procmail -f- || exit 75 #user"
myemailaddress@ucsd.edu

You can copy and paste this into a new .forward file, ensuring that all pathnames are correct for your host. The email address on the bottom line gets a copy of every message sent to the account where this .forward file is stored.

Details aside, this is essentially first making a backup of the email message and storing it in /path/to/useraccount/Backup_mail/backupmail and passing the message to procmail. Procmail is an autonomous mail processor (see man procmail) that reads the message recieved from stdin and then looks for a file .procmailrc. Procmail then applies whatever commands are in the rc file to the message. So we now need to create .procmailrc...

Note: Remember that you must need to create the back–up directory you specified on the first line (see step 0)!

2. Create a .procmailrc file

Using your favourite text editor, create the .procmailrc file:

user@localhost 59> cat .procmailrc
SHELL=/usr/bin/sh
VERBOSE=yes
LOGFILE=/path/to/useraccount/procfile_log
LOGABSTRACT=all

:0:
* ^X-Spam-Flag: Spam YES
Spam

:0w
|/path/to/useraccount/mail_parser_wrapper

Lets dissect this file a little. First up, we define which shell to use (sh), then if we wish to see verbose output (yes). Then we need to define the log file that procmail will output all its processing comments into. Then we tell procmail to log everything (LOGABSTRACT=all). Finally we tell procmail that if there is a header with X-Spam-Flag and the value is Spam YES then we mark the mail as Spam. This header is UCSD–centric and is a product of Spam Assassin — a third party application that runs on the UCSD mail server.

If the mail is not spam then we pass the mail message to the application mail_parser_wrapper, which we have not yet created.

Note: Some email utilities are picky about the permissions on .forward and .procmailrc — something to watch out for.

3. Create your own mail_parser_wrapper

Below is a copy of the contents of our working copy:

user@localhost 61> cat mail_parser_wrapper
#!/usr/bin/tcsh -f

cd ~
set ANTELOPE=/opt/antelope/4.7p
source $ANTELOPE/setup.csh
setenv PFPATH $ANTELOPE/data/pf:$HOME/data/pf:./pf:.
exec $ANTELOPE/bin/mail_parser

What does this all mean? First up we define the shell to use (the tcsh shell). Then the script changes directory to the users home directory (cd ~) and creates all the environmental variables for the Antelope toolkit applications to correctly execute. Following this the executable mail_parser is run. For it to know how to act on the mail message it needs a parameter file....

3. Create your own mail_parser parameter file

Ours is below:

user@localhost 69> cat mail_parser.pf
Handlers        &Tbl{
    &Arr{
        archive_dir     archive
        database        /path/to/useraccount/mail_archive_database
        handler mp_filemail
        schema  Mail1.3
        dirmode 0755
        filemode 0444
        sender  .*
        subject .*
    }
}
pf_revision_time        1090958639

This is a standard Antelope parameter file consisting of an ordered table of associative arrays. In it we define the archive directory (archive), the path to the email database (/path/to/useraccount/mail_archive_database), the handler name (mp_filemail, see man mp_filemail for more info) , the database schema (Mail1.3), permissions for the directories and files created, and two Perl regular-expressions specifying the mail-messages for which the handler should be applied (sender and subject respectively, with values of .* in our example).

4. Checking what we have created

You should now have the following files in your home directory of where you are running the mail archiver:

user@localhost 71> ls -la
total 37
-rw-r--r--   1 user   group         133 Aug 26 10:37 .forward
-rw-r--r--   1 user   group         154 Aug 26 10:39 .procmailrc
drwxrwsr-x   2 user   group         512 Aug 26 10:50 Backup_mail
-rw-r--r--   1 user   group         255 Aug 26 10:50 mail_parser.pf
-rwxr-xr-x   1 user   group         168 Aug 26 10:40 mail_parser_wrapper

5. Test the archiving system

Send an email to the account which is acting as the archive. Once the email is received the following files will be created (based on the example above):

user@localhost 72> ls -la
total 37
-rw-r--r--   1 user   group         133 Aug 26 10:37 .forward
-rw-r--r--   1 user   group         154 Aug 26 10:39 .procmailrc
drwxrwsr-x   2 user   group         512 Aug 26 10:50 Backup_mail
-rw-rw-r--   1 user   group          52 Aug 26 10:50 mail_archive_database
-rw-rw-r--   1 user   group         282 Aug 26 10:50 mail_archive_database.correspondents
-rw-rw-r--   1 user   group         420 Aug 26 10:50 mail_archive_database.in
drwxr-xr-x   3 user   group         512 Aug 26 10:50 archive
-rw-r--r--   1 user   group         255 Aug 26 10:50 mail_parser.pf
-rwxr-xr-x   1 user   group         168 Aug 26 10:40 mail_parser_wrapper
-rw-------   1 user   group         405 Aug 26 10:50 procfile_log

Items created the first time an email is processed are marked in red.

Looking in detail at these files we can see that the first file is just a standard database descriptor file:

user@localhost 73> cat mail_archive_database
# Datascope Database Descriptor file
schema Mail1.3

The next two files (mail_archive_database.correspondents & mail_archive_database.in) are the newly created database tables.

You can now open mail_archive_database in dbe and you should see something like*:

Opening up the correspondents table should show you something like*:

Opening up the in table should show you something like*:


* — database names and paths to files obscured due to security reasons

The final file (procfile_log) is the logged output of Procmail.

6. Wrapping up

That's it! You should now have a fully working email archiving system, which can be parsed, subsetted, etc, etc using all you favourite Antelope tools (dbverify, dbsubset). What I typically do with these databases is put them online so station operators can view, search and subset the archive through a web–interface, gleaning the information they need to plan installations, maintainence, etc, etc. Do what you will with your own versions!

Acknowledgements

Thanks to Kent Lindquist for looking over a first edition of this how–to and providing useful feedback.

Addendum (2006-03-27)

After a software upgrade to Antelope 4.8 the mail archiving system failed. I am still unsure what broke, but assume that it was something deep in the Antelope software structure. Updating the file mail_parser_wrapper to read:

set ANTELOPE=/opt/antelope/4.8

fixed the problem.