procmail to IMAP

My employer provides an IMAP server for email. Historically, it behaved badly with Message-Id headers, overwriting the original (the first sin, breaking all threading), and overwriting it with a globally indistinct value. It also had limited filtering capabilities. So I avoided it, and eventually set up my own home IMAP server (using Dovecot, Postfix and Procmail). I still had to get emails forwarded from the employer-provided server, and was forced to use ‘forward as attachment’ to ensure that the message id was preserved. An earlier post was on how to extract the email attachment to a pipe for interfacing with Procmail, but the latest incarnation of the employer-provided server has a proper redirection facilty, so that's redundant for me now.

Problem

Running your own IMAP server at home is no fun, especially with a dynamic IP. You have to have a Dynamic DNS service, and hope no-one else gets your mail while it's re-syncing. I need the filtering with procmail, but it only delivers to mailboxes in mbox and Maildirs formats directly (and maybe one other…?); everything else has to be handled through an external command. Is there a beast that will deliver to an IMAP server? If so, I could redirect the mail via SMTP to my own host running procmail, and have it deposit the mail in specific folders back on the server via IMAP.

Stack Overflow has one question on procmail and delivering to an IMAP server?, and suggests the use of mailtool, which is part of the Courier mail server. However, I don't want to install all of Courier to get it, and an apt-file search can't seem to find it, so I suspect it's also ‘historical’.

Solution

Python to the rescue! (I hate Python syntax, by the way. Indentation for syntax is a step backwards, in my opinion. Still, it works…) Here's a small program pushimap, which uses imaplib to write a file to an IMAPS server:

#!/usr/bin/env python

import imaplib
import time
import ConfigParser
import os
import sys
import email.message
import email

import getopt
from pprint import pprint

if __name__ == '__main__':
    ## Parse arguments.
    cfg_name = os.environ['PUSHIMAP_CONFIG'] \
               if 'PUSHIMAP_CONFIG' in os.environ \
                  else os.path.expanduser('~/.config/pushimap/conf.ini')
    acc_name = os.environ['PUSHIMAP_ACCOUNT'] \
               if 'PUSHIMAP_ACCOUNT' in os.environ \
                  else 'default'
    msg_file = None
    mb_name = 'INBOX'
    flags = ''
    opts, args = getopt.getopt(sys.argv[1:], "f:a:d:s")
    for opt, val in opts:
        if opt == '-f':
            cfg_name = val
        elif opt == '-d':
            mb_name = val
        elif opt == '-a':
            acc_name = val
        elif opt == '-s':
            flags += ' \Seen'
    flags = flags[1:]
    if len(args) == 0:
        args.append('/dev/stdin')

    ## Read the account details.  Should include port too.
    config = ConfigParser.ConfigParser()
    config.read([os.path.expanduser(cfg_name)])
    hostname = config.get('account %s' % acc_name, 'hostname')
    username = config.get('account %s' % acc_name, 'username')
    password = config.get('account %s' % acc_name, 'password')

    ## Connect to the server.
    c = imaplib.IMAP4_SSL(hostname)
    try:
        ## Authenticate with the server.
        c.login(username, password)

        ## Process the plain arguments as filenames.
        for fn in args:
            if fn is None or fn == '':
                continue

            ## Read in the message.
            fp = open(fn, "r")
            try:
                msg = email.message_from_file(fp)
            finally:
                fp.close()

            ## Attempt to add the file's contents as a message, or
            ## create the folder and try again.
            created = False
            while True:
                typ, erk = c.append(mb_name, flags,
                                    imaplib.Time2Internaldate(time.time()),
                                    str(msg))
                if typ != 'NO':
                    sys.exit()
                if created:
                    sys.exit(1)
                typ, erk = c.create(mb_name)
                created = True
    finally:
        c.logout()

-f file specifies an INI file to hold credentials and the server address, and defaults to $PUSHIMAP_CONFIG, then to ~/.config/pushimap/conf.ini. The file should contain the likes of:

[account default]
hostname = imap.example.org
username = bloggsj
password = mind-your-own-sodding-business

-a acc specifies a section [account acc] to read fields from, and defaults to default.

-d mailbox specifies a mailbox on the server to add a message to, and defaults to INBOX. Some servers seem to use / (U+002F) as a folder-name separator, and others use . (U+002E, full stop). You could check by temporarily modifying the code to run typ, data = c.list() ; pprint(data).

-s says that messages should be flagged as ‘seen’, i.e., read.

Remaining arguments are filenames. If none are given, /dev/stdin is assumed. Each file is read as an mbox-formatted message (what will it do with an mbox with more than one message?), and appended to the specified mailbox. If that fails, an attempt is made to create the mailbox, then the appending operation is tried again. If that fails, the program exits with a non-zero status. This can be used with Procmail's W flag, in which the exit status of the piped command determines whether to continue filtering.

See imaplib - IMAP4 client library - Python Module of the Week for some example uses of imaplib. The c.list() call is useful in determining what the folder-name separator is.

Now let's Procmail it up:

PUSHIMAP_CONFIG = $HOME/.myimapstuff

:0
* ^List-Id:.*<some\.list\.example\.org>
{
  :0 W
  | pushimap -d "Work/My Employer/Mailing list"

  :0
  ".Work.My Employer.Mailing list/"
}

Caveats

• As a precaution, I'm keeping the old Maildirs action, but it's only done if the pushimap command fails.
  

• I used email.message_from_file in solving the earlier problem (extracting an attachment from a pipe), and it seemed to cope okay with fairly big messages, but I'm not sure. Ideally, it would transparently cache them on disc when they went over a threshold. It's quite possible that the script will simply terminate abnormally, so the :0 W flag is being relied upon to ensure some back-up form of delivery.