Updating mail receiving stack

Calling it a "stack" is probably too much, but fetchmail and procmail seem to go together and they are both subject for replacement for good reasons.

After finally deciding to replace fetchmail and procmail with something newer, came realization that there aren't that many choices out there and still quite a lot of people continue recommending fetchmail+procmail. This post is here to contribute to transferring to potentially more secure software.

What's wrong with procmail

Search for replacement was prompted by finding out that last maintainer of procmail recommends against using it (source):

Executive summary: delete the procmail port; the code is not safe and should not be used as a basis for any further work.

On top of that procmail's syntax has some limitations which make it hard to use.

What's wrong with fetchmail

Fetchmail is primarily criticised for actions its developer took or didn't take, which makes me wonder why didn't those critics just make a fork and be done with that... Anyway, current issues with fetchmail seem to be:

passwords in configuration file have to be in plain text
only MD5 fingerprints of SSL certificates are supported
syntax of configuration file isn't obvious (relatively minor issue, since you don't edit it everyday)

Replacing procmail

The closest alternative to procmail seems to be maildrop. And it's pretty much the only one around, it seems (although there is at least fdm, which is both mail retrieval and mail delivery agent at the same time, it also stores passwords in plain text).

Syntax of maildrop resembles that of procmail, supports most of it functionality and doesn't have limitations that syntax of procmail has. Compare for yourself:

# this is procmail

MAILDIR=$HOME/mail
LOGFILE=$HOME/.procmail-log

:0:
* ^To.*[vV]ifm.*
vifm

:0:
* ^Subject.*vifm.*
vifm

# this is maildrop

MAILDIR="$HOME/mail"
logfile "$HOME/.maildrop-log"

if ( /^To.*[vV]ifm.*/:h || /^Subject.*vifm.*/:h )
    to $MAILDIR/vifm

The syntax does differ, but fortunately there is helpful tutorial by Warren Block here.

Replacing fetchmail

More options here (wiki) and one of them is mpop. It's from the author of msmtp, which is quite popular project. So, if you're already using msmtp, this is a good choice since configuration format is pretty much the same.

mpop has good documentation, which explains how to use passwords encrypted with your gpg-key.

Compared to fetchmail mpop:

allows to use passwords from encrypted storage
verify SSL certificates against SHA1 (now deprecated) and SHA256 fingerprints
can be run in parallel to check multiple accounts at the same time
doesn't have daemon mode
doesn't write logs

Example of configuration conversion:

# this is fetchmail

set logfile "/home/user/.fetchmail-log"
set postmaster "user"
set bouncemail
set no spambounce
set softbounce
set properties ""
set daemon 300

poll server.net proto pop3
       user 'user@server.net' with pass 'password' is 'user' here ssl sslfingerprint '55:F5:81:51:91:CD:88:64:14:D5:AA:E2:D5:2E:2C:AB'
#      mda "/usr/bin/env procmail -f -"
       mda "/usr/bin/env maildrop"

# this is mpop

defaults

keep off
tls on
uidls_file ~/mail/.mpop/uidls/%U@%H.uids
delivery mda /usr/bin/env maildrop -f '%F'

account server
host server.net
user user@server.net
passwordeval gpg2 --no-tty -q -d ~/.mail-password.gpg
tls_fingerprint 30:2A:06:B8:CF:A8:5B:93:66:5A:44:66:E2:BB:84:05:FE:80:95:3F:5A:FE:D1:08:DB:3B:B0:0D:7C:42:B4:39

Daemon mode vs. manual retrieval

Being used to automatic retrieval of mail, absence of daemon mode in mpop matters. mpop site and documentation suggests using crond, but I decided to try manual mail retrieval and wrote this script:

#!/bin/bash

DECODE='use Encode qw(decode); binmode STDOUT, ":utf8"; print decode("MIME-HEADER", $_)'

export GPG_TTY="$(tty)"

# make sure that necessary private key is cached or there will be many prompts
if ! gpg2 -q -d ~/.mail-password.gpg; then
    echo 'FATAL ERROR: failed to retrieve key!' | less --clear-screen
    exit 1
fi

# append last log to full log and truncate it
touch ~/.maildrop-log
cat ~/.maildrop-log >> ~/.maildrop-fulllog
truncate --size=0 ~/.maildrop-log

# collect mailboxes in parallel
(

echo '======== Retrieving ========'
echo

cat \
    <( mpop server 2>&1 ) \
    <( echo ) \
    <( mpop server2 2>&1 ) \
    <( echo )

echo '======== New mail ========'
echo

perl -ne "$DECODE" < ~/.maildrop-log 2>&1 ) |&
less --clear-screen

The script does the following:

Sets up $GPG_TTY for gpg and decrypts one of passwords (or really anything that's encrypted with key(s) that protect e-mail passwords) to ensure relevant gpg-key is cached and you won't need to enter passphrase more than once.
Clears file with last log after saving its contents elsewhere.
Collects e-mails from all mailboxes in parallel.
Displays list of new e-mails with their subjects and where they were put.

For some reason (probably related to terminal initialization) gpg can fail if you run this script like xterm -e script, but everything is fine for xterm -e 'usleep 100000 && script' or when it's run from already running terminal (thus it works fine from mutt).

This approach enforces a different workflow: instead of sometimes seeing new mail right after answering previously received message you now can process e-mail in batches and do not get distracted by newly arrived e-mails while dealing with the previous ones.

// load comments

Some notes on computer stuff code tags rss about