I’ll start with the crucial point: My interest in giving sendmail a Let’s Encrypt certificate (along with a secret key, of course) has nothing to do with security. The real reason is that some mail servers won’t deliver their mail to my server unless the link is encrypted. As of today (March 2026), I know only of one such case, but that’s enough for me to understand that I must at least support an opportunistic TLS upgrade for arriving mails. In other words, my mail server must allow STARTTLS for servers who want to drop a mail at my server.

The problem with servers that won’t play ball without STARTTLS is that the mail is lost, sometimes without the sender being notified. I discovered this issue when one of those confirm-your-email messages didn’t arrive, and I sent the sender’s tech support a complaint. To which they responded with the reason: My server didn’t support a STARTTLS upgrade.

So the goal is to reduce the risk of mail loss, nothing else. And the main concern is that mail will not be delivered in cases it would have before adding STARTTLS. For example, where a cleartext connection would have been OK, but the sides attempted and failed to initiate a STARTTLS encrypted session and then the connection is terminated altogether, no mail delievered.

I’m running sendmail 8.14.4 on a Debian 8 machine (yes, it’s really ancient, but don’t play around with a stable system).

As for sources of information, there are a lot of guides out there. For those preferring the horse’s mouth, there’s the README on configuration files at /usr/share/sendmail-cf/README. Look for the section about STARTTLS. An example configuration can be found at /usr/share/sendmail/examples/tls/starttls.m4.

And since this topic revolves around certificates, maybe check out another post of mine which attempts to explain this topic. And my own little tutorial on setting up sendmail.

Is encryption worth anything?

If a mail server refuses to talk with anyone unless a TLS link is established with a mutually verified certificate (both sides check each other), encryption indeed adds security. Otherwise, it’s quite pointless.

Let’s begin with accepting arriving emails: If a server agrees to cleartext connections by clients for dropping off mails, it’s up to the sender to decide the level of security. Even if the arriving connection is encrypted with TLS, that doesn’t mean that connection is secure. Surely, the server is requied to submit a certificate when setting up the TLS session, but did the client verify it? And it the verification failed, did it terminate the connection? If it didn’t, a simple man-in-the-middle attack allows an eavesdropper can come in the middle, feed the client with a cooked-up certificate, accept the email, and then relay this email to the real destination, this time with a proper TLS connection. The creates an illusion that the mail was transmitted securely.

As a receiver of this mail, you can’t be sure who’s on the other side without checking the client’s certificate. A lot of clients won’t supply a certificate, though (my own server included, more about this below).

As for sending emails, an eavesdropping server might pretend to be the destination (possibly by DNS poisoning of the MX record). In the simplest man-in-the-middle attack, the eavesdropper doesn’t allow STARTTLS, and the message is transmitted in cleartext. If someone bothers to look in the mail server’s logs, this can be detected. Alternatively, the eavesdropper might suggest STARTTLS and offer an invalid certificate. For example, a self-signed certificate might seem like an innocent mistake. If the sending server agrees to sending the email nevertheless, the attack is successful (but with a “verify=FAIL” in the logs, allowing spotting the attack, if verifications usually are successful).

So to be really secure all mail servers must insist on TLS and verify each other’s certificates, or else the mail doesn’t go through. At present, going this path with a public mail server means a lot of undelivered mails (from legit sources). In particular, insisting that the sender of the email offers a valid certificate is not going to end well.

The situation before I made any changes

With the default configuration, sendmail has no certificates available for use. With no certificates, I mean no certificates to identify itself, but also no root certificates that allow verifying other servers.

This doesn’t prevent my server from connecting to other servers with TLS for outbound mails. In the mail log, the relevant entry looks like this:

sm-mta: 6237AaWA022257: from=<my@address.com>, size=2503, class=0, nrcpts=1, msgid=<f11f0@address.com>, bodytype=8BITMIME, proto=ESMTP, daemon=IPv4-port-587, relay=localhost.localdomain [127.0.0.1]
sm-mta: 6237AaWA022257: Milter insert (1): header: DKIM-Signature: [ ... ]
sm-mta: STARTTLS=client, relay=mx.other.com., version=TLSv1/SSLv3, verify=FAIL, cipher=ECDHE-RSA-AES128-GCM-SHA256, bits=128/128
sm-mta: 6237AaWA022257: to=<friend@other.com>, ctladdr=<my@address.com> (1000/1000), delay=00:00:03, xdelay=00:00:03, mailer=esmtp, pri=122503, relay=mx.other.com. [128.112.34.45], dsn=2.0.0, stat=Sent (Ok: queued as )

Note the “VERIFY=fail” on the third row, discussed in length below. For sending email, the only drawback for not setting up anything encryption-related is that the server’s identity isn’t verified. Plus, my server didn’t send a certificate if it was asked to do so, but that’s quite usual.

So to the server receiving the email, everything is normal. My server, acting as a client, upgraded the connection to encrypted with STARTTLS, and went through with it. No problem at all.

Should I install root certificates?

In order to allow my server to prevent a man-in-the-middle attack for outbound email, I can install root certificates and make them available to sendmail by virtue of configuration parameters. I also need to configure sendmail to refuse anything else than a TLS with a verified server. Otherwise, it’s pointless, see above.

At the very least, I will need to update the root certificates often enough, so that new root certificates that are generally accepted are recognized by sendmail, and that expired root certificates are replaced by new ones.

And even if I do everything right, some mails will probably not go through because the destination mail server isn’t configured correctly. Or doesn’t support STARTTLS at all, just as my own didn’t, before the changes I describe here.

So clearly, no root certificates on my server. I want all emails to fail verification, so if I mistakenly enable some kind of enforcement, all deliveries will fail, and not one isolated case a couple of weeks after making the mistake.

Now to the practical part

In the existing installation, /etc/mail is where sendmail keeps its configuration file. So I created the /etc/mail/certs/ directory, and populated it with three files:

• my.pem: The certificate obtained from Let’s Encrypt.
• my.key: The secret key for which this certificate is made (or to be really accurate, the certificate is made for the public key that pairs with this key). Readable by root only (for security and to make sendmail happy).
• ca.pem: The intermediate certificate which completes the trust chain from my.pem to the root certificate (Let’s Encrypt’s entire chain consists of just three certificates, root included).

I was a bit sloppy in the description for my.pem, because I use bacme to obtain the certificate from Let’s Encrypt, and that script gives me a certificate file that contains both my.pem and ca.pem, concatenated. The script that separates these two into separate files is shown further below.

And then I added these rows to sendmail.mc:

define(`confCACERT_PATH', `/etc/mail/certs')dnl
define(`confCACERT', `/etc/mail/certs/ca.pem')dnl
define(`confSERVER_CERT', `/etc/mail/certs/my.pem')dnl
define(`confSERVER_KEY', `/etc/mail/certs/my.key')dnl

Note that with this setting, the server doesn’t supply any certificate when acting as a client (i.e. when submitting an outbound email), even if asked for it. To enable this, the confCLIENT_CERT and confCLIENT_KEY options need to be assigned. This option should not be used with Let’s Encrypt’s certificates, as discussed below.

Actually, my initial attempt was to add only the two last rows, defining confSERVER_CERT and confSERVER_KEY. As the certificate file I get from my renewal utility already contains both my own and intermediate certificates, why not give sendmail only this combo file and forget about CAs? I mean, this is how I do it with Apache’s web server!

That idea failed royally in two different ways:

• If confCACERT_PATH and confCACERT aren’t defined, sendmail will start, but won’t activate the STARTTLS option. Actually, even if both are defined as above, and ca.pem is empty, no STARTTLS. Raising the loglevel with this definition allowed sendmail to complain specifically about this:

  define(`confLOG_LEVEL', `14')dnl

• When I put an irrelevant certificate in ca.pem, sendmail activated STARTTLS, ignored ca.pem (which is fine, it doesn’t help) but presented only the first certificate in my.pem to the client connecting.

To put it simple, sendmail wants my own certificate in my.pem and the CA’s certificate(s) in ca.pem, and kindly asks me not to fool around. And I have no problem with this, as the intermediate certificate can’t be used by my server to verify other server’s certificates. So adding it doesn’t work against my decision that all attempts to verify certificates by my server will fail.

But this arrangement requires writing a little script to separate the certificates, which is listed below. As far as I understand, those using the mainstream certbot don’t have this problem, as it generates separate files.

Reloading the daemon

It says everywhere, that sendmail must be restarted after updating the certificates. Even though I have the impression that sendmail is designed to shut itself down properly and safely in response to a SIGTERM, and even more importantly, that it’s designed not to lose or duplicate any mails, I didn’t fancy the idea of sending the server the signal that is usually used when shutting down the entire computer.

Instead, it’s possible to send the server a SIGHUP, which makes the server reload its configuration files (except for sendmail.conf, I read somewhere?) and of course the certificates. It’s actually a quick shutdown and restart, so maybe the difference isn’t so great, but reloading is the way it was meant to be. And it’s easily done with this command:

# /etc/init.d/sendmail reload > /dev/null

Redirection to /dev/null silences output (suitable for cron jobs).

It works!

There are two ways to see that mails actually arrive with TLS. One is through the mail logs:

sm-mta: STARTTLS=server, relay=mail-lj1-f180.google.com [209.85.208.180], version=TLSv1/SSLv3, verify=FAIL, cipher=ECDHE-RSA-AES128-GCM-SHA256, bits=128/128

“STARTTLS=server” in the log, indicates that a client has connected with STARTTLS for inbound mail. A reminder from above, if it says “STARTTLS=client”, it’s the server that has connected to another one for outbound mail. And along with that, sendmail tells us how the verification of the other side went.

Even easier, the TLS session leaves its tracks in the relevant Received: header in the mail itself:

Received: from mail-lj1-f180.google.com (mail-lj1-f180.google.com
 [209.85.208.180])	by mx.server.com (8.14.4/8.14.4/Debian-8+deb8u2) with
 ESMTP id 6245JWdJ019568	(version=TLSv1/SSLv3
 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=FAIL)	for
 <me@server.com>; Wed, 4 Mar 2026 12:19:34 GMT

No need to be alarmed about the verify=FAIL part. It just indicates that my server failed to verify the certificates that Gmail sent, which is quite natural, as it has no root certificates to go with. Which, as mentioned above, is intentional. See below for more about the verify=FAIL thing.

If something goes wrong…

In order to obtain debug messages, increase the log level to 14 by adding this to sendmail.mc (and compile with make).

define(`confLOG_LEVEL', `14')dnl

This is the log output after a reload (with SIGHUP), ending with STARTTLS working fine:

sm-mta: restarting /usr/sbin/sendmail-mta due to signal
sm-mta: error: safesasl(/etc/sasl2/Sendmail.conf) failed: No such file or directory
sm-mta: error: safesasl(/etc/sasl/Sendmail.conf) failed: No such file or directory
sm-mta: starting daemon (8.14.4): SMTP+queueing@00:10:00
sm-mta: STARTTLS: CRLFile missing
sm-mta: STARTTLS=server, Diffie-Hellman init, key=1024 bit (1)
sm-mta: STARTTLS=server, init=1
sm-mta started as: /usr/sbin/sendmail-mta -Am -L sm-mta -bd -q10m

Except for the fact that there are no errors related to STARTTLS, it says init=1, which is the indication it works.

The complaints about files missing in /etc/sasl2/ can be ignored, as I don’t use any kind of authentication (i.e. asking the client for credentials).

Checking the server for real

The easiest way to test the mail server is with CheckTLS.

This tool connects to the server and attempts starting a STARTTLS session (but doesn’t send a mail). The tool shows the details of session with lots of details, along with clarifications on the meaning of those details. So no need to know all the technicalities to get an idea on how you’re doing. If CheckTLS says all is fine, it’s really fine. If it says otherwise, take the remarks seriously (but never mind if MTASTS and DANE aren’t tested and marked yellow).

But even more importantly, CheckTLS gives the details of the certificates that the server provides. This is a good way to verify that the correct certificates are used, and that the certificate chain is valid.

Note however that this only checks how the server behaves when receiving emails. So if something is wrong with how the server sends email, for example it offers a problematic client certificate, that will go undetected.

For a more do-it-yourself approach, the first thing is to check that the server offers STARTTLS:

$ nc localhost 25
220 mx.server.com ESMTP MTA; Wed, 4 Mar 2026 13:14:20 GMT
EHLO there.com
250-mx.server.com Hello localhost.localdomain [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-STARTTLS
250-DELIVERBY
250 HELP
^C

It’s easier to do this on the local machine, because some ISP block connections to port 25 (to avoid spamming from their IP addresses). Note that it’s necessary to type an “EHLO” command to get the server saying something.

All fine? Connect to the server (see “man s_client):

$ openssl s_client -connect localhost:25 -starttls smtp < /dev/null
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 C = US, O = Let's Encrypt, CN = R12
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = mx.server.com
verify return:1
---
Certificate chain
 0 s:CN = mx.server.com
   i:C = US, O = Let's Encrypt, CN = R12
 1 s:C = US, O = Let's Encrypt, CN = R12
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

[ ... ]

And then a lot of mumbo-jumbo follows. The certificates sent by the server are listed in the “Certificate chain” section. The part above it shows openssl’s attempts to validate the certification chain. In the case shown above, there was no root certificate available on the server, hence the “unable to get local issuer certificate” error.

The reason I focus on the “Certificate chain” section is that if a root certificate server is present on the computer that runs openssl, it is listed in the section above it. It will then look like this:

CONNECTED(00000005)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R12
verify return:1
depth=0 CN = mx.server.com
verify return:1
---
Certificate chain
 0 s:CN = mx.server.com
   i:C = US, O = Let's Encrypt, CN = R12
 1 s:C = US, O = Let's Encrypt, CN = R12
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---

So openssl is happy now, but it lists the root certificate which it picked from its own repository. That is a bit confusing when checking the server.

Notes:

• The exact same test can be run on port 587, if it’s open for connections.
• The reason for the </dev/null part is that openssl actually opens a netcat-like session, so /dev/null terminates the session right away.
• The number in parentheses after CONNECTED is just the file descriptor number of the TCP socket. Nothing really interesting, even though it happens to be different when the certificate chain is validated and when it’s not.
• Add the -showcerts flag to dump all certificates that the server sent, not just the first one. Certificates from the local machine that help validation but weren’t sent from server are not dumped. For example:

  openssl s_client -connect localhost:25 -starttls smtp -showcerts < /dev/null

  As shown on a different post of mine, this allows examining them closer by copying each certificate to a separate file, and going

  openssl x509 -in thecertificate.crt -text

• If the verfication of the cerficate fails, openssl establishes the session regardless. The status of this verification is indicated on the output line saying “Verify return code: 0 (ok)” for a successful verification, and another number code and message otherwise (this part isn’t shown above). It’s also possible to add the -verify_return_error flag, which causes openssl to abort the session if the certificate chain verification fails, and indicate that with an error exit code.

verify=FAIL? Is that a problem?

The short answer is, no (in my case).

For the longer answer, let’s start with a brief explanation on the establishment of the TLS session. Encrypted mail delivery, regardless of whether it’s initiated with a STARTTLS on port 25 or by connecting to port 587, is based upon the TLS protocol. This is exactly the same protocol used by web browsers when they establish an https connection.

According to the TLS protocol, the server (as in client / server) is required to send its certificate (along with intermediate certificates) in order to prove that it’s indeed the server of the domain requested (domain as in example.com). The purpose is to avoid a man-in-the-middle attack.

The server may request the client to send its certificate (if it has any) in order to identify itself. This might seem odd in a web browser / server connection, but is supported by the commonly used browsers: It’s possible to supply them with a client certificate, which may be used on websites that request such.

This is why the practical step for making a mail server support STARTTLS is to supply it with a certificate. This certificate is part of the TLS handshake. The thing is, that it’s verified by the client, not our server. So we don’t know if the verification was successful or not. The fact that the TLS connection was established doesn’t mean that the client was happy with the certificate it got. It might have continued setting up the encrypted link regardless. To compare with web browsers, they issue a scary warning when they fail to verify the web server’s certificate. But that’s how it is today. In the distant past, the common reaction to an unverified certificate was a slightly different icon in the browser’s address bar, nothing more.

Likewise, a mail client is likely to continue the TLS handshake if the verification of the server’s certificate fails. This is a reasonable choice in particular if the client initiated a STARTTLS session, but would have sent the email in cleartext if this option wasn’t availble: An encrypted session with the possibility of a man-in-the-middle attack is better than sending the mail in cleartext, might be a way to think about it. And when looking at tutorials on the Internet from the early 2000′s on how to set up a mail server, it’s quite often suggested to use self-signed certificates, such that can’t be validated no matter what. This was a reasonable idea before the Let’s Encrypt era, when a certificate was an expensive thing.

It’s somewhat amusing that the TLS protocol doesn’t include a mechanism for the client to say “listen, your certificate stinks, but I’ll continue anyhow”. It could have been useful for server maintainers, but I suppose crypto people didn’t even want to think about this possibility.

Now to the point: The “verify” part in the mail log (as well as in the related Received header row in arriving mails) indicates the result of verifying the other side’s certificate, if such was requested.

As for the meaning of this attribute, here’s a copy-paste from the relevant part in the README file mentioned above:

${verify} holds the result of the verification of the presented cert.
	Possible values are:
	OK	 verification succeeded.
	NO	 no cert presented.
	NOT	 no cert requested.
	FAIL	 cert presented but could not be verified,
		 e.g., the cert of the signing CA is missing.
	NONE	 STARTTLS has not been performed.
	TEMP	 temporary error occurred.
	PROTOCOL protocol error occurred (SMTP level).
	SOFTWARE STARTTLS handshake failed.

Recall from above that I deliberately didn’t give my mail server any root certificates, with the intention that all verifications of certificates will fail.

It’s important to distinguish between two cases:

• Outbound emails, my server acting as a client, STARTTLS=client in the log: In this case, the certificate is required by the TLS protocol. Had I provided the mail server with root certificates, anything but verify=OK would have indicated a problem, and could have been a reason to terminate the TLS session. As I’ve already mentioned, the reason I didn’t provide the root certificates is to ensure that my server won’t be this picky.
• Inbound emails, my server acting as server, STARTTLS=server in the log: The certificate isn’t required to establish the TLS connection, but the server is allowed to ask for it.

So there are two normal options:

• verify=FAIL, meaning that my side got a certificate (as a client in order to establish a TLS session, or as a server because it asked for it), and the other side submitted something in response. And the verification failed, which is normal when you don’t have a root certificate.
• verify=NO, meaning that the other side didn’t submit any certificate.

Had I supplied root certificates to my server, I should have seen verify=OK most of the time for STARTTLS=client, and possibly verify=NO for STARTTLS=server. Would this information help me? Would this help telling spam servers from legit ones? I doubt that.

It’s a bit questionable why my server asks for certificates it can’t verify in the STARTTLS=server case, but that’s the default setting, and I guess this is how normal servers behave. According to the README file, setting the confTLS_SRV_OPTIONS to ‘V’ tells the server not to ask clients for certificates. Haven’t tried that personally, but I suppose one gets verify=NOT (as opposed to NO).

But what could be the point in asking the client for certificates? One possibility is that Sendmail has an access database, which defines rules for who is allowed to talk with the server. It’s possible to add rules related to TLS that depend on a successful verification of the certificate by making the rules depend on ${verify}, which is the macro containing the result of this verification (should be “OK”). The rules can of course also depend other attributes of the sender. This is more useful for relaying mails inside an organization. More about this in the README file.

To summarize, both verify=FAIL and verify=NO indicate no problem in my case, because the server has no way to validate the client’s certificate, and this validation isn’t necessary anyhow in my setting, as the server isn’t configured to refuse unverified partners by default, and I surely didn’t change that. It might have been a bit more elegant to set the confTLS_SRV_OPTIONS option to ‘V’ to spare clients the pointless task of sending a certificate that isn’t checked, but I stayed with the default configuration. I try to make a few changes as possible.

Should my server provide a certificate as a client?

Short answer, no, because Let’s Encrypt’s certificates don’t support TLS client authentication anymore.

Besides, this feature isn’t required, and opens for a possibility that weird things might happen if the other side doesn’t manage to verify my certificate for some reason. One could argue that sending certificates improves my server’s position as a non-spammer, and that it makes my server behave a bit more like Google’s. However I searched the web for indications that this would improve the spam score, and found none. So even if I had a certificate that allows client authentication, I wouldn’t use it. And the fact that Let’s Encrypt phased it out proves the point: I would most likely not have noticed the change, and my server would have sent an inadequate certificate, and now go figure why my mails aren’t delivered.

That said, it’s possible to configure Gmail (for business?) to accept mails only from servers that have presented a valid certificate.

Let’s get a bit technical about this: This is taken from the output of “openssl x509 -in thecertificate.crt -text” of a Let’s Encrypt certificate, issued in March 2026:

        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE

Compare with the same part on a certificate from Gmail, when acting as a server:

        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE

So far, the same “Extended Key Usage”. But when Gmail’s server identifies as a client, the relevant part is this:

        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE

So “TLS Web Client Authentication” is a thing, and it’s not wise to issue a certificate without this option when identifying as a client.

Fun fact, I dug up the certificate for the same server from October 2023 from a backup. And indeed, client authentication was enabled:

        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE

Not surprising, given the phase-out message from Let’s Encrypt from the link above. Otherwise, the old and new certificates are pretty much alike.

I’ve attached the printouts of both Google’s certificates below for reference. Anyhow, the main takeaways are:

• Even though it says “TLS Web Server Authentication”, it’s fine for mail servers. I would otherwise think “Web Server” refers to https. So Let’s Encrypt’s certificates are really legit for a mail server.
• When Gmail acts as a client, it indeed has the TLS Web Client Authentication option
• Don’t try using Let’s Encrypt’s certificates for client authentication.

The script splitting certificates

I promised the script that splits the PEM certificate file obtained by bacme from Let’s Encrypt into my own certificate and the intermediate certificate. I do such things in Perl, so here it is:

#!/usr/bin/perl
use warnings;
use strict;

local $/; # Slurp mode

my $cert = <>;

my @chunks = ($cert =~ /(-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----)/gs);

my $found = @chunks;

die("$0: Expected to find two certificates, found $found instead.\n")
  unless ($found == 2);

writefile("my.pem", "$chunks[0]\n");
writefile("ca.pem", "$chunks[1]\n");

exit(0);

sub writefile {
  my ($fname, $data) = @_;

  open(my $out, ">", $fname)
    or die "Can't open \"$fname\" for write: $!\n";
  print $out $data;
  close $out;
}

I run this as a regular user, not root, which is why I’m relatively sloppy with just writing out a couple of files in the current directory. Even though the hardcoded filenames makes this rather safe anyhow.

This script is given the combined PEM file through standard input (or with the file name as the first argument), and emits the two PEM files to the current directory. Deliberately unsophisticated, and deliberately very picky about the existence of exactly two certificates in the input, so I get notified if something in Let’s Encrypt’s setting suddenly changes.

For example, in the old Let’s Encrypt certificate from 2023 I mentioned above, there were three certificates. The third certificate affirmed ISRG Root X1 with the help of DST Root CA X3, the latter considered the root certificate at the time. The former is nowadays an established root certificate by itself, hence a third certificate unnecessary. But it can change, and if it does, I suppose the solution will be to concatenate everything but the first certificate into ca.pem. And if that happens, I want to be aware of the change and verify that the server gives the correct intermediate certificates.

Summary

After all said and done, I could have just split the certificate from Let’s Encrypt into two as shown above, and added the sendmail configuration options mentioned everywhere on tutorials, and everything would have been just fine. And had I used certbot, like everyone else, I would have had the ready-to-used certificate files directly.

As it turns out, there was no real need to delve into the details. Sendmail does the right thing anyhow. But understanding what’s going on under the hood is still better, and worth the effort, I think. In particular with a crucial component like sendmail.

Appendix: The Gmail’s client certificate

As it was quite difficult to obtain this certificate (running tcpdump on my server, feeding the result to Wireshark, exporting the certificate as raw bytes and opening the file with openssl as DER), I thought I’d show its printout:

    Data:
        Version: 3 (0x2)
        Serial Number:
            2b:64:a8:5a:82:a3:d2:c2:10:5b:9b:25:ab:75:c1:af
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Google Trust Services, CN = WR4
        Validity
            Not Before: Feb 10 17:58:28 2026 GMT
            Not After : May 11 17:58:27 2026 GMT
        Subject: CN = smtp.gmail.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:9e:75:cf:b1:84:c9:a8:f2:bb:c8:89:fe:ef:09:
                    ad:71:d7:2a:1e:e2:b0:51:e2:0b:d5:b9:a7:52:70:
                    e8:c1:ff:5b:60:b6:7c:65:c0:b1:8b:90:cb:cd:ab:
                    0c:da:ef:10:8f:17:79:ed:a5:b9:95:57:f2:28:f2:
                    da:3d:d3:1d:ed:03:a2:6f:88:da:7f:0c:cc:b9:f4:
                    f6:44:ac:bc:fa:95:62:c0:7b:31:8d:44:9c:3f:bf:
                    cf:05:66:8b:a2:7d:9a:dd:af:2b:dc:05:16:b8:37:
                    3c:1f:c5:23:9f:4d:2b:15:a4:97:87:ab:a7:70:3a:
                    4a:5d:2a:8d:d4:21:1a:68:48:da:74:89:6e:1a:27:
                    2f:ef:06:4b:38:b5:65:5f:c4:da:49:96:c5:4e:9f:
                    78:7f:cb:2b:6a:61:ff:f7:0f:f6:f3:d4:d0:7d:94:
                    84:a8:0c:21:8a:a2:a4:20:04:f7:83:ac:00:83:85:
                    eb:9e:01:7a:ea:a5:2a:b9:89:3b:ad:94:2d:c4:c1:
                    2f:49:86:17:52:f7:85:1a:97:76:9d:2f:cf:c4:20:
                    a3:9c:c5:7b:74:57:28:f2:35:d8:ab:fa:d8:53:b9:
                    ee:c9:24:cb:f3:aa:d4:0b:f9:1a:8e:3d:b9:ad:16:
                    7c:99:7c:40:ef:3f:25:5a:c7:94:87:e8:20:bb:19:
                    92:6f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                65:7C:AF:FE:54:FD:A3:0A:53:90:AB:9A:94:E7:AD:DF:DC:B9:8B:58
            X509v3 Authority Key Identifier:
                keyid:9B:C8:11:BC:3D:AA:36:B9:31:8C:4E:8F:44:D5:57:32:2F:C3:C0:61

            Authority Information Access:
                OCSP - URI:http://o.pki.goog/s/wr4/K2Q
                CA Issuers - URI:http://i.pki.goog/wr4.crt

            X509v3 Subject Alternative Name:
                DNS:smtp.gmail.com
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1

            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://c.pki.goog/wr4/F-WFK5nQurE.crl

            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 96:97:64:BF:55:58:97:AD:F7:43:87:68:37:08:42:77:
                                E9:F0:3A:D5:F6:A4:F3:36:6E:46:A4:3F:0F:CA:A9:C6
                    Timestamp : Feb 10 18:58:35.396 2026 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:BE:28:85:4E:52:7D:B5:FC:0C:C7:FA:
                                26:98:AE:D5:C4:86:E1:E1:70:A6:6A:3C:CA:CE:9E:21:
                                17:27:D4:09:BE:02:21:00:89:B7:00:57:51:76:41:FB:
                                D3:73:9B:27:FA:E1:40:2F:51:E1:4F:14:D1:65:18:EE:
                                81:C7:7C:A1:60:BA:6A:BF
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : CB:38:F7:15:89:7C:84:A1:44:5F:5B:C1:DD:FB:C9:6E:
                                F2:9A:59:CD:47:0A:69:05:85:B0:CB:14:C3:14:58:E7
                    Timestamp : Feb 10 18:58:35.450 2026 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:B0:B1:6E:A6:C2:1B:49:2A:28:2C:C9:
                                EC:AE:C6:F9:F4:EC:89:64:AC:88:6A:BE:08:86:09:36:
                                17:66:63:49:D0:02:20:5C:CE:E6:21:C3:21:88:15:E1:
                                D9:17:13:D6:0B:E3:F6:54:71:58:C9:55:9F:DA:14:63:
                                F8:69:F1:BC:DD:4B:32
    Signature Algorithm: sha256WithRSAEncryption
         8f:fa:cf:2b:ab:6a:66:07:2a:32:ae:15:39:c8:bf:a6:22:e1:
         b1:55:6d:1f:04:26:4b:34:54:fe:91:cd:61:92:6c:b1:2a:8b:
         47:81:28:84:ee:d1:b7:c2:fc:da:81:fd:74:c4:bf:6e:ba:f1:
         ef:b2:81:77:f1:0b:80:73:78:e1:86:1f:92:c8:92:a7:45:e6:
         26:93:4d:92:a2:2b:d2:02:db:1c:b8:81:4e:56:79:bc:4a:f6:
         8c:6c:f3:2a:a8:09:b2:5f:c2:74:bb:2d:74:0b:ea:3a:50:e7:
         dd:33:61:fa:ed:df:6c:ed:6e:ba:50:8c:54:9d:19:76:03:1b:
         56:7e:55:be:ee:3f:a3:c5:d6:ad:6b:fc:1b:43:ce:aa:50:52:
         af:f6:83:f0:38:f5:62:8d:0b:91:f3:72:f1:b7:10:64:1a:ca:
         02:97:8e:f9:13:a3:5d:1a:1b:ee:5d:01:dd:b0:48:f2:f3:30:
         cf:8d:6a:98:21:8d:83:23:38:c7:80:22:59:97:f0:45:76:fb:
         8c:a9:4e:f8:37:38:de:ba:4e:94:c5:1f:b1:d0:3c:87:69:11:
         ea:90:0d:75:72:82:5a:a3:c3:99:c6:e5:ce:57:05:ed:63:a9:
         2e:20:ab:b6:41:8c:53:e1:92:5c:55:de:bf:3b:d1:d3:ec:08:
         a8:87:9e:c0
-----BEGIN CERTIFICATE-----
MIIFMjCCBBqgAwIBAgIQK2SoWoKj0sIQW5slq3XBrzANBgkqhkiG9w0BAQsFADA7
MQswCQYDVQQGEwJVUzEeMBwGA1UEChMVR29vZ2xlIFRydXN0IFNlcnZpY2VzMQww
CgYDVQQDEwNXUjQwHhcNMjYwMjEwMTc1ODI4WhcNMjYwNTExMTc1ODI3WjAZMRcw
FQYDVQQDEw5zbXRwLmdtYWlsLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAJ51z7GEyajyu8iJ/u8JrXHXKh7isFHiC9W5p1Jw6MH/W2C2fGXAsYuQ
y82rDNrvEI8Xee2luZVX8ijy2j3THe0Dom+I2n8MzLn09kSsvPqVYsB7MY1EnD+/
zwVmi6J9mt2vK9wFFrg3PB/FI59NKxWkl4erp3A6Sl0qjdQhGmhI2nSJbhonL+8G
Szi1ZV/E2kmWxU6feH/LK2ph//cP9vPU0H2UhKgMIYqipCAE94OsAIOF654Beuql
KrmJO62ULcTBL0mGF1L3hRqXdp0vz8Qgo5zFe3RXKPI12Kv62FO57skky/Oq1Av5
Go49ua0WfJl8QO8/JVrHlIfoILsZkm8CAwEAAaOCAlIwggJOMA4GA1UdDwEB/wQE
AwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIw
ADAdBgNVHQ4EFgQUZXyv/lT9owpTkKualOet39y5i1gwHwYDVR0jBBgwFoAUm8gR
vD2qNrkxjE6PRNVXMi/DwGEwXgYIKwYBBQUHAQEEUjBQMCcGCCsGAQUFBzABhhto
dHRwOi8vby5wa2kuZ29vZy9zL3dyNC9LMlEwJQYIKwYBBQUHMAKGGWh0dHA6Ly9p
LnBraS5nb29nL3dyNC5jcnQwGQYDVR0RBBIwEIIOc210cC5nbWFpbC5jb20wEwYD
VR0gBAwwCjAIBgZngQwBAgEwNgYDVR0fBC8wLTAroCmgJ4YlaHR0cDovL2MucGtp
Lmdvb2cvd3I0L0YtV0ZLNW5RdXJFLmNybDCCAQUGCisGAQQB1nkCBAIEgfYEgfMA
8QB3AJaXZL9VWJet90OHaDcIQnfp8DrV9qTzNm5GpD8PyqnGAAABnEjrcQQAAAQD
AEgwRgIhAL4ohU5SfbX8DMf6Jpiu1cSG4eFwpmo8ys6eIRcn1Am+AiEAibcAV1F2
QfvTc5sn+uFAL1HhTxTRZRjugcd8oWC6ar8AdgDLOPcViXyEoURfW8Hd+8lu8ppZ
zUcKaQWFsMsUwxRY5wAAAZxI63E6AAAEAwBHMEUCIQCwsW6mwhtJKigsyeyuxvn0
7IlkrIhqvgiGCTYXZmNJ0AIgXM7mIcMhiBXh2RcT1gvj9lRxWMlVn9oUY/hp8bzd
SzIwDQYJKoZIhvcNAQELBQADggEBAI/6zyuramYHKjKuFTnIv6Yi4bFVbR8EJks0
VP6RzWGSbLEqi0eBKITu0bfC/NqB/XTEv2668e+ygXfxC4BzeOGGH5LIkqdF5iaT
TZKiK9IC2xy4gU5WebxK9oxs8yqoCbJfwnS7LXQL6jpQ590zYfrt32ztbrpQjFSd
GXYDG1Z+Vb7uP6PF1q1r/BtDzqpQUq/2g/A49WKNC5HzcvG3EGQaygKXjvkTo10a
G+5dAd2wSPLzMM+NapghjYMjOMeAIlmX8EV2+4ypTvg3ON66TpTFH7HQPIdpEeqQ
DXVyglqjw5nG5c5XBe1jqS4gq7ZBjFPhklxV3r870dPsCKiHnsA=
-----END CERTIFICATE-----

For comparison, this is the certificate obtained on the same day, when Gmail responded as a server:

    Data:
        Version: 3 (0x2)
        Serial Number:
            e9:b6:68:79:fa:91:bf:49:10:8d:b9:1e:cc:e8:63:b0
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Google Trust Services, CN = WR2
        Validity
            Not Before: Feb  2 08:37:58 2026 GMT
            Not After : Apr 27 08:37:57 2026 GMT
        Subject: CN = mx.google.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:55:ba:49:43:8f:d9:72:9d:f9:d0:fa:1c:76:ec:
                    73:44:39:69:e7:21:68:49:1f:d0:0e:c4:70:bb:1f:
                    61:15:71:58:a7:44:df:bd:9f:d5:f6:e9:d1:8a:77:
                    73:79:ac:82:e7:30:88:53:95:62:ff:f3:cd:32:71:
                    9e:68:21:a7:62
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                51:B6:13:35:D8:FB:85:27:72:70:77:EE:D7:5B:1D:06:5E:63:FD:51
            X509v3 Authority Key Identifier:
                keyid:DE:1B:1E:ED:79:15:D4:3E:37:24:C3:21:BB:EC:34:39:6D:42:B2:30

            Authority Information Access:
                OCSP - URI:http://o.pki.goog/wr2
                CA Issuers - URI:http://i.pki.goog/wr2.crt

            X509v3 Subject Alternative Name:
                DNS:mx.google.com, DNS:smtp.google.com, DNS:aspmx.l.google.com, DNS:alt1.aspmx.l.google.com, DNS:alt2.aspmx.l.google.com, DNS:alt3.aspmx.l.google.com, DNS:alt4.aspmx.l.google.com, DNS:gmail-smtp-in.l.google.com, DNS:alt1.gmail-smtp-in.l.google.com, DNS:alt2.gmail-smtp-in.l.google.com, DNS:alt3.gmail-smtp-in.l.google.com, DNS:alt4.gmail-smtp-in.l.google.com, DNS:gmr-smtp-in.l.google.com, DNS:alt1.gmr-smtp-in.l.google.com, DNS:alt2.gmr-smtp-in.l.google.com, DNS:alt3.gmr-smtp-in.l.google.com, DNS:alt4.gmr-smtp-in.l.google.com, DNS:mx1.smtp.goog, DNS:mx2.smtp.goog, DNS:mx3.smtp.goog, DNS:mx4.smtp.goog, DNS:aspmx2.googlemail.com, DNS:aspmx3.googlemail.com, DNS:aspmx4.googlemail.com, DNS:aspmx5.googlemail.com, DNS:gmr-mx.google.com
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1

            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://c.pki.goog/wr2/oQ6nyr8F0m0.crl

            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : D1:6E:A9:A5:68:07:7E:66:35:A0:3F:37:A5:DD:BC:03:
                                A5:3C:41:12:14:D4:88:18:F5:E9:31:B3:23:CB:95:04
                    Timestamp : Feb  2 09:38:00.395 2026 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:20:21:B9:8B:BD:E8:4E:B3:F4:24:46:6B:25:
                                17:CF:53:2E:2E:B7:83:A3:F5:DB:7B:F7:91:70:62:A2:
                                D5:74:B8:20:02:21:00:C9:3D:D4:79:5C:05:59:7C:68:
                                ED:6F:EA:45:59:55:D5:A6:9B:F8:9B:A3:62:AD:8B:2B:
                                30:A0:CC:4A:62:A1:EB
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 96:97:64:BF:55:58:97:AD:F7:43:87:68:37:08:42:77:
                                E9:F0:3A:D5:F6:A4:F3:36:6E:46:A4:3F:0F:CA:A9:C6
                    Timestamp : Feb  2 09:38:00.184 2026 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:3A:11:AE:85:B9:06:AF:A9:EF:88:25:64:
                                EB:2A:F3:4B:07:50:AF:B9:63:0F:4C:7A:B0:13:F4:CA:
                                0E:58:55:B7:02:20:50:81:1C:CF:06:47:39:AF:8A:F3:
                                27:00:78:34:FD:40:3F:1E:36:E3:2E:42:08:8E:14:B0:
                                09:B0:CA:CE:FD:B9
    Signature Algorithm: sha256WithRSAEncryption
         5e:bf:fc:22:aa:45:d9:35:37:c7:f3:9b:95:5a:e1:eb:2d:72:
         70:ba:ea:c5:ce:10:2e:53:b6:da:f0:54:77:f4:f4:7d:43:df:
         ff:fe:45:18:f3:cb:85:1c:ae:df:0d:a3:10:f1:01:7a:6f:81:
         03:af:c8:1c:d9:26:2b:4d:69:c1:4a:ef:bf:e2:98:cb:a8:c6:
         42:fe:78:4f:d9:82:d9:2c:39:fc:3e:d3:c2:6f:de:b8:e6:dc:
         82:51:04:00:0d:13:1d:2b:0e:fd:2f:56:7c:bf:73:a6:35:46:
         85:12:99:99:1f:1e:cb:9c:a5:e3:64:7f:b0:66:45:f5:ba:97:
         f0:ac:88:41:7e:c7:b0:7d:7f:04:15:c6:8b:0f:58:cd:19:1e:
         fb:b2:8c:f4:a6:dd:7f:8c:84:98:12:49:60:1b:20:c8:14:da:
         b1:fe:11:06:09:be:92:6b:cc:33:cd:e1:93:7c:bd:ca:1c:c9:
         70:71:cf:46:60:6c:db:22:72:9c:0d:00:e0:6a:72:bc:32:13:
         11:f0:8d:2f:95:d5:d9:20:76:9b:86:dd:73:10:8f:fc:a9:51:
         de:1c:90:d2:c8:a6:f9:ff:ab:a9:a8:5f:75:56:ae:a9:25:6a:
         7f:37:ff:67:5e:53:4e:2b:b7:c0:72:3c:9c:1b:68:f9:9a:0a:
         ef:60:6f:f2
-----BEGIN CERTIFICATE-----
MIIGxDCCBaygAwIBAgIRAOm2aHn6kb9JEI25HszoY7AwDQYJKoZIhvcNAQELBQAw
OzELMAkGA1UEBhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczEM
MAoGA1UEAxMDV1IyMB4XDTI2MDIwMjA4Mzc1OFoXDTI2MDQyNzA4Mzc1N1owGDEW
MBQGA1UEAxMNbXguZ29vZ2xlLmNvbTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IA
BFW6SUOP2XKd+dD6HHbsc0Q5aechaEkf0A7EcLsfYRVxWKdE372f1fbp0Yp3c3ms
gucwiFOVYv/zzTJxnmghp2KjggSvMIIEqzAOBgNVHQ8BAf8EBAMCB4AwEwYDVR0l
BAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUUbYTNdj7hSdy
cHfu11sdBl5j/VEwHwYDVR0jBBgwFoAU3hse7XkV1D43JMMhu+w0OW1CsjAwWAYI
KwYBBQUHAQEETDBKMCEGCCsGAQUFBzABhhVodHRwOi8vby5wa2kuZ29vZy93cjIw
JQYIKwYBBQUHMAKGGWh0dHA6Ly9pLnBraS5nb29nL3dyMi5jcnQwggKGBgNVHREE
ggJ9MIICeYINbXguZ29vZ2xlLmNvbYIPc210cC5nb29nbGUuY29tghJhc3BteC5s
Lmdvb2dsZS5jb22CF2FsdDEuYXNwbXgubC5nb29nbGUuY29tghdhbHQyLmFzcG14
LmwuZ29vZ2xlLmNvbYIXYWx0My5hc3BteC5sLmdvb2dsZS5jb22CF2FsdDQuYXNw
bXgubC5nb29nbGUuY29tghpnbWFpbC1zbXRwLWluLmwuZ29vZ2xlLmNvbYIfYWx0
MS5nbWFpbC1zbXRwLWluLmwuZ29vZ2xlLmNvbYIfYWx0Mi5nbWFpbC1zbXRwLWlu
LmwuZ29vZ2xlLmNvbYIfYWx0My5nbWFpbC1zbXRwLWluLmwuZ29vZ2xlLmNvbYIf
YWx0NC5nbWFpbC1zbXRwLWluLmwuZ29vZ2xlLmNvbYIYZ21yLXNtdHAtaW4ubC5n
b29nbGUuY29tgh1hbHQxLmdtci1zbXRwLWluLmwuZ29vZ2xlLmNvbYIdYWx0Mi5n
bXItc210cC1pbi5sLmdvb2dsZS5jb22CHWFsdDMuZ21yLXNtdHAtaW4ubC5nb29n
bGUuY29tgh1hbHQ0Lmdtci1zbXRwLWluLmwuZ29vZ2xlLmNvbYINbXgxLnNtdHAu
Z29vZ4INbXgyLnNtdHAuZ29vZ4INbXgzLnNtdHAuZ29vZ4INbXg0LnNtdHAuZ29v
Z4IVYXNwbXgyLmdvb2dsZW1haWwuY29tghVhc3BteDMuZ29vZ2xlbWFpbC5jb22C
FWFzcG14NC5nb29nbGVtYWlsLmNvbYIVYXNwbXg1Lmdvb2dsZW1haWwuY29tghFn
bXItbXguZ29vZ2xlLmNvbTATBgNVHSAEDDAKMAgGBmeBDAECATA2BgNVHR8ELzAt
MCugKaAnhiVodHRwOi8vYy5wa2kuZ29vZy93cjIvb1E2bnlyOEYwbTAuY3JsMIIB
AwYKKwYBBAHWeQIEAgSB9ASB8QDvAHYA0W6ppWgHfmY1oD83pd28A6U8QRIU1IgY
9ekxsyPLlQQAAAGcHbdWSwAABAMARzBFAiAhuYu96E6z9CRGayUXz1MuLreDo/Xb
e/eRcGKi1XS4IAIhAMk91HlcBVl8aO1v6kVZVdWmm/ibo2KtiyswoMxKYqHrAHUA
lpdkv1VYl633Q4doNwhCd+nwOtX2pPM2bkakPw/KqcYAAAGcHbdVeAAABAMARjBE
AiA6Ea6FuQavqe+IJWTrKvNLB1CvuWMPTHqwE/TKDlhVtwIgUIEczwZHOa+K8ycA
eDT9QD8eNuMuQgiOFLAJsMrO/bkwDQYJKoZIhvcNAQELBQADggEBAF6//CKqRdk1
N8fzm5Va4estcnC66sXOEC5TttrwVHf09H1D3//+RRjzy4Ucrt8NoxDxAXpvgQOv
yBzZJitNacFK77/imMuoxkL+eE/ZgtksOfw+08Jv3rjm3IJRBAANEx0rDv0vVny/
c6Y1RoUSmZkfHsucpeNkf7BmRfW6l/CsiEF+x7B9fwQVxosPWM0ZHvuyjPSm3X+M
hJgSSWAbIMgU2rH+EQYJvpJrzDPN4ZN8vcocyXBxz0ZgbNsicpwNAOBqcrwyExHw
jS+V1dkgdpuG3XMQj/ypUd4ckNLIpvn/q6moX3VWrqklan83/2deU04rt8ByPJwb
aPmaCu9gb/I=
-----END CERTIFICATE-----

Note the long list of alternative names. I wasn’t sure if they are respected by mail servers as well, but here they are. I fetched this certificate from alt1.gmail-smtp-in.l.google.com, actually, and not “directly” from mx.google.com.

In this post, I suggest manipulating the Message IDs of outgoing mails, so that legit inbound replies to my mails are easily detected as non-spam. I also show how to do this with Thunderbird (Linux version 91.10.0, but it works with practically all versions, I believe).

Briefly about Message-ID

Each email should have a Message-ID header, which uniquely identifies this message. The value of this header should consist of a random string, followed by an ‘@’ and a string that represents the domain name (referred to as FQDN, Fully Qualified Domain Name). This is often the full domain name of the “From” header (e.g. gmail.com).

For example, an email generated by Gmail’s web client had Message-ID: <CAD8P7-R2OuJvGiuQ-0RQqgSSmDguwv1VdjHgQND4jMJxPc628w@mail.gmail.com>. A similar result (same FQDN) was obtained when sending from the phone. However, when using Thunderbird to send an email, only “gmail.com” was set as the FQDN.

Does the Message-ID matter?

Like anything related to email, there are a lot of actors, and each has its own quirks. For example, rspamd adds the spam score by 0.5, with the MID_RHS_NOT_FQDN rule, if the Message ID isn’t an FQDN. I’m not sure to which extent it checks that the FQDN matches the email’s From, but even if it does, it can’t be that picky, given the example I showed above in relation to gmail.com.

It’s quite rare that people care about this header. I’ve seen somewhere that someone sending mails from a work computer didn’t like that the name of the internal domain leaking.

All in all, it’s probably a good idea to make sure that the Message-ID header looks legit. Putting the domain from the From header seems to be a good idea to keep spam filters happy.

Why manipulate the Message-ID?

In an reply, the In-Reply-To header gets the value of the Message ID of the message replied to. So if a spam filter can identify that that the email is genuinely a reply to something I sent, it’s definitely not spam. It’s also a good idea to scan the References header too, in order to cover more elaborate scenarios when there are several people corresponding.

The rigorous way to implement this spam filtering feature is storing the Message IDs of all sent mails in some small database, and check for a match with the content of In-Reply-To of arriving mails. Possible, however daunting.

A much easier way is to change the FQDN part, so that it’s easily identifiable. This is unnecessary if you happen send emails with your own domain, as spam senders are very unlikely to add an In-Reply-To with a matching domain (actually, very few spam messages have an In-Reply-To header at all).

But for email sent through gmail, changing the FQDN to something unique is required to make a distinction.

Will this mess up things? I’m not sure any software tries to fully match the FQDN with the sender, but I suppose it’s safe to add a subdomain to the correct domain. I mean, if both “mail.gmail.com” and “gmail.com” are commonly out there, why shouldn’t “secretsauce.gmail.com” seem likewise legit to any spam filter that checks the message?

And by the way, as of August 2024, a DNS query for mail.gmail.com yields no address, neither for A nor MX. In other words, Gmail itself uses an invalid domain in its Message ID, so any other invented subdomain should do as well.

Changing the FQDN on Thunderbird

Click the hamburger icon, choose Preferences, and scroll down all the way (on the General tab) and click on Config Editor.

First, we need to find Thunderbird’s internal ID number for the mail account to manipulate.

To get a list of IDs, write “useremail” in the search text box. This lists entries like mail.identity.id1.useremail and their values. This listing allows making the connection between e.g. “id1″ and the email address related to it.

For example, to change the FQDN of the mail account corresponding to “id3″, add a string property (using the Config Editor). The key of this property is “mail.identity.id3.FQDN” and the value is something like “secretsauce.gmail.com”.

There is no need to restart Thunderbird. The change is in effect on the next mail sent, and it remains in the settings across restarts.

The need for this feature has been questioned, as was discussed here. So if any Thunderbird maintainer reads this, please keep this feature up and running.

A possible alternative approach

Instead of playing around with the Message-ID, it would be possible to add an entry to the References header (or add this header if there is none). The advantage of this way is that this can also be done by the MTA further down the delivery path, and it doesn’t alter anything that is already in place.

And since it’s an added entry, it can also be crafted arbitrarily. For example, it may contain a timestamp (epoch time in hex) and the SHA1 sum of a string that is composed by this timestamp and a secret string. This way, this proof of genuine correspondence is impossible to forge and may expire with time.

I haven’t looked into how to implement this in Thunderbird. Right now I’m good with the Message-ID solution.

There is a widespread belief, that in order to use git send-email with Gmail, there’s a need to subscribe to Google Cloud services and obtain some credentials. Or that a two-factor authentication (2fa) is required.

This is not the case, however. If Thunderbird can manage to fetch and send emails through Google’s mail servers (as well as other OAUTH2 authenticated mail services), there’s no reason why a utility won’t be able to do the same.

The subscription to Google’s services is indeed required if the communication with Google’s server must be done without human supervision. That’s the whole point with API keys. If a human is around when the mail is dispatched, there’s no need for any special measures. And it’s quite obvious that there’s a responsive human around when a patch is being submitted.

What is actually needed, is a client ID and a client secret, and these are indeed obtained by registering to Google’s cloud service (this explains how). But here’s the thing: Someone at Mozilla has already obtained these, and hardcoded them into Thunderbird itself. So there’s no problem using these to access Gmail with another mail client. It seems like many believe that the client ID and secret must be related to the mail account to access, and therefore each and every one has to obtain their own pair. That’s a mistake that has made a lot of people angry for nothing.

This post describes how to use git send-email without any further involvement with Google, except for having a Gmail account. The same method surely applies for other mail service providers that rely on OAUTH2, but I haven’t gotten into that. It should be quite easy to apply the same idea to other services as well however.

For this to work, Thunderbird must be configured to access the same email account. This doesn’t mean that you actually have to use Thunderbird for your mail exchange. It’s actually enough to configure the Gmail server as an outgoing mail server for the relevant account. In other words, you don’t even need to fetch mails from the server with Thunderbird.

The point is to make Thunderbird set up the OAUTH2 session, and then fetch the relevant piece of credentials from it. And take it from there with Google’s servers. Thunderbird is a good candidate for taking care of the session’s setup, because the whole idea with OAUTH2 is that the user / password session (plus possible additional authentication challenges) is done with a browser. Since Thunderbird is Firefox in disguise, it integrates the browser session well into its general flow.

If you want to use another piece of software to maintain the OAUTH2 session, that’s most likely possible, given that you can get its refresh token. This will also require obtaining its client ID and client secret. Odds are that it can be found somewhere in that software’s sources, exactly as I found it for Thunderbird. Or look at the https connection it runs to get an access token (which isn’t all that easy, encryption and that).

Outline of solution

All below relates to Linux Mint 19, Thunderbird 91.10.0, git version 2.17.1, Perl 5.26 and msmtp 1.8.14. But except for Thunderbird and msmtp, I don’t think the versions are going to matter.

It’s highly recommended to read through my blog post on OAUTH2, in particular the section called “The authentication handshake in a nutshell”. You’re going to need to know the difference between an access token and a refresh token sooner or later.

So the first obstacle is the fact that git send-email relies on the system’s sendmail to send out the emails. That utility doesn’t support OAUTH2 at the time of writing this. So instead, I used msmtp, which is a drop-in replacement for sendmail, plus it supports OAUTH2 (since version 1.8.13).

msmtp identifies itself to the server by sending it an access token in the SMTP session (see a dump of a sample session below). This access token is short-lived (3600 seconds from Google as of writing this), so it can’t be fetched from Thunderbird just like that. In particular because most of the time Thunderbird doesn’t have it.

What Thunderbird does have is a refresh token. It’s a completely automatic task to ask Google’s server for the access token with the refresh token at hand. It’s also an easy task (once you’ve figured out how to do it, that is). It’s also easy to get the refresh token from Thunderbird, exactly in the same way as getting a saved password. In fact, Thunderbird treats the refresh token as a password.

msmtp allows executing an arbitrary program in order to get the password or the access token. So I wrote a Perl script (oauth2-helper.pl) that reads the refresh token from a file and gets an access token from Google’s server. This is how msmtp manages to authenticate itself.

So everything relies on this refresh token. In principle, it can change every time it’s used. In practice, as of today, Google’s servers don’t change it. It seems like the refresh token is automatically replaced every six months, but even if that’s true today, it may change.

But that doesn’t matter so much. All that is necessary is that the refresh token is correct once. If the refresh token goes out of sync with Google’s server, a simple user / password session rectifies this. And as of now, than virtually never happens.

So let’s get to the hands-on part.

Install msmtp

Odds are that your distribution offers msmtp, so it can be installed with something like

# apt install msmtp

Note however that the version needs to be at least 1.8.13, which wasn’t my case (Linux Mint 19). So I installed it from the sources. To do that, first install the TLS library, if it’s not installed already (as root):

# apt install gnutls-dev

Then clone the git repository, compile and install:

$ GIT_SSL_NO_VERIFY=true git clone http://git.marlam.de/git/msmtp.git
$ cd msmtp
$ git checkout msmtp-1.8.14
$ autoreconf -i
$ ./configure
$ make && echo Success
$ sudo make install

The installation goes to /usr/local/bin and other /usr/local/ paths, as one would expect.

I checked out version 1.8.14 because later versions failed to compile on my Linux Mint 19. OAUTH2 support was added in 1.8.13, and judging by the commit messages it hasn’t been changed since, except for commit 1f3f4bfd098, which is “Send XOAUTH2 in two lines, required by Microsoft servers”. Possibly cherry-pick this commit (I didn’t).

Once everything has been set up as described below, it’s possible to send an email with

$ msmtp -v -t < ~/email.eml

The -v flag is used only for debugging, and it prints out the entire SMTP session.

The -t flag tells msmtp to fetch the recipients from the mail’s own headers. Otherwise, the recipients need to be listed in the command line, just like sendmail. Without this flag or recipients, msmtp just replies with

msmtp: no recipients found

The -t flag isn’t necessary with git send-email, because it explicitly lists the recipients in the command line.

The oauth2-helper.pl script

As mentioned above, Thunderbird has the refresh token, but msmtp needs an access token. So the script that talks with Google’s server and grabs the access token can be downloaded from its Github repo. Save it, with execution permission to /usr/local/bin/oauth2-helper.pl (or whatever, but this is what I assume in the configurations below).

Some Perl libraries may be required to run this script. On a Debian-based system, the packages’ names are  probably something like libhttp-message-perl, libwww-perl and libjson-perl.

It’s written to access Google’s token server, but can be modified easily to access a different service provider by changing the parameters at its beginning. For other email providers, check if it happens to be listed in OAuth2Providers.sys.mjs. I don’t know how well it will work with those other providers, though.

The script reads the refresh token from ~/.oauth2_reftoken as a plain file containing the blob only. There’s an inherent security risk of having this token stored like this, but it’s basically the same risk as the fact that it can be obtained from Thunderbird’s credential files. The difference is the amount of security by obscurity. Anyhow, the reference token isn’t your password, and it can’t be derived from it. Either way, make sure that this file has a 0600 or 0400 permission, if you’re running on a multi-user computer.

The script caches the access token in ~/.oauth2_acctoken, with an expiration timestamp. As of today, it means that the script talks with the Google’s server once in 60 minutes at most.

Setting up config files

So with msmtp installed and the script downloaded into /usr/local/bin/oauth2-helper.pl, all that is left is configuration files.

First, create ~/.msmtprc as follows (put your Gmail username instead of mail.username, of course):

account default
host smtp.gmail.com
port 587
tls on
tls_starttls on
auth xoauth2
user mail.username
passwordeval /usr/local/bin/oauth2-helper.pl
from mail.username@gmail.com

And then change the [sendemail] section in ~/.gitconfig to

[sendemail]
        smtpServer = /usr/local/bin/msmtp

That’s it. Only that single line. It’s however possible to use smtpServerOption in the .gitconfig to add various flags. So for example, to get the entire SMTP session shown while sending the email, it should say:

[sendemail]
        smtpServer = /usr/local/bin/msmtp
        smtpServerOption = -v

But really, don’t, unless there’s a problem sending mails.

Other than that, don’t keep old settings. For example, there should not be a “from=” entry in .gitconfig. Having such causes a “From:” header to be added into the mail body (so it’s visible to the reader of the mail). This header is created when there is a difference between the “From” that is generated by git send-email (which is taken from the “from=” entry) and the patch’ author, as it appears in the patch’ “From” header. The purpose of this in-body header is to tell “git am” who the real author is (i.e. not the sender of the patch). So this extra header won’t appear in the commit, but it nevertheless makes the sender of the message look somewhat clueless.

So in short, no old junk.

Sending a patch

Unless it’s the first time, I suggest just trying to send the patch to your own email address, and see if it works. There’s a good chance that the refresh token from the previous time will still be good, so it will just work, and no point hassling more.

Actually, it’s fine to try like this even on the first time, because the Perl script will fail to grab the access token and then tell you what to do to fix it, namely:

• Make sure that Thunderbird has access to the mail account itself, possibly by attempting to send an email through Gmail’s server.
• Go to Thunderbird’s Preferences > Privacy & Security and click on Saved Passwords. Look for the account, where the Provider start with oauth://. Right-click that line and choose “Copy Password”.
• Create or open ~/.oauth2_reftoken, and paste the blob into that file, so it contains only that string. No need to be uptight with newlines and whitespaces: They are ignored.

And then go, as usual:

$ git send-email --to 'my@test.mail' 0001-my.patch

I’ve added the output of a successful session (with the -v flag) below.

Room for improvements

It would have been nicer to fetch the refresh token automatically from Thunderbird’s credentials store (that is from logins.json, based upon the decryption key that is kept in key4.db), but the available scripts for that are written in Python. And to me Python is equal to “will cause trouble sooner or later”. Anyhow, this tutorial describes the mechanism (in the part about Firefox).

Besides, it could have been even nicer if the script was completely standalone, and didn’t depend on Thunderbird at all. That requires doing the whole dance with the browser, something I have no motivation to get into.

A successful session

This is what it looks like when a patch is properly sent, with the smtpServerOption = -v line in .gitignore (so msmtp produces verbose output):

Send this email? ([y]es|[n]o|[q]uit|[a]ll): y
ignoring system configuration file /usr/local/etc/msmtprc: No such file or directory
loaded user configuration file /home/eli/.msmtprc
falling back to default account
Fetching access token based upon refresh token in /home/eli/.oauth2_reftoken...
using account default from /home/eli/.msmtprc
host = smtp.gmail.com
port = 587
source ip = (not set)
proxy host = (not set)
proxy port = 0
socket = (not set)
timeout = off
protocol = smtp
domain = localhost
auth = XOAUTH2
user = mail.username
password = *
passwordeval = /usr/local/bin/oauth2-helper.pl
ntlmdomain = (not set)
tls = on
tls_starttls = on
tls_trust_file = system
tls_crl_file = (not set)
tls_fingerprint = (not set)
tls_key_file = (not set)
tls_cert_file = (not set)
tls_certcheck = on
tls_min_dh_prime_bits = (not set)
tls_priorities = (not set)
tls_host_override = (not set)
auto_from = off
maildomain = (not set)
from = mail.username@gmail.com
set_from_header = auto
set_date_header = auto
remove_bcc_headers = on
undisclosed_recipients = off
dsn_notify = (not set)
dsn_return = (not set)
logfile = (not set)
logfile_time_format = (not set)
syslog = (not set)
aliases = (not set)
reading recipients from the command line
<-- 220 smtp.gmail.com ESMTP m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
--> EHLO localhost
<-- 250-smtp.gmail.com at your service, [109.186.183.118]
<-- 250-SIZE 35882577
<-- 250-8BITMIME
<-- 250-STARTTLS
<-- 250-ENHANCEDSTATUSCODES
<-- 250-PIPELINING
<-- 250-CHUNKING
<-- 250 SMTPUTF8
--> STARTTLS
<-- 220 2.0.0 Ready to start TLS
TLS session parameters:
    (TLS1.2)-(ECDHE-ECDSA-SECP256R1)-(CHACHA20-POLY1305)
TLS certificate information:
    Subject:
        CN=smtp.gmail.com
    Issuer:
        C=US,O=Google Trust Services LLC,CN=GTS CA 1C3
    Validity:
        Activation time: Mon 26 Sep 2022 11:22:04 AM IDT
        Expiration time: Mon 19 Dec 2022 10:22:03 AM IST
    Fingerprints:
        SHA256: 53:F3:CA:1D:37:F2:1F:ED:2C:67:40:A2:A2:29:C2:C8:E8:AF:9E:60:7A:01:92:EC:F0:2A:11:E8:37:A5:88:F3
        SHA1 (deprecated): D4:69:6E:59:2D:75:43:59:02:74:25:67:E7:57:40:E0:28:43:A8:62
--> EHLO localhost
<-- 250-smtp.gmail.com at your service, [109.186.183.118]
<-- 250-SIZE 35882577
<-- 250-8BITMIME
<-- 250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
<-- 250-ENHANCEDSTATUSCODES
<-- 250-PIPELINING
<-- 250-CHUNKING
<-- 250 SMTPUTF8
--> AUTH XOAUTH2 dXNlcj1lbGkuYmlsbGF1ZXIBYXV0aD1CZWFyZXIgeWEyOS5hMEFhNHhyWE1GM1gtOTJMVWNidjE4MFdVOBROENRcUdSbk5KaUFSY0VSckVaXzdzbDlHMTNpdFIyUTk0NjlKWG45aHVGLQVRBU0FSTVXJpSjRqMjBLcWh6WU9GekxlcU5BYVpFNUU4WXRhNjdLUXpCRm1HRDg3dFgzeHJ4amNPTnRVTkZFVWdESXhsUlcxOFhVT0pqQ1hPSlFwZlNGUUVqRHZMOWw4RExkTjlKZlNbGRTazNNbFNMNjVfQWFDZ1lLVVF2Y0luOWNSSUEwMTY2AQE=
<-- 235 2.7.0 Accepted
--> MAIL FROM:<mail.username@gmail.com>
--> RCPT TO:<test@mail.com>
--> RCPT TO:<mail.username@gmail.com>
--> DATA
<-- 250 2.1.0 OK m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
<-- 250 2.1.5 OK m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
<-- 250 2.1.5 OK m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
<-- 354  Go ahead m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
--> From: Eli Billauer <mail.username@gmail.com>
--> To: test@mail.com
--> Cc: Eli Billauer <mail.username@gmail.com>
--> Subject: [PATCH v8] Gosh! Why don't you apply this patch already!
--> Date: Sun, 30 Oct 2022 07:01:14 +0200
--> Message-Id: <20221030050114.49299-1-mail.username@gmail.com>
--> X-Mailer: git-send-email 2.17.1
--> 

[ ... email body comes here ... ]

--> --
--> 2.17.1
-->
--> .
<-- 250 2.0.0 OK  1667106108 m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
--> QUIT
<-- 221 2.0.0 closing connection m8-20020a7bcb88000000b003c6d21a19a0sm3316430wmi.29 - gsmtp
OK. Log says:
Sendmail: /usr/local/bin/msmtp -v -i test@mail.com mail.username@gmail.com
From: Eli Billauer <mail.username@gmail.com>
To: test@mail.com
Cc: Eli Billauer <mail.username@gmail.com>
Subject: [PATCH v8] Gosh! Why don't you apply this patch already!
Date: Sun, 30 Oct 2022 07:01:14 +0200
Message-Id: <20221030050114.49299-1-mail.username@gmail.com>
X-Mailer: git-send-email 2.17.1

Result: OK

Ah, and the fact that the access token can be copied from here is of course meaningless, as it has expired long ago.

Thunderbird debug notes

These are some random notes I made while digging in Thunderbird’s guts to find out what’s going on.

So this is Thunderbird’s official git repo. Not that I used it.

To get logging info from Thunderbird: Based upon this page, go to Thunderbird’s preferences > General and click the Config Editor button. Set mailnews.oauth.loglevel to All (was Warn). Same with mailnews.smtp.loglevel. Then open the Error Console with Ctrl+Shift+J.

The cute thing about these logs is that the access code is written in the log. So it’s possible to skip the Perl script, and use the access code from Thunderbird’s log. Really inconvenient, but possible.

The OAuth2 token requests is implemented in Oauth2.jsm. It’s possible to make a breakpoint in this module by through Tools > Developer Tools > Developer Toolbox, and once it opens (after requesting permission for external connection), go to the debugger.

Find Oauth2.jsm in the sources pane to the left (of the Debugger tab), under resource:// modules > sessionstore. Add a breakpoint in requestAccessToken() so that the clientID and consumerSecret properties can be revealed.

Sending a patch from Thunderbird directly

This is a really bad idea. But if you have Thunderbird, and need to send a patch right now, this is a quick, dirty and somewhat dangerous procedure for doing that.

Why is it dangerous? Because at some point, it’s easy to pick “Send now” instead of “Send later”, and boom, a junk patch is mailed to the whole world.

The problem with Thunderbird is that it makes small changes into the patch’ body. So to work around this, there’s a really silly procedure. I used it once, and I’m not proud of that.

So here we go.

First, a very simple script that outputs the patch mail into a file. Say that I called it dumpit (should be executable, of course):

#!/bin/bash

cat > /home/eli/Desktop/git-send-email.eml

Then change ~/.gitconfig, so it reads something like this in the [sendemail] section:

[sendemail]
        from = mail.username@gmail.com
        smtpServer = /home/eli/Desktop/dumpit

So basically it uses the silly script as a mail server, and the content goes out to a plain file.

Then run git send-email as usual. The result is a git-send-email.eml as a file.

And now comes the part of making Thunderbird send it.

• Close Thunderbird. All windows.
• Change directory to where Thunderbird keeps its profile files, to under Mail/Local Folders
• Remove “Unsent Messages” and “Unsent Messages.msf”
• Open Thunderbird again
• Inside Thunderbird, go to Hamburger Icon > File > Open > Saved Message… and select git-send-email.eml. The email message should appear.
• Right-Click somewhere in the message’s body, and pick Edit as New Message…
• Don’t send this message as is! It’s completely messed up. In particular, there are some indentations in the patch itself, which renders it useless.
• Instead, pick File > Send Later.
• Once again, close Thunderbird. All windows.
• Remove “Unsent Messages.msf” (only)
• Edit “Unsent Messages” as follows: Everything under the “Content-Transfer-Encoding: 7bit” part is the mail’s body. So remove the “From:” line after it, and paste the email’s body from git-send-email.eml instead.
• Note that there are normally two blank lines after the mail’s body. Retain them.
• Open Thunderbird again. Verify that those indentations are away.
• Look at the mail inside Outbox, and verify that it’s OK now. These are the three things to look for in particular:
  • The “From:” part at the beginning of the message is gone.
  • At the end of the message, there’s a “–” and git’s version number. These should be in separate lines.
  • Look at the mail’s source. The “+” and “-” signs of the diffs must not be indented.
• If all is fine, right-click Outbox, and pick “Send unsent messages”. And hope for good.

Are you sure you want to do this?

Introduction

After a long time that Google’s smtp server occasionally refused to play ball with fetchmail, tons of Critical Alerts on “someone knowing my password” and requests to move away from “Less Secure Apps” (LSA) and other passive-aggressive behaviors, I eventually got the famous “On May 30, you may lose access” mail. That was in the beginning of March.

My reaction to that was exactly the same as to previous warnings: I ignored it. Given the change that was needed to conform to Google’s requirements, I went on with the strategy that “may” doesn’t necessarily mean “will”, and that I don’t move until they pull the plug. Which they didn’t on May 30th, as suggested. Instead, it happened on June 8th.

Since I don’t intend to ditch my Gmail address, it follows that OAuth2 is going to be part of my life from now on, and that I’ll probably have to figure out why this or that doesn’t work out every now and then. So I might as well learn the darn thing once and for all. Which I just did.

This post consists of my own understanding on this matter. More than anything, it’s intended as a reminder to myself each time I’ll need to shove my hands deep into some related problem.

None of the covered subjects have anything to do with my professional activity as an engineer.

Spoiler: My decisions on my own mail processing

This is my practical conclusion of everything written below, so the TL;DR is that I decided to stop using Fetchmail with Google’s mail servers. Instead, I’ve set Gmail to forward all emails to one of my other email accounts, to which I have pop3 access with Fetchmail (it’s actually on a server I control). I just added another transmission leg.

This is mainly because of the possibility that continuing to use Fetchmail with Google’s server will require my personal attention every now and then, for reasons I elaborate on below. It’s not impossible, but I’m not sure Fetchmail with Gmail is going to be a background thing anymore.

Mail forwarding is a solid solution to this. It doesn’t create a single point of failure, because I can always access the mails with Gmail’s web interface, should there be a problem with the forward-to-Fetchmail route. The only nasty thing that can happen is that the forwarding’s destination email address may be disclosed to the sender, if the delivery fails for some reason: It appears in the bounce message.

So if you want to use the forwarding method for an email address that you keep for the sake of anonymity, you’ll have to use a destination email address that says nothing about you either. There are plenty of email services with POP3 support at a fairly low cost.

The only reason I still need to live with OAuth2 support is that emails that I send with my Gmail address must go through Google’s servers, or else they are rejected by a whole lot of mail servers out there by virtue of DMARC.

So I upgraded Thunderbird to a version that supports OAuth2, and it works nicely with Google. I could have fetched the emails with Thunderbird too, but I still want to run my own spam filter, which I why I want fetchmail to remain in the loop for arriving mails.

And now, to the long story.

What is OAuth2?

To make a long story short, it’s the mechanism behind “Login with Google / Facebook / whatever”. Rather than having the user maintain a username and password for every service it accesses, there’s one Authorization Server, say Google, that maintains the capability to verify the actual user.

The idea is that when the user wants to use some website with “Login with Google”, the website doesn’t need to check the user’s identity itself, but instead it relies on the authentication made by Google. As a bonus, the fact that the user has logged into the site with Google, allows the site’s back-end (that is, the web server) to access some Google services on behalf of the user. For example, to add an entry in the user’s Google calendar.

To make this work, the site’s back-end needs to be able to prove that it’s eligible to act on behalf of the said user. For this purpose, it obtains an access token from the Authorization Server. In essence, this access token is a short-lived password for performing certain tasks on behalf of a certain user.

So an access token is limited in three ways:

• It’s related to a specific Google user
• It’s limited in time
• It gives its owner only specific permissions to carry out operations, or as they’re called, scopes.

For the sake of fetching emails, the recent change was that Gmail moved from accepting username + password authentication to only accepting an access token that allows the relevant user to perform pop / imap operations.

Interactions with the Authorization Server

The Authorization Server is responsible mainly for two tasks:

• The initial authentication, which results in obtaining an access token, a refresh token and various information (in JSON format).
• The refreshing of the access token, which is performed to replace an expired or soon-to-expire access token with a valid one.

So the overall picture is that it starts with some initial authentication, and then the owner of the access token keeps extending its validity by recurring refresh requests.

The initial authentication is done by a human using a web browser. That’s the whole point. This allows the Authorization Server to control the level of torture necessary to obtain the access token. It may not require any action if the user is already safely logged in, and it may suddenly decide to ask silly questions and/or perform two-factor authentication and whatnot.

Refreshing the access token is a computer-to-computer protocol that requires no human interaction. In principle, access can be granted forever based upon that initial authentication by refreshing the token indefinitely. But the Authorization Server is nevertheless allowed to refuse a refresh request for any or no reason. In fact, this is the way Google can force us humans to pay attention. The documentation tends to imply that tokens are always refreshed, but at the same time clearly state that the requester of a refresh should handle a refusal gracefully by reverting to the browser thing.

Remember those “suspicious activity” notifications from Google, begging us to confirm that it was us doing something on an uknown device? No need to beg anymore. If Google wants us to confirm something, it just denies the token refresh request. The only way to resume access is going back to initial authentication. This brings the human user to a browser soon enough to re-authenticate, which is a good opportunity to sort out whatever needs sorting out.

For example, if Thunderbird is used to access mail from Gmail with OAuth2, it must have the capability to open a browser window in order to perform the initial authentication (which it does nowadays). Hence if a refresh requests fails, this browser window will be opened again for further action. So there’s a means to talk with the human user. This possibility didn’t exist with the old password authentication, because if that failed, the user was prompted for a new password. So there was no reasonable way to initiate communication with the human user by refusing access.

How obnoxious service providers intend to be with this new whip is yet to be seen, but it’s clear that OAuth2 opens that possibility. The fact that access tokens are currently refreshed forever without the need to re-authenticate, doesn’t say how it’s going to be in the future.

As a bit of a side note, it’s common practice that access to cloud services can be made with an initial authentication that doesn’t involve a web browser. This makes sense, as software that consumes these services typically runs on servers with no human around. Today, this can be used to obtain tokens for Gmail access, but I doubt that will go on for long.

The authentication handshake in a nutshell

There are plenty of resources on OAuth2: To be begin with, there’s RFC 6749, which defines OAuth2, and several tutorials on the matter, for example this one. And there’s Google’s page on using OAuth2 for accessing Google APIs, which is maybe the most interesting one, as it walks through the different usage scenarios, including devices that can’t run a web browser.

This way or another, it boils down to the following stages for a website with “Login with X”:

• A web browser goes to the Authorization Server with a URL that includes information about the request, by virtue of a link saying “Login with Google” or something like that. It’s typically a very long and tangled URL with several CGI-style parameters (it’s a GET request).  Among the parameters in the link, there’s the client ID (who is requesting access), what kind of access is required from Google’s servers (the scopes) and to what URL the Authorization Server should redirect the browser when it’s done torturing the human in front of the browser. For example, the link used by TikTok’s “Continue with Google” goes

  https://accounts.google.com/o/oauth2/v2/auth/identifier?client_id=1096011445005-sdea0nf5jvj14eia93icpttv27cidkvk.apps.googleusercontent.com&response_type=token&redirect_uri=https%3A%2F%2Fwww.tiktok.com%2Flogin%2F&state=%7B%22client_id%22%3A%221096011445005-sdea0nf5jvj14eia93icpttv27cidkvk.apps.googleusercontent.com%22%2C%22network%22%3A%22google%22%2C%22display%22%3A%22popup%22%2C%22callback%22%3A%22_hellojs_5kkckpps%22%2C%22state%22%3A%22%22%2C%22redirect_uri%22%3A%22https%3A%2F%2Fwww.tiktok.com%2Flogin%2F%22%2C%22scope%22%3A%22basic%22%7D&scope=openid%20profile&prompt=consent&flowName=GeneralOAuthFlow

• The Authorization Server does whatever it does in that browser window, and when that ends, it redirects the browser with a 302 HTTP redirect to the URL that appeared in the request. It appends a CGI-style “code=” parameter to the URL, and by doing that it gives the back-end server an authorization code. If there was a “state” parameter in the link to the Authorization Server, it’s copied as a second parameter in this redirection. This is how the back-end server knows which request it got a response for.
• Now that the back-end server has the authorization code, it contacts the Authorization Server directly over HTTP, and requests access tokens, using this code in the request. The Authorization Server responds with a JSON string, that contains the access token, the refresh token and other information.
• Using the access token, the back-end server can access various Google API servers.
• Using the refresh token, the back-end server can obtain a new access token (and possibly a new refresh token) when the existing access token is about to expire. Refresh tokens have no given expiration time, but if a new one is obtained during refresh, it should be used in following refresh requests.

It may be required to add additional credentials in requests for an access token (i.e. along with an authorization code or a refresh token), namely the client_id and client_secret parameters. These credentials are relevant in particular with cloud applications, and they are obtained when registering for such.

So this was the scenario for a website. What about fetching mails with Thunderbird and alike? It’s basically the same principle, only that the redirection with the authorization code is handled differently. There are several other variations, depending on the capabilities of the device that needs access. Among others, there’s a browser-less option for cloud applications, which is once again a variant of the above.

As for Thunderbird and other MUAs, they take the role of the back-end server: If they don’t have a valid access token, they open a browser window with the Authorization Server’s URL, with all necessary parameters. The redirection to the website is done differently, but it boils down to Thunderbird obtaining the authorization code and subsequently using it to obtain the access token. And then refreshing it as necessary.

So to summarize: There’s a browser session that ends with an authorization code, and the application uses this authorization code to get an access token. This access token is effectively a short-lived password that is used with Google’s API servers, Google’s smtp server included.

And by the way, there’s a maintained Perl module for OAuth2. I don’t know if I should be surprised about that.

fetchmail and OAuth2

Fetchmail 7 is apparently going to to support OAuth2, but there’s little enthusiasm for supporting it on the long run. It also appears like OAuth2 will not be backported to fetchmail-6.x.x.

To Fetchmail, the authentication tokens are just a replacement for the password. It’s another secret to send away to the server. So the entry in .fetchmailrc goes something like this:

poll <imap_server> protocol imap
  auth oauthbearer username <your_email>
  passwordfile "/home/yourname/.fetchmail-token"
  is yourname here
[ ... ]

For this to work, there must be a mechanism for keeping the token valid. The mechanism suggested in fetchmail’s own git repository is that a cronjob first invokes a Python script that refreshes the token if necessary (and updates .fetchmail-token). Fetchmail is then called (in non-daemon mode) as part of this cronjob, and does its thing.

The approach for making this work automatically is to rely on the API for Google and Microsoft’s cloud services, which is intended for allowing scripts to access these services in a safely authenticated way. It seems to be an attempt to avoid the browser session at all costs. Which is understandable, given that fetchmail is traditionally a daemon that works silently in the background.

However using fetchmail like this requires registering the user as a Google cloud API user, which is quite difficult and otherwise annoying. So I can definitely understand the lack to of enthusiasm expressed by Fetchmail’s authors (more on that below).

But I beg to differ on this approach. The browser session is what Google really wants, so there’s no choice but to embrace it. Since my own motivation to use fetchmail is zero at this point, I didn’t implement anything, but this is what I would have done. And maybe will do, if it becomes relevant in the future:

A simple systemd-based daemon keeps track on when tokens expire, and issues refresh requests as necessary. If a valid token for a Gmail account is missing (because the refresh requests failed, or because an account was just added), this daemon draws the user’s attention to the need for an authentication session. Maybe a popup, maybe an icon on the system tray. When the user responds to that alert, a browser window opens with the relevant URL, and the authentication process takes place, ending with an authorization code, which is then turned into a valid token.

As for Fetchmail itself, it keeps running as usual as a daemon, only using access tokens instead of passwords. If a token is invalid, Google’s server will reject it, and if that goes on for too long, Fetchmail issues the warning mail message we’re probably all familiar with. Nothing new.

This doesn’t require any registration to any service. Just to enter the username and password the first time the daemon is launched, and then possibly go through whatever torture Google requires when it gets paranoid. But this is the way Google probably wants it to work, so no point trying to fight it. Frankly, I don’t quite understand why the Fetchmail guys didn’t go this way to begin with.

Future of OAuth2 support

Personally, I think Fetchmail should support OAuth2 authentication to the extent that it’s capable of using an access token for authentication. As for obtaining and maintaining the tokens, I can’t see why that has anything to do with Fetchmail.

The authors’ view is currently somewhat pessimistic. To cite the relevant entry in the NEWS file:

OAuth2 access so far seems only to be supported by providers who want to exert control over what clients users can use to access their very own personal data, or make money out of having clients verified. There does not appear to be a standard way how service end-points are configured, so fetchmail would have to carry lots of provider-specific information, which the author cannot provide for lack of resources.

OAuth2 is therefore generally considered as experimental, and unsupported, OAuth2 may be removed at any time without prior warning.

As for their affection for OAuth2, see the preface in README.OAUTH2 file. This file nevertheless explains how to obtain an OAuth2 client id and client secret from Google and Microsoft. Something I suggested to skip, but anyhow.

App passwords

This isn’t really related, but it’s often mentioned as a substitute for OAuth2, so here are a few words on that.

It seems like there’s a possibility to generate a 16-digit password, which is specific to an app. So at least in theory, this app password could be given to Fetchmail in order to perform a regular login.

I didn’t pursue this direction, mainly because the generation of an app password requires two-step verification. Forwarding sounds so much nicer all of the sudden.

Besides, I will not be surprised if Google drops App passwords sooner or later, in particular for Gmail access.

Summary

I can’t say that I’m happy with OAuth2 becoming mandatory, but I guess it’s here to stay. My personal speculation is that it has become mandatory to allow Google to re-authenticate humans gracefully, possibly with increasingly annoying means. This is a fight against spammers, scammers and account hijackers, so paranoia is the name of the game.

Apparently, forcing the owner of the Google account into an authentication session, either with a browser on the desktop or on the mobile phone, possibly both combined, is the future weapon in this fight. It’s quite annoying indeed, but I guess there are worse problems on this planet.

These are my notes as I upgraded Thunderbird from version 3.0.7 (released September 2010) to 91.10.0 on Linux Mint 19. That’s more than a ten year’s gap, which says something about what I think about upgrading software (which was somewhat justified, given the rubbish issues that arose, as detailed below). What eventually forced me to do this was the need to support OAuth2 in order to send emails through Google’s Gmail server (supported since 91.8.0).

Thunderbird is essentially a Firefox browser which happens to be set up with a GUI that processes emails. So for example, the classic menubar is hidden, but can be revealed by pressing Alt.

Using the correct profile

When attempting to run a new version of Thunderbird, be sure to rename ~/.thunderbird into something else, or else the current profile will be upgraded right away. With some luck, the suffixes (e.g. -release) might make Thunderbird ignore the old information, but don’t trust that.

Actually, it seems like this is handled gracefully anyhow. When I installed exactly the same version on a different position on the disk, it ignored the profile with -release suffix, and added one with -release-1. So go figure.

To select which profile to work with, invoke Thunderbird with Profile Manager with

$ thunderbird -profilemanager &

For making the upgrade, first make a backup tarball from the original profile directory.

To adopt in into the new version of Thunderbird, invoke the Profile Manager and pick Create Profile…, create a new directory (I called it “mainprofile”), and pick that as the place for the new profile. Launch Thunderbird, quit right away, and then delete the new directory. Rename the old directory with the new deleted directory’s name. Then launch Thunderbird again.

Add-ons

Previously, I had the following add-ons:

• BiDi Mail UI (apparently still necessary)
• Clippings. Just import the previous clippings from ~/clipdat2.rdf. Unlike the old version, the data is kept in a database file inside the profile, so the old file can be deleted.
• Gnome Integration Options for calling a command on mail arrival. It was deprecated. So I went for Mailbox Alert, which allows adding specific actions: Sound, a message and/or command. With mail folder granularity, in fact.
• Mail Tweak. It’s really really old, and probably unnecessary since long.
• Outgoing Message Format (for text vs. HTML messages). Deprecated since long, as these options are integrated into Thunderbird itself.

So I remained with the first two only.

Installing Thunderbird

The simplest Thunderbird installation involves downloading it from their website and extract the tarball somewhere in the user’s own directories. For a proper installation, I installed it under /usr/local/bin/ with

# tar -C /usr/local/bin -xjvf thunderbird-91.10.0.tar.bz2

as root. And then reorganize it slightly:

# cd /usr/local/bin
# mv thunderbird thunderbird-91.10.0
# ln -s thunderbird-91.10.0/thunderbird

Composing HTML messages

Right-click the account at the left bar, pick Settings and select the Composition & Addressing item. Make sure Compose messages in HTML is unchecked: Messages should be composed as plain text by default.

Then go through each of the mail identities and verify that Compose messages in HTML is unchecked under the Composition & Addressing tab.

However if Shift is pressed along with clicking Write, Reply or whatever for composing a new message, Thunderbird opens it as HTML.

Recover old contacts

Thunderbird went from the old *.mab format to SQLite for keeping the address books. So go Tools > Import… > Pick Address Books… and pick Monk Database, and from there pick abook.mab (and posssibly repeat this with history.mab, but I skipped this, because it’s too much).

Silencing update notices

Thunderbird, like most software nowadays, wants to update itself automatically, because who cares if something goes wrong all of the sudden as long as the latest version is installed.

I messed around with this for quite long until I found the solution. So I’m leaving everything I did written here, but it’s probably enough with just adding policies.json, as suggested below.

So to the whole story (which you probably want to skip): Under Preferences > General > Updates I selected “check for updates” rather than install automatically (it can’t anyhow, since I’ve installed Thunderbird as root), but then it starts nagging that there are updates.

So it’s down to setting the application properties manually by going to Preferences > General > Config Editor… (button at the bottom).

I changed app.update.promptWaitTime to 31536000 (365 days) but that didn’t have any effect. So I added an app.update.silent property and set it true, but that didn’t solve the problem either. So the next step was to change app.update.staging.enabled to false, and that did the trick. Well, almost. With this, Thunderbird didn’t issue a notification, but its tab on the system tray gets focus every day. Passive aggressive.

As a side note, there are other suggestions I’ve encountered out there: To change app.update.url so that Thunderbird doesn’t know where to look for updates, or set app.update.doorhanger false. Haven’t tried either.

So what actually worked: Create a policies.json in /usr/local/bin/thunderbird/distribution/, with “DisableAppUpdate“: true, that is:

{
 "policies": {
  "DisableAppUpdate": true
 }
}

Note that the “distribution” directory must be in the same the directory as the actual executable for Thunderbird (that is, follow the symbolic link if such exists). In my case, I had to add this directory myself, because of a manual installation.

And, as suggested on this page, the successful deployment can be verified by restarting Thunderbird, and then looking at Help > About inside Thunderbird, which now says (note the comment on updates being disabled):

In hindsight, I can speculate on why this works: The authors of Thunderbird really don’t want us to turn off automatic updates, mainly because if people start running outdated software, that increases the chance of a widespread attack on some vulnerability, which can damage the software’s reputation. So Thunderbird is designed to ignore previous possibilities to turn the update off.

There’s only one case where there’s no choice: If Thunderbird was installed by the distribution. In this case, it’s installed as root, so it can’t be updated by a plain user. Hence it’s the distribution’s role to nag. And it has the same interest to nag about upgrades (reputation and that).

So I guess that’s why Thunderbird respects this JSON file only.

Folders with new mails in red

Exactly like 10 years ago, the trick is to create a “chrome” directory under .thunderbird/ and then add the following file:

$ cat ~/.thunderbird/sdf2k45i.default/chrome/userChrome.css
@namespace
url("http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"); /* set default namespace to XUL */

/* Setting the color of folders containing new messages to red */

treechildren::-moz-tree-cell-text(folderNameCol, newMessages-true) {
 font-weight: bold;
 color: red !important;
}

But unlike old Thunderbird, this file isn’t read by default. So to fix that, go to Preferences > General > Config Editor… (button at the bottom) and there change toolkit.legacyUserProfileCustomizations.stylesheets to true.

New mail icon in system tray

Thunderbird sends a regular notification when a new mail arrives, but exactly like last time, I want a dedicated icon that is dismissed only when I click it. The rationale is to be able to see if a new mail has arrived at a quick glance of the system tray. Neither zenity –notification nor send-notify were good for this, since they send the common notification (zenity used to just add an icon, but it “got better”).

But then there’s yad. I began with “apt install yad”, but that gave me a really old version that distorted the icon in the system bar. So I installed it from the git repository’s tag 1.0. I first attempted v12.0, but I ended up with the problem mentioned here, and didn’t want to mess around with it more.

Its “make install” adds /usr/local/bin/yad, as well as a lot of yad.mo under /usr/local/share/locale/*, a lot of yad.png under /usr/local/share/icons/*, yad.m4 under /usr/local/share/aclocal/ and yad.1 + pfd.1 in /usr/local/share/man/man1. So quite a lot of files, but in a sensible way.

With this done, the following script is kept (as executable) as /usr/local/bin/new-mail-icon:

#!/usr/bin/perl
use warnings;
use strict;
use Fcntl qw[ :flock ];

my $THEDIR="$ENV{HOME}/.thunderbird";
my $ICON="$THEDIR/green-mail-unread.png";

my $NOW=scalar localtime;

open(my $fh, "<", "$ICON")
  or die "Can't open $ICON for read: $!";

# Lock the file. If it's already locked, the icon is already
# in the tray, so fail silently (and don't block).

flock($fh, LOCK_EX | LOCK_NB) or exit 0;

fork() && exit 0; # Only child continues

system('yad', '--notification', "--text=New mail on $NOW", "--image=$ICON", '--icon-size=32');

This script is the improved version of the previous one, and it prevents multiple icons in the tray much better: It locks the icon file exclusively and without blocking. Hence if there’s any other process that shows the icon, subsequent attempts to lock this file fail immediately.

Since the “yad” call takes a second or two, the scripts forks and exits before that, so it doesn’t delay Thunderbird’s machinery.

With this script in place, the Mailbox Alert is configured as follows. Add a new item to the list as in this dialog box:

The sound should be set to a WAV file of choice.

Then right-click the mail folder to have covered (Local Folders in my case), pick Mailbox Alert and enable “New Mail” and “Alert for child folders”.

Then right-click “Inbox” under this folder, and verify that nothing is checked for Mailbox Alert for it (in particular not “Default sound”). That except for the Outbox and Draft folders, for which “Don’t let parent folders alert for this one” should be checked, or else there’s a false alarm on autosaving and when using “send later”.

Later on, I changed my mind and added a message popup, so now all three checkboxes are ticked, and the Message tab reads:

I picked the icon as /usr/local/bin/thunderbird-91.10.0/chrome/icons/default/default32.png (this depends on the installation path, of course).

I’m not 100% clear why the original alert didn’t show up, even though “Show an alert” was still checked under “Incoming Mails” at Preferences > General. I actually preferred the good old one, but it seems like Mailbox Alert muted it. I unchecked it anyhow, just to be safe.

Refusing to remember passwords + failing to sent through gmail

It’s not a real upgrade if a weird problem doesn’t occur out of the blue.

So attempting to Get Messages from pop3 server at localhost failed quite oddly: Every time I checked the box to use Password Manager to remember the password, it got stuck with “Main: Connected to 127.0.0.1…”. But checking with Wireshark, it turned out that Thunderbird asked the server about its capabilities (CAPA), got an answer and then did nothing for about 10 seconds, after which it closed the connection.

On the other hand, when I didn’t request remembering the password, it went fine, and so did subsequent attempts to fetch mail from the pop3 server.

Another thing was that when attempting to use Gmail’s server, I went through the entire OAuth2 thing (the browser window, and asking for my permissions) but then the mail was just stuck on “Sending message”. Like, forever.

So I followed the advice here, and deleted key3.db, key4.db, secmod.db, cert*.db and all signon* files with Thunderbird not running of course. Really old stuff.

And that fixed it.

The files that were apparently created when things got fine were logins.json, cert9.db, key4.db and pkcs11.txt. But I might have missed something.

The GUI stuck for a few seconds every now and then

This happened occasionally when I navigated from one mail folder to another. The solution I found somewhere was to delete all .msf files from where Thunderbird keeps the mail info, and that did the trick. Ehm, just for a while. After a few days, it was back.

As a side effect, it forgot the display settings for each folder, i.e. which columns to show and in what order.

These .msf files are apparently indexes to the files containing the actual messages, and indeed it took a few seconds before something appeared when I went to view each mail folder for the first time. At which time the new .msf files went from zero bytes to a significant figure.

Since the problem remains, I watched “top” when the GUI got stuck. And indeed, Thunderbird’s process was at 100%, but so was a completely different process: caribou. Which is a virtual keyboard. Do I need one? No. So to get rid of this process (which runs all the time, but doesn’t eat a lot of CPU normally), go Accessibility settings, the Keyboard tab and turn “Enable the on-screen keyboard” off. The process is gone, and so is the problem with the GUI? Nope. It’s basically the same, but instead of two processes taking 100% CPU, now it’s Thunderbird alone. I have no idea what to do next.

Checking up the server’s logs, there were tons of these messages:

dovecot: master: Warning: service(pop3-login): process_limit (100) reached, client connections are being dropped

Restarting dovecot got it back running properly again, and I got a flood of the mails that were pending on the server. This was exceptionally nasty, because mails stopped arriving silently.

So what was the problem? The clue is in these log messages, which occurred about a minute after the system’s boot (it’s a VPS virtual machine):

Jul 13 11:21:46 dovecot: master: Error: service(anvil): Initial status notification not received in 30 seconds, killing the process
Jul 13 11:21:46 dovecot: master: Error: service(log): Initial status notification not received in 30 seconds, killing the process
Jul 13 11:21:46 dovecot: master: Error: service(ssl-params): Initial status notification not received in 30 seconds, killing the process
Jul 13 11:21:46 dovecot: master: Error: service(log): child 1210 killed with signal 9

These three services are helper processes for dovecot, as can be seen in the output of systemctl status:

            ├─dovecot.service
             │ ├─11690 /usr/sbin/dovecot -F
             │ ├─11693 dovecot/anvil
             │ ├─11694 dovecot/log
             │ ├─26494 dovecot/config
             │ ├─26495 dovecot/auth
             │ └─26530 dovecot/auth -w

What seems to have happened is that these processes failed to launch properly within the 30 second timeout limit, and were therefore killed by dovecot. And then attempts to make pop3 connections seem to have got stuck, with the forked processes that are made for each connection remaining. Eventually, they reached the maximum of 100.

The reason this happened only now is probably that the hosting server had some technical failure and was brought down for maintenance. When it went up again, all VMs were booted at the same time, so they were all very slow in the beginning. Hence it took exceptionally long to kick off those helper processes. The 30 seconds timeout kicked in.

The solution? Restart dovecot once in 24 hours with a plain cronjob. Ugly, but works. In the worst case, mail will be delayed for 24 hours. This is a very rare event to begin with.

$ perl -MMIME::QuotedPrint -e 'local $/; $x=<>; print decode_qp($x)' < quoted.txt > unquoted.html

and I needed this to extract an HTML segment of an email.

One morning, I got a bounce message from my own mail sendmail server, saying that it failed to deliver a message I never sent. That’s red alert. It means that someone managed to provoke my mail server to send an outbound message. It’s red alert, because my mail server effectively relays spam to any destination that the spammer chooses. This could ruin the server’s reputation horribly.

It turned out that an arriving mail required a return receipt, which was destined to just some mail address. There’s an SMTP feature called Delivery Status Notification (DSN), which allows the client connecting to the mail server to ask for a mail “in return”, informing the sender of the mail if it was properly delivered. The problem is that the MAIL FROM / From addresses could be spoofed, pointing at a destination to spam. Congratulations, your mail server was just tricked into sending spam. This kind of trickery is called backscatter.

Checking my own mail logs, the DSN is a virtually unused feature. So it’s probably just something spammers can take advantage of.

The relevant RFC for DSN is RFC1891. Further explanations on DSN can be found in one of sendmail’s tutorial pages.

How to turn DSN off

First, I recommend checking if it’s not disabled already, as explained below. In particular, if the paranoid-level “goaway” privacy option is used, DSN is turned off anyhow.

It’s actually easy. Add the noreceipts option to PrivacyOptions. More precisely, edit /etc/mail/sendmail.mc and add noreceipts to the list of already existing options. In my case, it ended up as

define(`confPRIVACY_FLAGS',dnl
`needmailhelo,needexpnhelo,needvrfyhelo,restrictqrun,restrictexpand,nobodyreturn,noetrn,noexpn,novrfy,noactualrecipient,noreceipts')dnl

and then run “make” in /etc/mail, and restart sendmail.

Turning off DSN is often recommended against in different sendmail guides, because it’s considered a “valuable feature” or so. As mentioned above, I haven’t seen it used by anyone else than spammers.

Will my mail server do DSN?

Easy to check, because the server announces its willingness to fulfill DSN requests at the beginning of the SMTP session, with the line marked in red in the sample session below:

<<< 220 mx.mymailserver.com ESMTP MTA; Wed, 15 Jul 2020 10:22:32 GMT
>>> EHLO localhost.localdomain
<<< 250-mx.mymailserver.com Hello 46-117-33-227.bb.netvision.net.il [46.117.33.227], pleased to meet you
<<< 250-ENHANCEDSTATUSCODES
<<< 250-PIPELINING
<<< 250-8BITMIME
<<< 250-SIZE
<<< 250-DSN
<<< 250-DELIVERBY
<<< 250 HELP
>>> MAIL FROM:<spamvictim@billauer.se>
<<< 250 2.1.0 <spamvictim@billauer.se>... Sender ok
>>> RCPT TO:<legal_address@billauer.se> NOTIFY=SUCCESS
<<< 250 2.1.5 <legal_address@billauer.se>... Recipient ok
>>> DATA
<<< 354 Enter mail, end with "." on a line by itself
>>> MIME-Version: 1.0
>>> From: spamvictim@billauer.se
>>> To: legal_address@billauer.se
>>> Subject: Testing email.
>>>
>>>
>>> Just a test, please ignore
>>> .
<<< 250 2.0.0 06FAMWa1014200 Message accepted for delivery
>>> QUIT
<<< 221 2.0.0 mx.mymailserver.com closing connection

To test a mail server for its behavior with DSN, the script that I’ve already published can be used. To make it request a return receipt, the two lines that set the SMTP recipient should be changed to

  die("Failed to set receipient\n")
    if (! ($smtp->recipient( ($to_addr ), { Notify => ['SUCCESS'] } ) ) );

This change causes the NOTIFY=SUCCESS part in the RCPT TO line, which effectively requests a receipt from the server when the mail is properly delivered.

Note that if DSN isn’t supported by the mail server (possibly because of the privacy option fix shown above), the SMPT session looks exactly the same, except that the SMTP line marked in red will be absent. Then the mail server just ignores the NOTIFY=SUCCESS part silently, and responds exactly as before.

However when running the Perl script, the Net::SMTP will be kind enough to issue a warning to its stderr:

Net::SMTP::recipient: DSN option not supported by host at ./testmail.pl line 36.

The mail addresses I used in the sample session above are bogus, of courses, but note that the spam victim is the sender of the email, because that’s where the return receipt goes. On top of that, the RCPT TO address will also get a spam message, but that’s the smaller problem, as it’s yet another spam message arriving — not one that is sent away from our server.

I should also mention that Notify can be a comma-separated list of events, e.g.

RCPT TO:<bad_address@billauer.se> NOTIFY=SUCCESS,FAILURE,DELAY

however FAILURE doesn’t include the user not being known to the server, in which case the message is dropped anyhow without any DSN message generated. So as a spam trick, one can’t send mails to random addresses, and issue spam bounce messages because they failed. That would have been too easy.

In the mail logs

The sample session shown above causes the following lines in mail.log. Note the line marked in red, which indicates that the return receipt mechanism was fired off.

Jul 15 10:15:31 sm-mta[12697]: 06FAFTbL012697: from=<spamvictim@billauer.se>, size=121, class=0, nrcpts=1, msgid=<202007151015.06FAFTbL012697@mx.mymailserver.com>, proto=ESMTP, daemon=IPv4-port-587, relay=46-117-33-227.bb.netvision.net.il
[46.117.33.227]
Jul 15 10:15:31 sm-mta[12698]: 06FAFTbL012697: to=<legal_address@billauer.se>, ctladdr=<spamvictim@billauer.se> (1010/500), delay=00:00:01, xdelay=00:00:00, mailer=local, pri=30456, dsn=2.0.0, stat=Sent
Jul 15 10:15:31 sm-mta[12698]: 06FAFTbL012697: 06FAFVbL012698: DSN: Return receipt
Jul 15 10:15:31 sm-mta[12698]: 06FAFVbL012698: to=<spamvictim@billauer.se>, delay=00:00:00, xdelay=00:00:00, mailer=local, pri=30000, dsn=2.0.0, stat=Sent

The receipt

Since I’m at it, this is what a receipt message for the sample session above looks like:

Received: from localhost (localhost)	by mx.mymailserver.com
 (8.14.4/8.14.4/Debian-8+deb8u2) id 06FAFVbL012698;	Wed, 15 Jul 2020
 10:15:31 GMT
Date: Wed, 15 Jul 2020 10:15:31 GMT
From: Mail Delivery Subsystem <MAILER-DAEMON@billauer.se>
Message-ID: <202007151015.06FAFVbL012698@mx.mymailserver.com>
To: <spamvictim@billauer.se>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
 boundary="06FAFVbL012698.1594808131/mx.mymailserver.com"
Subject: Return receipt
Auto-Submitted: auto-generated (return-receipt)
X-Mail-Filter: main

This is a MIME-encapsulated message

--06FAFVbL012698.1594808131/mx.mymailserver.com

The original message was received at Wed, 15 Jul 2020 10:15:30 GMT
from 46-117-33-227.bb.netvision.net.il [46.117.33.227]

   ----- The following addresses had successful delivery notifications -----
<legal_address@billauer.se>  (successfully delivered to mailbox)

   ----- Transcript of session follows -----
<legal_address@billauer.se>... Successfully delivered

--06FAFVbL012698.1594808131/mx.mymailserver.com
Content-Type: message/delivery-status

Reporting-MTA: dns; mx.mymailserver.com
Received-From-MTA: DNS; 46-117-33-227.bb.netvision.net.il
Arrival-Date: Wed, 15 Jul 2020 10:15:30 GMT

Final-Recipient: RFC822; legal_address@billauer.se
Action: delivered (to mailbox)
Status: 2.1.5
Last-Attempt-Date: Wed, 15 Jul 2020 10:15:31 GMT

--06FAFVbL012698.1594808131/mx.mymailserver.com
Content-Type: text/rfc822-headers

Return-Path: <spamvictim@billauer.se>
Received: from localhost.localdomain (46-117-33-227.bb.netvision.net.il [46.117.33.227])
	by mx.mymailserver.com (8.14.4/8.14.4/Debian-8+deb8u2) with ESMTP id 06FAFTbL012697
	for <legal_address@billauer.se>; Wed, 15 Jul 2020 10:15:30 GMT
Date: Wed, 15 Jul 2020 10:15:29 GMT
Message-Id: <202007151015.06FAFTbL012697@mx.mymailserver.com>
MIME-Version: 1.0
From: spamvictim@billauer.se
To: legal_address@billauer.se
Subject: Testing email.

--06FAFVbL012698.1594808131/mx.mymailserver.com--

But note that if DSN is used by a spammer to trick our mail server, we will get the failure notice that results from sending this message to the other server. If we’re lucky enough to get anything at all: If the message is accepted, we’ll never know our server has been sending spam.

Still in 2020, it seems like Microsoft lives up to its reputation: Being arrogant, thinking that anyone in business must be a huge corporate, and in particular ending up completely ridiculous. Microsoft’s mail servers, which accept on behalf of Hotmail, MSN, Office 365, Outlook.com, or Live.com users are no exception. This also affects companies and other entities which use their own domain names, but use Microsoft’s services for handling mail.

This post summarizes my personal experience and accumulated knowledge with delivering mail to their servers. I use a simple Linux sendmail SMTP MTA on a virtual server for handling the delivery of my own private mails as well as a very low traffic of transactional mails from a web server. All in all, it’s about 100 mails / month coming out from that server to all destinations.

So one server, one IP address with a perfect reputation on all open spam reputation trackers, with SPF, DKIM and DMARC records all in place properly.

One may ask why I’m not relying on existing mail delivery services or my ISP. Answer is simple: Any commercial mail delivery server is likely to have its reputation contaminated by some spammer, no matter what protection measures they take. When that happens, odds are that emails will just disappear, because the ISP has little interest in forwarding the bounce message saying that delivery failed. On a good day, they will be handling the problem quickly, and yet the sender of the lost mail won’t be aware that the correspondence is broken.

For this reason, it’s quite likely that small businesses will go on keeping their own, small, email delivery servers, maintaining their own reputation. So when Outlook’s servers are nasty with a single-IP server, they’re not just arrogant, but they are causing delivery issues with small to medium businesses.

To do when setting up the server

For starter info, go here. Microsoft is pretty upfront about not being friendly to new IP addresses (see troubleshooting page for postmasters).

So it’s a very good idea to create a Microsoft account to log into their services, and then join their Smart Network Data Service (SDNS) and Junk Mail Reporting Program. This is the start page for both of these services.

SDNS allows the owner of a mail server to register its IP address range (“Request Access“), so its status can be monitored (“View IP Status”) over time. When all is fine, the IP Status page says “All of the specified IPs have normal status”, and when they don’t like this or other IP address, it’s more like this (click to enlarge):

The Junk Mail Reporting Program (JMRP) allows the owner of the mail server to receive notifications (by email) when a mail message is delivered however deemed suspicious, either by an end-user (marking it as spam) or by automatic means. So it’s a good idea to create a special email address for this purpose and fill in the JMRP form. Even for the sake of claiming that you got no complaints when contacting support later on.

Note that this is important for delivery of mail to any institution relies on Microsoft’s mail infrastructure. A proper IP address blacklist delisting takes you from

Mar 11 20:18:23 sm-mta[5817]: x2BKIL2H005815: to=<xxxxxxx@mit.edu>, delay=00:00:02, xdelay=00:00:02, mailer=esmtp, pri=121914, relay=mit-edu.mail.protection.outlook.com. [104.47.42.36], dsn=5.7.606, stat=User unknown

(but the bounce message indicated that it’s not an unknown user, but a blacklisted IP number) to

Mar 11 21:15:12 sm-mta[6170]: x2BLF8rT006168: to=<xxxxxxx@mit.edu>, delay=00:00:03, xdelay=00:00:03, mailer=esmtp, pri=121915, relay=mit-edu.mail.protection.outlook.com. [104.47.42.36], dsn=2.0.0, stat=Sent (<5C86CFDC.6000206@example.com> [InternalId=11420318042095, Hostname=DM5PR01MB2345.prod.exchangelabs.com] 11012 bytes in 0.191, 56.057 KB/sec Queued mail for delivery)

Note that the session response said nothing about a blacklisted IP, however the bounce message (not shown here) did.

Finally, Microsoft suggest getting a certification from Return Path. A paid-for service, clearly intended for large companies and in particular mass mailers to get their spam delivered. Microsoftish irony at its best.

To do when things go wrong

First thing first, read the bounce message. If it says that it’s on Microsoft’s IP blacklist, go to the Office 365 Anti-Spam IP Delist Portal and delist it.

Then check the IP’s status (requires logging in). If you’re blocked, contact support. This doesn’t require a Microsoft login account, by the way. I’m not sure if this link to the support page is valid in the long run, so it’s on SNDS’ main page (“contact sender support”) as well as Troubleshooting page.

My own ridiculous experience

I kicked off my mail server a bit more than a year ago. There was some trouble in the beginning, but that was no surprise. Then things got settled and working for a year, and only then, suddenly & out of the blue, a mail to a Hotmail address bounced with:

Action: failed
Status: 5.7.1
Diagnostic-Code: SMTP; 550 5.7.1 Unfortunately, messages from [193.29.56.92] weren't sent. Please contact your Internet service provider since part of their network is on our block list (S3140). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. [VE1EUR01FT021.eop-EUR01.prod.protection.outlook.com]

And indeed, checking the IP status indicated that is was blocked “because of user complaints or other evidence of spamming”.

So first I went to the mail logs. Low traffic. No indication that the server has been tricked into sending a lot of mails. No indication that it has been compromised in any way. And when a server has been compromised, you know it.

No chance that there were user complaints, because I got nothing from JMRP. So what the “evidence of spamming”?

My best guess: A handful transactional mail messages (at most) to their servers for authenticating email addresses that were marker suspicious by their super software. Putting these messages in quarantine for a few hours is the common solution when that happens. Spam is about volume. If all you got was 4-5 messages, how could that be a spam server? Only if you look at percentage. 100% suspicious. Silly or what?

So I filled in the contact support form, and soon enough I got a message saying a ticket has been opened, and 30 minutes later saying

We have completed reviewing the IP(s) you submitted. The following table contains the results of our investigation.

Not qualified for mitigation
193.29.56.92
Our investigation has determined that the above IP(s) do not qualify for mitigation. These IP(s) have previously received mitigations from deliverability support, and have failed to maintain patterns within our guidelines, so they are ineligible for additional mitigation at this time.

Cute, heh? And that is followed by a lot of general advice, basically copied from the website, recommending to join JMRP and SDNS. Which I had a year earlier, of course. The script that responded didn’t even bother to check that.

But it also said:

To have Deliverability Support investigate further, please reply to this email with a detailed description of the problem you are having, including specific error messages, and an agent will contact you.

And so I did. I wrote that I had joined those two programs a year ago, that the mail volume is low and so on. I doubt it really made a difference. After sending the reply, I got a somewhat automated response rather quickly, now with a more human touch:

Hello,

My name is Ayesha and I work with the Outlook.com Deliverability Support Team.

IP: 193.29.56.92

We will be looking into this issue along with the Escalations Team. We understand the urgency of this issue and will provide an update as soon as this is available. Rest assured that this ticket is being tracked and we will get back to you as soon as we have more information to offer.

Thank you for your patience.

Sincerely,
Ayesha

Outlook.com Deliverability Support

And then, a few days later, another mail:

Hello,

My name is Yaqub and I work with the Outlook.com Deliverability Support Team.

Recent activity coming from your IP(s): ( 193.29.56.92) has been flagged by our system as suspicious, causing your IP to become blocked. I have conducted an investigation into the emails originating from your IP space and have implemented mitigation for your deliverability problem. This process may take 24 – 48 hours to replicate completely throughout our system.

Please note that lifting the block does not guarantee that your email will be delivered to a user’s inbox. However, here are some things that can help you with delivery:

(and here came the same suggestions on JMRP and SDNS)

And about 24 hours later, the IP status went back to OK again. And my emails went through normally.

Well, almost. A few days even further down, I attempted to send an email to a live.co.uk destination, and once again, I got the same rejection message (in block list, S3140). The only difference was that the mail server on the other side was hotmail-com.olc.protection.outlook.com (residing in the US), and now eur.olc.protection.outlook.com (somewhere in Europe).

I checked the IP’s status in SDNS and it was fine. So updating the Europeans on the updated IP status takes a bit time, or what?

So I replied to last email I got from Microsoft’s support, saying it failed with live.co.uk. I didn’t get any reply, but a few hours later I tried again, and the mail went through. Coincidence or not.

This time I also caught the related messaged from the mail log. It’s

May 01 15:10:28 sm-mta[2239]: 041FASMh002237: to=<xxxxx@live.co.uk>, ctladdr=<eli@billauer.se> (510/500), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=121816, relay=eur.olc.protection.outlook.com. [104.47.1.33], dsn=5.0.0, stat=Service unavailable
May 01 15:10:28 sm-mta[2239]: 041FASMh002237: 041FASMh002239: DSN: Service unavailable

for a failure, and

May 02 06:23:00 sm-mta[4024]: 0426Mx1I004021: to=<xxxxx@live.co.uk>, ctladdr=<eli@billauer.se> (510/500), delay=00:00:01, xdelay=00:00:01, mailer=esmtp, pri=121808, relay=eur.olc.protection.outlook.com. [104.47.18.97], dsn=2.0.0, stat=Sent (<5EAD11C3.20105@billauer.se> [InternalId=21887153366859, Hostname=AM6EUR05HT060.eop-eur05.prod.protection.outlook.com] 10627 bytes in 0.246, 42.064 KB/sec Queued mail for delivery -> 250 2.1.5)

for success.

Lesson learned: Contact support and insist.

And the lesson to all those using Microsoft’s mail services: Your provider cuts off your email contacts arbitrarily. Because they are Microsoft.

Fact number one: Running your own mail server is the most likely cause for messing up, and that can mean an intrusion to the server or just turning it into a public toilet for spam.

Nevertheless, if mail delivery is important to you, there’s probably no way around. And I’m not talking about the ability to mass-mail. Even having plain, manually written, messages delivered to that semi-security-paranoid company, even if it has a ZIP attachment, can be a challenge. And no matter what ISP you have or other paid-for mail relay, there will always be someone else pushing junk through the same channel, and make the used mail relay’s reputation questionable.

And I’m also under the impression that paid-for mail relays won’t send you a bounce message if the destination server refuses to talk with them, or if they dropped the message for the sake of their own reputation. Once I got my own server running, I suddenly got a few of these. I now realize how some emails I sent in the past just vanished.

Not to mention that the emails reach their destination much faster with a private workhorse.

The key issue is to take control of your reputation. As simple as that. Use all possible means (detailed below) to ensure the recipient that it was you who sent it, and let the lack of blacklisting do the rest.

But, ehm, after all this preaching, the real reason I set up my own mail server was that I had no choice: My web host, which also took care of outgoing mail from my website drove me crazy with upgrades out of nowhere. So I went for VPS hosting, and that requires your own mailing server. For better and worse. Or use the services of some mail forwarding service, which effectively means that all my mails look like spam.

Port 25 might be blocked by ISP

This isn’t directly related, but important enough: My ISP, Netvision, blocks connection to port 25 from my computer, probably to avoid blacklisting of their IP addresses due to spamming from them.

This means that testing port 25 from my local computer is worthless and misleading. I suppose other ISPs do the same.

Use Mxtoolbox for testing port 25. Or even better, CheckTLS.

Selecting server software

Debian 8 arrived with Postfix by default. Exim is popular. I’m used to qmail and sendmail. Difficult to choose. Security is important. If the server gets compromised, my domain turns into a spamhouse at best. I also need some advanced features (DKIM in particular).

I went for sendmail 8.14.4. It has a bad word of mouth, but its security advisory record over the last ten years is better than Postfix and surely better than Exim. That’s a surprise, but you can’t argue with facts.

And I could go for qmail there, but it seems like it needs patching to support DKIM, and then who knows if I haven’t just made a hole.

Goals

The server should

• Open ports 25 and 587 for anyone to connect.
• Relay any email received on ports 25 and 587 from localhost only, without authentication
• Accept emails to local recipients ports 25 and 587 when connecting from foreign host. Port 25 is essential for inbound mail, but is sometimes blocked by firewalls, so open the other port as well.
• Add a DKIM signature to emails going to foreign hosts only
• Refuse to VRFY and EXPN
• Accept all emails (from external mailers as well) even if they don’t have rDNS entries etc (let the spam filter handle them)

Checklist

Note that a lot of these items are detailed further down this post. I also suggest taking a look on my post on SPF, DKIM and DMARC and possibly this, this and this as well.

• Be sure that your IP address is sustainable. The reputation which your mail server builds over the years is related to the IP address more than anything else. This also implies that the hosting service provider is stable, because if they go down, moving the server is maybe easy, but you lose the IP address.
• Verify that neither your IP nor your domain name have a bad reputation with a blacklist check.
• Make an rDNS record for the mail server’s IP. It should better begin a “mail” or “smtp” or “mx” subdomain. Make sure there’s only one rDNS record with a reverse DNS lookup (e.g. dig -x). Sounds silly, but happened to me.
• Set up the mail server properly and safely. Run a security check.
• Turn off DSN, so that the server won’t send return receipts. Discussed in detail in my other post, but the TLDR is to add noreceipts to confPRIVACY_FLAGS in /etc/mail/sendmail.mc.
• Set up the firewall to kill any IPv6 traffic (in particular reject, not drop, outgoing packets)
• Create SPF, DKIM and DMARC DNS record for the server. For SPF, with and without the “mx” subdomain.
• If the server has another name internally (other than the mx subdomain), make sure it has an A DNS record as well as an SPF one.
• Verify that the outgoing mail goes out right with this DKIM validator, which allows sending mail to it, and then see exactly how it arrived + results on the validation. Invaluable.
• Run a verbose manual mail submission and verify everything makes sense. In particular, make sure the HELO/EHLO domain matches the rDNS. However don’t expect the EHLO on the internal submission (from the program we’re running to the local server) to be the externally known one.
• Validate the DMARC DNS record for your domain by sending a test email to autoreply@dmarctest.org (or any other one listed here). My anecdotal experience is that Gmail refused to accept mail (as in “Service unavailable” SMTP rejection) from a domain until I added a DMARC record.
• Check any programs (web applications in particular) than send email, and verify that the envelope sender (MAIL FROM) makes sense (preferably the same as the From header). Best to send mail to some Gmail account, and see what it found the smtp.mailfrom to be. It it’s not a legal domain there, Gmail refuses to accept the mail.
• Enable STARTTLS for incoming mails. See separate post on this.

• Query your IP’s status at SPFBL and possibly delist it from the blacklist. It requires a working MTA on the server with postmaster being user on the domain. Spamassassin relies on this service.
• Register the domain at Gmail’s Postmaster Tools to solve delivery problems to Gmail if such occur. I also have a feeling that this might reduce Gmail’s spam rating of the domain (it’s like someone takes responsibility for it).
• Set up a Microsoft account and join SDNS and JMRP. I’ve written a separate post on this, because somehow everything related to Microsoft requires extra effort.

Setting up sendmail

Important general note: Sendmail is made to work sensibly out of the box. It’s clever enough to relay any mail received from localhost to external servers, and not to do that with mails from external connections. Unless you explicitly tell it to become a spam relay. The default configuration files are installed with apt are fine and probably secure.

Sendmail’s internals, on the other hand, with all macros and stuff, is completely horrible.

So the trick is to make minimal changes. There really isn’t much that needs to be done. For a fairly regular mail configuration, there is very little to do (on Debian 8, that is).

So first, install it:

# apt install sendmail

Not just sendmail-bin. It won’t work. Don’t install rmail — it’s for UUCP. Which is ancient and disabled anyhow.

Now changes in the configuration file. By default on Debian 8, sendmail listens to port 25 and 587 at IPv4′s localhost only, and relays mails to external servers as necessary. In order to open ports 25 and 587 for incoming mail to local addresses only from any host, change the line in /etc/mail/sendmail.mc saying

DAEMON_OPTIONS(`Family=inet,  Name=MTA-v4, Port=smtp, Addr=127.0.0.1')dnl
DAEMON_OPTIONS(`Family=inet,  Name=MSP-v4, Port=submission, M=Ea, Addr=127.0.0.1')dnl

to

DAEMON_OPTIONS(`Family=inet,  Name=IPv4-port-25, Port=smtp, M=E')dnl
DAEMON_OPTIONS(`Family=inet,  Name=IPv4-port-587, Port=submission, M=E')dnl

Let’s explain the changes:

• Most important, the “Addr=” part was dropped, meaning connections from any host is allowed. Sendmail isn’t stupid: If the connection is from localhost, the destination can be any (including relaying to any host on the web), but if it comes from anywhere else, it’s for local addresses only. So we don’t turn into a spam machine. In other words, this is what a session with an external client looks like:

  >>> MAIL FROM:<sender@nowhere.com>
  <<< 250 2.1.0 <sender@nowhere.com>... Sender ok
  >>> RCPT TO:<anybody@not-here.com>
  <<< 550 5.7.1 <anybody@not-here.com>... Relaying denied

• It’s “M=E” for both. Note that the “a” part was dropped, so access is without authentication. It’s intended for anyone to drop mails to local users. On the other hand “E” prevents ETRN on both, as they are both exposed.
• The change in “Name”. Well, it’s just a name with the sole purpose of appearing in the logs on the “daemon=” part. So it better say something meaningful to humans, like which port the connection took place on.

This is a good time to mention that in sendmailish, it’s as if there were two separate MTA daemons, one for each port. This is the terminology used in the log.

Quote at the top of the file, after the DOMAN() assignment, I added a

define(`confDOMAIN_NAME', `mx.example.com')dnl

This sets the sendmail’s host name, as presented while talking to clients, in particular on HELO/EHLO (there is no need to set the confHELO_NAME / HeloName option). Even if it happens to give the correct name without it, I would set it like this. It’s crucial that identifies itself with the name it’s expected to give, or SPF checks can fail.

And of course, set it to the rDNS of your IP address, not mx.example.com.

Get verbose output on log

If something doesn’t work, more info on the log helps a lot. Enable this by adding

define(`confLOG_LEVEL', `14')dnl

to the sendmail.mc, and run “make” to compile the change. One of the pieces of information is the command used to invoke sendmail, which happens to be:

/usr/sbin/sendmail-mta -Am -L sm-mta -bd -q10m

Setting up “virtual users”

Having email addresses that don’t match any actual user names on the machine requires defining “virtual users”. But first, it’s essential to tell sendmail to accept emails to other domains than its own. To do this, add one line for each domain. If there are subdomains, add one line for each as well (by default, sendmail wants this explicitly). So I added the following line to /etc/mail/local-host-names:

billauer.se

This makes sendmail consider these domains local. An important side effect of this is that now root@billauer.se is a legal alias for the local root account. This is an often guessed address by spammers. Handled below.

Then enable virtual users. I put this after the other FEATURE statements in/etc/mail/sendmail.mc:

FEATURE(`virtusertable')dnl

And then run “make” under /etc/mail to update sendmail.cf. And restart sendmail.

Finally, prepare a file with a list of mail addresses, and to which read user they should be routed. First column in the mail address, the second is the target. For simplicity, keep the second column with real local users, but it’s also possible to use other first-column entries as the target. By why messing.

This goes to the file named /etc/mail/virtusertable. This is what it could look like:

someone@billauer.se		root
not-me@billauer.se		root

And then call “make” under /etc/mail, which updates /etc/mail/virtusertable.db. There is no need to restart sendmail to make the changes in virtusertable.db take effect.

Mail addresses as well as domains are case-insensitive, of course. But there are no shortcuts with subdomains: Everything after the “@” must match.

Now preventing spammers from sending mails to root@billauer.se. Just add this line to /etc/mail/virtusertable:

root@billauer.se                     error:nouser User unknown

This causes sendmail to reject the mail address flat at connection:

Apr 23 08:29:05 sm-mta[12752]: x3N8T4Sn012752: <root@billauer.se>... User unknown

But what happens if an internal mail to root is sent, from some cron job, for example? Is it rejected as well? That wasn’t the purpose. Well, on my machine this isn’t a problem, because these mails are sent to root@theserver.billauer.se (as defined in /etc/hosts?), so they’re not caught by the virtual user rule above. I don’t know what the result would be without this subdomain thing.

Rejecting IPv6

Why? Because IPv6 is where everything gets messy. Sendmail is already configured not to listen to IPv6, but then, when it’s about to relay to another server, things get ugly. In particular with Gmail, which supplies an IPv6 AAAA DNS entry for its MX servers.

The problem is that sendmail first attempts IPv6, no matter what (see Nov 30 2018 remark after some discussion on this page).It seems to be an Microsoft-style attempt to push IPv6 by forcing everyone to use it. I would have compiled sendmail myself to get rid of this “feature”, but there’s an easier way. So my own attempt to add a

CLIENT_OPTIONS(`Family=inet')dnl

in the sendmail.mc file, turning into

O ClientPortOptions=Family=inet

in sendmail.cf, didn’t make any difference. It should have turned IPv6 off, but didn’t: Sendmail tries IPv6 first, fails, among others because my firewall kills all incoming IPv6 packets, and after a minute goes for IPv4. So why wait?

My solution was to set the firewall to reject the outgoing IPv6 packet, so any TCP connection gets an immediate RST. This doesn’t prevent sendmail from trying IPv6, but makes it clear it’s a no-go. So it doesn’t waste time on it.

These are my firewall rules for that. It’s the OUTPUT rules that I added specially for sendmail:

# ip6tables -A INPUT -i lo -j ACCEPT
# ip6tables -A INPUT -j DROP
# ip6tables -A OUTPUT -o lo -j ACCEPT
# ip6tables -A OUTPUT -j REJECT --reject-with icmp6-addr-unreachable

Add trusted users

If you have CGI scripts on the web server using the -f flag to set the sender, a message of this sort is likely to appear (with sendmail 8.14, at least), unless www-data (or whatever user the web server runs with) is considered trusted:

X-Authentication-Warning: www-data set sender to mailer@mydomain.com using -f

That seems to provoke spam filters. To stop this, add the following line to /etc/mail/submit.mc

FEATURE(use_ct_file)dnl

just before the line saying “FEATURE(`msp’, ” etc.

Then add a list of users that are allowed to use -f flag without the warning to /etc/mail/trusted-users. It’s just a user for each line. So it can be like this:

www-data
eli

and then run “make” to produce submit.cf, and restart sendmail.

Note that this procedure depends strongly on the sendmail version, and this worked on sendmail 8.14.

Reviewing sendmail’s setup

For the real masochists out there, open /etc/sendmail.cf.

• Lines starting with # are comments, of course.
• Lines starting with “O” are options.
• Searching the file for “=/” reveals all file-relating settings (because it’s an assignment followed by the beginning of an absolute path)

Setting up DKIM

I have a separate post on DKIM and friends. Better take a look if this is Chinese to you.

opendkim is made to work sensibly. It is inserted as a mail filter (“Milter”) for sendmail, making it sign outbound messages, and check inbound messages. As with sendmail, there are a few things to set up, and it’s good to go.

Following this guide (more or less). And man opendkim.conf, which is good. First, install:

# apt install opendkim opendkim-tools

Then create the keys:

# mkdir -p opendkim/keys/billauer.se
# opendkim-genkey -D /etc/opendkim/keys/billauer.se/ -d billauer.se -s dkim2019
# chown -R opendkim:opendkim /etc/opendkim/keys/

Now Configuration. The only changes I needed to make from the default files were: Edit /etc/default/opendkim, adding the following line at the end, so a TCP port is opened:

SOCKET="inet:8891@localhost" # listen on localhost port 8891

and since I need to sign multiple domains, added these two lines to /etc/opendkim.conf

KeyTable		refile:/etc/opendkim/KeyTable
SigningTable		refile:/etc/opendkim/SigningTable

and added the two following files. /etc/opendkim/KeyTable reading

dkim2019._domainkey.billauer.se billauer.se:dkim2019:/etc/opendkim/keys/billauer.se/dkim2019.private
dkim2019._domainkey.example.com example.com:dkim2019:/etc/opendkim/keys/example.com/dkim2019.private

and /etc/opendkim/SigningTable:

*@billauer.se dkim2019._domainkey.billauer.se
*@example.com dkim2019._domainkey.example.com

For a server whose outbound messages come only from localhost, there’s no need to set neither InternalHosts nor ExternalIgnoreList, as this is the default. These appear in a lot of tutorials.

Finally, make the DKIM a mail filter (“Milter”) on sendmail by adding this line at the end of sendmail.mc (and run “make” + restart sendmail):

INPUT_MAIL_FILTER(`opendkim', `S=inet:8891@127.0.0.1, F=T')

Note the “F=T” part. It will make sendmail refuse to accept mails if the DKIM server isn’t responding properly with a

451 4.3.2 Please try again later

The default is to pass the mail through without the milter if it doesn’t work, which would mean sending unsigned mails without paying attention. The backside of this is that no mail will arrive either if this happens, but at least the delivery won’t fail completely (assuming the issue is resolved within a day or so).

Don’t forget to set up the TXT DNS records with the *.txt files generated with opendkim-genkey. These files are written in zone format for the bind daemon. The actual text is the concatenation of the two strings in quotation marks, after removing these quotation marks. Think of it as a multi-line string in C language.

All done? Use this DKIM validator to see exactly how well it went.

Remove outbound messages from the mailing queue

# mailq
MSP Queue status...
/var/spool/mqueue-client is empty
		Total requests: 0
MTA Queue status...
		/var/spool/mqueue (2 requests)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
x22FdWV1010569     1864 Sat Mar  2 10:39 MAILER-DAEMON
                 (Deferred: Connection timed out with server.com.)
					 <ze@server.com>
x22FMGAi009668       17 Sat Mar  2 10:23 <this@there.com>
                 (Deferred: Connection timed out with example.com.)
					 <eli@example.com>
		Total requests: 2
# cd /var/spool/mqueue
# rm *x22FdWV1010569
# rm *x22FMGAi009668
# systemctl restart sendmail

Gmail won’t talk with anyone

Gmail’s server doesn’t respond to a SYN at port 587 or 25, and won’t talk to you unless you have an rDNS. Only after having the rDNS set on the server:

# nc gmail-smtp-in.l.google.com. 25
220 mx.google.com ESMTP y6si2100605wmi.83 - gsmtp

And that’s just the beginning. Without having DMARC set up, it wouldn’t relay my mails. It also seems a very good idea to have the DMARC policy reject emails that don’t meet the criteria for significantly better deliverability with Gmail. Just a hunch, but based upon experience.

More on DMARC here.

Sources of information

• The basic doc for sendmail.
• The DKIM validator which accepts mails from your server and checks them.
• MX Tools for testing the mail server
• My own blog posts on outgoing SMTP servers and manual relaying (and getting the session log).
• This excellent guide
• This page on how to set up DKIM DNS entries, and Wikipedia’s.
• Amazon’s page on MAIL FROM.
• This guide to setting up postfix on Debian 8.
• Hardening of postfix and sendmail.
• This page on DMARC.