SMTP or DNS? Tales of a Mail Administrator
The servers in question. And the skulls of attackers who made it in, and paid the price.
This blog post is about an email problem I was having. What makes it interesting is the context.
I run my own mail server. Yes I do have the markloveless.net domain hosted on a Squarespace instance, and the mail for that domain is via Gmail. But my main domain, the one I’ve had since the last century, has a mail server that is sitting in my house. Wait, don't make fun of me for running a mail server just yet.
I do run and have always run Sendmail. Okay, now you can make fun of me.
I know, I am a security guy, and yes there have been issues with Sendmail and security in the past. I even helped find some of those security problems (including the last remote heap buffer overflow in 2004, assigned a CVE five years later). But this is all about one of those odd mail delivery problems that a mail server administrator has. I put this out there as an example of a hobby that causes me immense pain and anguish, but is still like a wonderfully complex Sunday crossword puzzle. Additionally, this has taught me a lot about various protocols, and since I know something about network protocol analysis, that has helped me keep that part of my skill set strong.
Wait, your own mail server?
The mail server in question is for the nmrc.org domain (I also manage DNS and HTTP servers for this domain as well). I've had this domain since the mid 90s, and having found the amazing registrar Gandi in 1997, migrated there and been there ever since. I mention this because Gandi also handles two backup DNS servers.
A great way to really learn how some of these mail protocols actually work is to get your hands dirty and run some of these protocols yourself on a public-facing server. Granted, I never trusted my mail spool to the likes of Yahoo or Hotmail like so many others, and running my own mail server for my own domain meant I could rest in the knowledge that if the Feds wanted my mail spool, I would absolutely know they showed up to get it. With someone like Gmail, the Feds just hand them some paperwork and Google hands it over - and I might not ever know about it. At home, after they crash down the door and are standing on my chest, I can say "Ah I see, you must want my mail spool!" I'd know about it.
But I digress. This mail server business is no picnic. I'm using several DNS blackholes, greylisting, configured my setup as best as I can for anti-spam including relay prevention. I even maintain a rather aggressive access file, where I block thousands of IP addresses I've accumulated over the years - including numerous class Cs (even a few class Bs and As) that I should never receive email from. I block spammy ISPs. I block all of Yahoo, Hotmail, and AOL (exceptions for a few friends who have those addresses). I use SMTPS so when other mail servers that support it connect to me, the email is encrypted. I've even configured up SPF, DKIM, and DMARC. Things are fairly decent and I learned tons while doing it.
The Problem
So imagine my head scratching when my wife reported that a local business was trying to email her and claimed to be getting errors, yet there was nothing in the mail logs to show an error. Even weirder, the email errors that the local business was getting made no sense, because they pointed at my mail server. And to top it all off, my wife still got all of the emails. Eventually the local business forwarded one of the emails with the error and I had to do some serious digging.
The main mail server for nmrc.org is talon.nmrc.org. I do have a couple of other domains with their email running on rigor-mortis.nmrc.org. The error stated that the NMRC mail was getting a relaying error on rigor-mortis.nmrc.org. This made no sense, as the MX record for the NMRC domain was in fact pointing at talon.nmrc.org, yet the logs on rigor-mortis.nmrc.org clearly show the relaying errors. There were plenty of them from other businesses as well, so something was going on.
May 28 08:25:35 rigor-mortis sm-mta[151721]: 04SDPXC5151721: ruleset=check_rcpt,
arg1=<xxxxxxxx@nmrc.org>, relay=mail-mw2nam10on2117.outbound.protection.outlook
.com [40.107.94.117], reject=550 5.7.1 <xxxxxxxx@nmrc.org>... Relaying denied.
Proper authentication required.One of the error messages. My MX record does not point here, what is going on?
Here was what happened. A few months ago, I disabled the extremely low-traffic nmrc.org web server from its location at a co-lo in California, and put the web server on rigor-mortis.nmrc.org. The web site's IP address was switched to rigor-mortis.nmrc.org as well, which was easy since I control my own DNS. So as nearly as I can tell, most of the email that was coming in for nmrc.org was going to the talon.nmrc.org as that was the MX server, while maybe 2-3% of email was resolving "nmrc.org" which points at rigor-mortis. Oddly, this 2-3% would later retry after failing and used the MX listing, which was why the error never showed up on talon, and email was still being delivered.
The Fix
Now how do I fix this? Well for starters, I looked at DNS. I confirmed that I had exactly one MX record - talon.nmrc.org. My SOA record on the main DNS server did not point at the name server (NS), which was a mistake that had been there for ages, from an ancient time years ago. I have no idea how many timed I edited the nmrc.org zone and simply missed it. This might have contributed to those 2-3% after the co-lo move, as that was when the IP address of "nmrc.org" changed. However I changed the IP address for nmrc.org to be the same as talon.nmrc.org, and that seemed to have taken care of the issue. Additionally I had set the SOA pointing to the NS server at the same time, and email began to work properly.
$TTL 604800 @ IN SOA daemon.nmrc.org. thegnome.nmrc.org. (
2020052800 ; serial
43200 ; Refresh period
7200 ; Retry interval
2419200 ; Expire time
86400 ; Negative caching TTL
) ;
@ IN NS daemon.nmrc.org.
@ IN A 65.70.17.131
@ IN MX 5 talon.nmrc.org.
daemon IN A 65.70.17.133
talon IN A 65.70.17.131
vortex IN A 65.70.17.130
rigor-mortis IN A 65.70.17.132
blackhole IN A 65.70.17.129
www IN A 65.70.17.132
web IN CNAME www
dragon IN A 65.70.17.134
_dmarc IN TXT "v=DMARC1; p=none"
default._domainkey IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSI
b3DQEBAQUAA4GNADCBiQKBgQDorm4fA6kdVeQty3S5LvhjJKp2H7aeKkw+UVP5vssEqBUZ/VS
0ph7Ou/DF0RxDricc7uGkqc0hxbe5k1pImvWD0p/ga61VM/54kozgiTKZ0yAX9UkdF5o8fC/M
b1PP5Bm+I9t5I2yyxRzXVTTQOATQEyhhurqWlgIVZv8Y9drBBQIDAQAB"
@ IN TXT "exploring the world one class C at a time"
@ IN TXT "there is no dana here, only zuul"
@ IN TXT "v=spf1 mx:nmrc.org -all"My now-corrected zone file. No, it is not a security risk to post this as this is easily obtained via reverse lookups - it is intended to be public information.
There are still a couple of backwards email servers out there sending to the wrong place, but once DNS fully updates across the interwebs, everything should be fine. Of course this broke part of the web server, but after quick adjustments to Apache2 configuration files, everything started working.
Is this common? Actually something seems to happen every few months. There are a lot of configuration choices to be made when setting up a mail server, and for most people once they have mail flowing they tend to leave it alone. I don't. Every time of the big players on the Internet decides to reinterpret an email-related RFC, most of the rest of the mail community have to play along (looking at you, Gmail). Is it frustrating? Sure, but as a result of changes over the years I've learned things about SMTPS, SPF, DKIM, DMARC, and a deeper understanding of how these all interact. The important lesson though, is that while the problem manifested itself as an SMTP problem, it was in fact a DNS problem.
If the idea of running your own mail server is as crazy as letting Google, Microsoft, or Yahoo run it for you, you could always use something like ProtonMail (which can do free accounts and has a pay option for organizations). But it is a fun hobby that helps keep me sharp, and I’ve enjoyed the fact that it is still up and running despite tons of changes over time to all of the protocols.
